---
library_name: peft
license: gemma
base_model: google/codegemma-7b
tags:
- trl
- sft
- generated_from_trainer
model-index:
- name: code-bench-CodeGemma-7B-cgv1-ds
  results: []
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# code-bench-CodeGemma-7B-cgv1-ds

This model is a fine-tuned version of [google/codegemma-7b](https://huggingface.co/google/codegemma-7b) on the None dataset.
It achieves the following results on the evaluation set:
- Loss: 0.0947

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 1
- eval_batch_size: 3
- seed: 42
- distributed_type: multi-GPU
- gradient_accumulation_steps: 8
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.03
- num_epochs: 2
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch  | Step | Validation Loss |
|:-------------:|:------:|:----:|:---------------:|
| 0.9203        | 0.0530 | 50   | 1.0306          |
| 0.551         | 0.1061 | 100  | 0.5383          |
| 0.4483        | 0.1591 | 150  | 0.4048          |
| 0.3469        | 0.2121 | 200  | 0.3013          |
| 0.2868        | 0.2652 | 250  | 0.2447          |
| 0.2307        | 0.3182 | 300  | 0.2061          |
| 0.1972        | 0.3713 | 350  | 0.1727          |
| 0.1716        | 0.4243 | 400  | 0.1525          |
| 0.1612        | 0.4773 | 450  | 0.1468          |
| 0.1631        | 0.5304 | 500  | 0.1400          |
| 0.1739        | 0.5834 | 550  | 0.1376          |
| 0.148         | 0.6364 | 600  | 0.1330          |
| 0.1413        | 0.6895 | 650  | 0.1274          |
| 0.1464        | 0.7425 | 700  | 0.1267          |
| 0.1376        | 0.7955 | 750  | 0.1240          |
| 0.1287        | 0.8486 | 800  | 0.1210          |
| 0.1402        | 0.9016 | 850  | 0.1198          |
| 0.1261        | 0.9547 | 900  | 0.1173          |
| 0.1195        | 1.0077 | 950  | 0.1145          |
| 0.1254        | 1.0607 | 1000 | 0.1133          |
| 0.1109        | 1.1138 | 1050 | 0.1119          |
| 0.1206        | 1.1668 | 1100 | 0.1093          |
| 0.1195        | 1.2198 | 1150 | 0.1084          |
| 0.1237        | 1.2729 | 1200 | 0.1073          |
| 0.1205        | 1.3259 | 1250 | 0.1064          |
| 0.1105        | 1.3789 | 1300 | 0.1048          |
| 0.1027        | 1.4320 | 1350 | 0.1038          |
| 0.1128        | 1.4850 | 1400 | 0.1035          |
| 0.1207        | 1.5381 | 1450 | 0.1030          |
| 0.1057        | 1.5911 | 1500 | 0.1013          |
| 0.1056        | 1.6441 | 1550 | 0.0996          |
| 0.1086        | 1.6972 | 1600 | 0.0985          |
| 0.1078        | 1.7502 | 1650 | 0.0982          |
| 0.0987        | 1.8032 | 1700 | 0.0968          |
| 0.1037        | 1.8563 | 1750 | 0.0960          |
| 0.1047        | 1.9093 | 1800 | 0.0957          |
| 0.1045        | 1.9623 | 1850 | 0.0947          |


### Framework versions

- PEFT 0.12.0
- Transformers 4.44.2
- Pytorch 2.5.1+cu121
- Datasets 2.21.0
- Tokenizers 0.19.1