metadata
library_name: peft
license: gemma
base_model: google/codegemma-7b
tags:
- trl
- sft
- generated_from_trainer
model-index:
- name: code-bench-CodeGemma-7B-cgv1-ds
results: []
code-bench-CodeGemma-7B-cgv1-ds
This model is a fine-tuned version of google/codegemma-7b on the None dataset. It achieves the following results on the evaluation set:
- Loss: 0.0653
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-08
- train_batch_size: 1
- eval_batch_size: 3
- seed: 42
- distributed_type: multi-GPU
- gradient_accumulation_steps: 8
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.03
- num_epochs: 1
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 0.9203 | 0.0530 | 50 | 1.0306 |
| 0.551 | 0.1061 | 100 | 0.5383 |
| 0.4483 | 0.1591 | 150 | 0.4048 |
| 0.3469 | 0.2121 | 200 | 0.3013 |
| 0.2868 | 0.2652 | 250 | 0.2447 |
| 0.2307 | 0.3182 | 300 | 0.2061 |
| 0.1972 | 0.3713 | 350 | 0.1727 |
| 0.1716 | 0.4243 | 400 | 0.1525 |
| 0.1612 | 0.4773 | 450 | 0.1468 |
| 0.1631 | 0.5304 | 500 | 0.1400 |
| 0.1739 | 0.5834 | 550 | 0.1376 |
| 0.148 | 0.6364 | 600 | 0.1330 |
| 0.1413 | 0.6895 | 650 | 0.1274 |
| 0.1464 | 0.7425 | 700 | 0.1267 |
| 0.1376 | 0.7955 | 750 | 0.1240 |
| 0.1287 | 0.8486 | 800 | 0.1210 |
| 0.1402 | 0.9016 | 850 | 0.1198 |
| 0.1261 | 0.9547 | 900 | 0.1173 |
| 0.1195 | 1.0077 | 950 | 0.1145 |
| 0.1254 | 1.0607 | 1000 | 0.1133 |
| 0.1109 | 1.1138 | 1050 | 0.1119 |
| 0.1206 | 1.1668 | 1100 | 0.1093 |
| 0.1195 | 1.2198 | 1150 | 0.1084 |
| 0.1237 | 1.2729 | 1200 | 0.1073 |
| 0.1205 | 1.3259 | 1250 | 0.1064 |
| 0.1105 | 1.3789 | 1300 | 0.1048 |
| 0.1027 | 1.4320 | 1350 | 0.1038 |
| 0.1128 | 1.4850 | 1400 | 0.1035 |
| 0.1207 | 1.5381 | 1450 | 0.1030 |
| 0.1057 | 1.5911 | 1500 | 0.1013 |
| 0.1056 | 1.6441 | 1550 | 0.0996 |
| 0.1086 | 1.6972 | 1600 | 0.0985 |
| 0.1078 | 1.7502 | 1650 | 0.0982 |
| 0.0987 | 1.8032 | 1700 | 0.0968 |
| 0.1037 | 1.8563 | 1750 | 0.0960 |
| 0.1047 | 1.9093 | 1800 | 0.0957 |
| 0.109 | 1.9631 | 1850 | 0.0953 |
| 0.1099 | 2.0162 | 1900 | 0.0938 |
| 0.102 | 2.0692 | 1950 | 0.0927 |
| 0.1063 | 2.1222 | 2000 | 0.0929 |
| 0.0985 | 2.1753 | 2050 | 0.0910 |
| 0.0936 | 2.2283 | 2100 | 0.0908 |
| 0.0998 | 2.2814 | 2150 | 0.0908 |
| 0.0935 | 2.3344 | 2200 | 0.0905 |
| 0.1028 | 2.3874 | 2250 | 0.0904 |
| 0.1036 | 2.4405 | 2300 | 0.0899 |
| 0.0998 | 2.4943 | 2350 | 0.0888 |
| 0.0923 | 2.5473 | 2400 | 0.0890 |
| 0.0979 | 2.6004 | 2450 | 0.0887 |
| 0.1012 | 2.6534 | 2500 | 0.0879 |
| 0.0936 | 2.7064 | 2550 | 0.0882 |
| 0.0948 | 2.7595 | 2600 | 0.0876 |
| 0.0862 | 2.8125 | 2650 | 0.0858 |
| 0.0979 | 2.8656 | 2700 | 0.0853 |
| 0.0873 | 2.9186 | 2750 | 0.0858 |
| 0.0901 | 2.9716 | 2800 | 0.0856 |
| 0.0862 | 3.0247 | 2850 | 0.0838 |
| 0.0901 | 3.0777 | 2900 | 0.0825 |
| 0.0838 | 3.1307 | 2950 | 0.0829 |
| 0.0873 | 3.1838 | 3000 | 0.0830 |
| 0.0798 | 3.2368 | 3050 | 0.0816 |
| 0.0845 | 3.2898 | 3100 | 0.0804 |
| 0.0831 | 3.3429 | 3150 | 0.0804 |
| 0.081 | 3.3959 | 3200 | 0.0792 |
| 0.0842 | 3.4490 | 3250 | 0.0790 |
| 0.0749 | 3.5020 | 3300 | 0.0792 |
| 0.0873 | 3.5550 | 3350 | 0.0795 |
| 0.0754 | 3.6081 | 3400 | 0.0794 |
| 0.0779 | 3.6611 | 3450 | 0.0793 |
| 0.0809 | 3.7141 | 3500 | 0.0774 |
| 0.0807 | 3.7672 | 3550 | 0.0773 |
| 0.0761 | 3.8202 | 3600 | 0.0775 |
| 0.0736 | 3.8732 | 3650 | 0.0757 |
| 0.0746 | 3.9263 | 3700 | 0.0759 |
| 0.0844 | 3.9793 | 3750 | 0.0760 |
| 0.0764 | 4.0324 | 3800 | 0.0758 |
| 0.0722 | 4.0854 | 3850 | 0.0754 |
| 0.068 | 4.1384 | 3900 | 0.0752 |
| 0.0641 | 4.1915 | 3950 | 0.0736 |
| 0.0665 | 4.2445 | 4000 | 0.0733 |
| 0.0674 | 4.2975 | 4050 | 0.0736 |
| 0.0693 | 4.3506 | 4100 | 0.0724 |
| 0.072 | 4.4036 | 4150 | 0.0714 |
| 0.0683 | 4.4566 | 4200 | 0.0716 |
| 0.061 | 4.5097 | 4250 | 0.0718 |
| 0.0653 | 4.5627 | 4300 | 0.0706 |
| 0.0707 | 4.6158 | 4350 | 0.0702 |
| 0.0719 | 4.6688 | 4400 | 0.0714 |
| 0.0669 | 4.7218 | 4450 | 0.0706 |
| 0.0673 | 4.7749 | 4500 | 0.0710 |
| 0.0677 | 4.8279 | 4550 | 0.0714 |
| 0.0795 | 4.8809 | 4600 | 0.0700 |
| 0.0724 | 4.9340 | 4650 | 0.0699 |
| 0.0648 | 4.9870 | 4700 | 0.0707 |
| 0.0614 | 5.0400 | 4750 | 0.0696 |
| 0.0606 | 5.0931 | 4800 | 0.0691 |
| 0.0579 | 5.1461 | 4850 | 0.0691 |
| 0.0645 | 5.1992 | 4900 | 0.0680 |
| 0.0648 | 5.2522 | 4950 | 0.0673 |
| 0.0624 | 5.3052 | 5000 | 0.0672 |
| 0.0604 | 5.3583 | 5050 | 0.0676 |
| 0.0582 | 5.4113 | 5100 | 0.0672 |
| 0.0622 | 5.4643 | 5150 | 0.0664 |
| 0.0589 | 5.5174 | 5200 | 0.0665 |
| 0.0586 | 5.5704 | 5250 | 0.0664 |
| 0.0577 | 5.6234 | 5300 | 0.0666 |
| 0.0538 | 5.6765 | 5350 | 0.0660 |
| 0.0616 | 5.7295 | 5400 | 0.0657 |
| 0.0582 | 5.7826 | 5450 | 0.0661 |
| 0.0599 | 5.8356 | 5500 | 0.0653 |
Framework versions
- PEFT 0.12.0
- Transformers 4.44.2
- Pytorch 2.5.1+cu121
- Datasets 2.21.0
- Tokenizers 0.19.1