--- library_name: peft license: gemma base_model: google/codegemma-7b tags: - trl - sft - generated_from_trainer model-index: - name: code-bench-CodeGemma-7B-cgv1-ds results: [] --- # code-bench-CodeGemma-7B-cgv1-ds This model is a fine-tuned version of [google/codegemma-7b](https://huggingface.co/google/codegemma-7b) on the None dataset. It achieves the following results on the evaluation set: - Loss: 0.0653 ## Model description More information needed ## Intended uses & limitations More information needed ## Training and evaluation data More information needed ## Training procedure ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 2e-08 - train_batch_size: 1 - eval_batch_size: 3 - seed: 42 - distributed_type: multi-GPU - gradient_accumulation_steps: 8 - total_train_batch_size: 8 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: linear - lr_scheduler_warmup_ratio: 0.03 - num_epochs: 1 - mixed_precision_training: Native AMP ### Training results | Training Loss | Epoch | Step | Validation Loss | |:-------------:|:------:|:----:|:---------------:| | 0.9203 | 0.0530 | 50 | 1.0306 | | 0.551 | 0.1061 | 100 | 0.5383 | | 0.4483 | 0.1591 | 150 | 0.4048 | | 0.3469 | 0.2121 | 200 | 0.3013 | | 0.2868 | 0.2652 | 250 | 0.2447 | | 0.2307 | 0.3182 | 300 | 0.2061 | | 0.1972 | 0.3713 | 350 | 0.1727 | | 0.1716 | 0.4243 | 400 | 0.1525 | | 0.1612 | 0.4773 | 450 | 0.1468 | | 0.1631 | 0.5304 | 500 | 0.1400 | | 0.1739 | 0.5834 | 550 | 0.1376 | | 0.148 | 0.6364 | 600 | 0.1330 | | 0.1413 | 0.6895 | 650 | 0.1274 | | 0.1464 | 0.7425 | 700 | 0.1267 | | 0.1376 | 0.7955 | 750 | 0.1240 | | 0.1287 | 0.8486 | 800 | 0.1210 | | 0.1402 | 0.9016 | 850 | 0.1198 | | 0.1261 | 0.9547 | 900 | 0.1173 | | 0.1195 | 1.0077 | 950 | 0.1145 | | 0.1254 | 1.0607 | 1000 | 0.1133 | | 0.1109 | 1.1138 | 1050 | 0.1119 | | 0.1206 | 1.1668 | 1100 | 0.1093 | | 0.1195 | 1.2198 | 1150 | 0.1084 | | 0.1237 | 1.2729 | 1200 | 0.1073 | | 0.1205 | 1.3259 | 1250 | 0.1064 | | 0.1105 | 1.3789 | 1300 | 0.1048 | | 0.1027 | 1.4320 | 1350 | 0.1038 | | 0.1128 | 1.4850 | 1400 | 0.1035 | | 0.1207 | 1.5381 | 1450 | 0.1030 | | 0.1057 | 1.5911 | 1500 | 0.1013 | | 0.1056 | 1.6441 | 1550 | 0.0996 | | 0.1086 | 1.6972 | 1600 | 0.0985 | | 0.1078 | 1.7502 | 1650 | 0.0982 | | 0.0987 | 1.8032 | 1700 | 0.0968 | | 0.1037 | 1.8563 | 1750 | 0.0960 | | 0.1047 | 1.9093 | 1800 | 0.0957 | | 0.109 | 1.9631 | 1850 | 0.0953 | | 0.1099 | 2.0162 | 1900 | 0.0938 | | 0.102 | 2.0692 | 1950 | 0.0927 | | 0.1063 | 2.1222 | 2000 | 0.0929 | | 0.0985 | 2.1753 | 2050 | 0.0910 | | 0.0936 | 2.2283 | 2100 | 0.0908 | | 0.0998 | 2.2814 | 2150 | 0.0908 | | 0.0935 | 2.3344 | 2200 | 0.0905 | | 0.1028 | 2.3874 | 2250 | 0.0904 | | 0.1036 | 2.4405 | 2300 | 0.0899 | | 0.0998 | 2.4943 | 2350 | 0.0888 | | 0.0923 | 2.5473 | 2400 | 0.0890 | | 0.0979 | 2.6004 | 2450 | 0.0887 | | 0.1012 | 2.6534 | 2500 | 0.0879 | | 0.0936 | 2.7064 | 2550 | 0.0882 | | 0.0948 | 2.7595 | 2600 | 0.0876 | | 0.0862 | 2.8125 | 2650 | 0.0858 | | 0.0979 | 2.8656 | 2700 | 0.0853 | | 0.0873 | 2.9186 | 2750 | 0.0858 | | 0.0901 | 2.9716 | 2800 | 0.0856 | | 0.0862 | 3.0247 | 2850 | 0.0838 | | 0.0901 | 3.0777 | 2900 | 0.0825 | | 0.0838 | 3.1307 | 2950 | 0.0829 | | 0.0873 | 3.1838 | 3000 | 0.0830 | | 0.0798 | 3.2368 | 3050 | 0.0816 | | 0.0845 | 3.2898 | 3100 | 0.0804 | | 0.0831 | 3.3429 | 3150 | 0.0804 | | 0.081 | 3.3959 | 3200 | 0.0792 | | 0.0842 | 3.4490 | 3250 | 0.0790 | | 0.0749 | 3.5020 | 3300 | 0.0792 | | 0.0873 | 3.5550 | 3350 | 0.0795 | | 0.0754 | 3.6081 | 3400 | 0.0794 | | 0.0779 | 3.6611 | 3450 | 0.0793 | | 0.0809 | 3.7141 | 3500 | 0.0774 | | 0.0807 | 3.7672 | 3550 | 0.0773 | | 0.0761 | 3.8202 | 3600 | 0.0775 | | 0.0736 | 3.8732 | 3650 | 0.0757 | | 0.0746 | 3.9263 | 3700 | 0.0759 | | 0.0844 | 3.9793 | 3750 | 0.0760 | | 0.0764 | 4.0324 | 3800 | 0.0758 | | 0.0722 | 4.0854 | 3850 | 0.0754 | | 0.068 | 4.1384 | 3900 | 0.0752 | | 0.0641 | 4.1915 | 3950 | 0.0736 | | 0.0665 | 4.2445 | 4000 | 0.0733 | | 0.0674 | 4.2975 | 4050 | 0.0736 | | 0.0693 | 4.3506 | 4100 | 0.0724 | | 0.072 | 4.4036 | 4150 | 0.0714 | | 0.0683 | 4.4566 | 4200 | 0.0716 | | 0.061 | 4.5097 | 4250 | 0.0718 | | 0.0653 | 4.5627 | 4300 | 0.0706 | | 0.0707 | 4.6158 | 4350 | 0.0702 | | 0.0719 | 4.6688 | 4400 | 0.0714 | | 0.0669 | 4.7218 | 4450 | 0.0706 | | 0.0673 | 4.7749 | 4500 | 0.0710 | | 0.0677 | 4.8279 | 4550 | 0.0714 | | 0.0795 | 4.8809 | 4600 | 0.0700 | | 0.0724 | 4.9340 | 4650 | 0.0699 | | 0.0648 | 4.9870 | 4700 | 0.0707 | | 0.0614 | 5.0400 | 4750 | 0.0696 | | 0.0606 | 5.0931 | 4800 | 0.0691 | | 0.0579 | 5.1461 | 4850 | 0.0691 | | 0.0645 | 5.1992 | 4900 | 0.0680 | | 0.0648 | 5.2522 | 4950 | 0.0673 | | 0.0624 | 5.3052 | 5000 | 0.0672 | | 0.0604 | 5.3583 | 5050 | 0.0676 | | 0.0582 | 5.4113 | 5100 | 0.0672 | | 0.0622 | 5.4643 | 5150 | 0.0664 | | 0.0589 | 5.5174 | 5200 | 0.0665 | | 0.0586 | 5.5704 | 5250 | 0.0664 | | 0.0577 | 5.6234 | 5300 | 0.0666 | | 0.0538 | 5.6765 | 5350 | 0.0660 | | 0.0616 | 5.7295 | 5400 | 0.0657 | | 0.0582 | 5.7826 | 5450 | 0.0661 | | 0.0599 | 5.8356 | 5500 | 0.0653 | ### Framework versions - PEFT 0.12.0 - Transformers 4.44.2 - Pytorch 2.5.1+cu121 - Datasets 2.21.0 - Tokenizers 0.19.1