Instructions to use wxzhang/dpo-selective-buffer-safeipo with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use wxzhang/dpo-selective-buffer-safeipo with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="wxzhang/dpo-selective-buffer-safeipo")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("wxzhang/dpo-selective-buffer-safeipo")
model = AutoModelForCausalLM.from_pretrained("wxzhang/dpo-selective-buffer-safeipo", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use wxzhang/dpo-selective-buffer-safeipo with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "wxzhang/dpo-selective-buffer-safeipo"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wxzhang/dpo-selective-buffer-safeipo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/wxzhang/dpo-selective-buffer-safeipo

SGLang

How to use wxzhang/dpo-selective-buffer-safeipo with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "wxzhang/dpo-selective-buffer-safeipo" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wxzhang/dpo-selective-buffer-safeipo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "wxzhang/dpo-selective-buffer-safeipo" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wxzhang/dpo-selective-buffer-safeipo",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use wxzhang/dpo-selective-buffer-safeipo with Docker Model Runner:
```
docker model run hf.co/wxzhang/dpo-selective-buffer-safeipo
```

wxzhang commited on Apr 15, 2024

Commit

b8aa131

verified ·

1 Parent(s): 7b42cac

Model save

Browse files

Files changed (9) hide show

README.md +12 -12
all_results.json +18 -18
eval_results.json +14 -14
model-00001-of-00003.safetensors +1 -1
model-00002-of-00003.safetensors +1 -1
model-00003-of-00003.safetensors +1 -1
train_results.json +4 -4
trainer_state.json +0 -0
training_args.bin +1 -1

README.md CHANGED Viewed

@@ -15,17 +15,17 @@ should probably proofread and complete it, then remove this comment. -->
 This model was trained from scratch on the None dataset.
 It achieves the following results on the evaluation set:
-- Loss: 4336.2285
-- Rewards/chosen: -0.7580
-- Rewards/rejected: -0.8224
-- Rewards/accuracies: 0.6084
-- Rewards/margins: 0.0644
-- Rewards/safe Rewards: -0.7526
-- Rewards/unsafe Rewards: -0.7556
-- Logps/rejected: -174.7150
-- Logps/chosen: -206.2398
-- Logits/rejected: 0.0353
-- Logits/chosen: -0.6151
 ## Model description
@@ -62,7 +62,7 @@ The following hyperparameters were used during training:
 | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Rewards/safe Rewards | Rewards/unsafe Rewards | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
 |:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------------:|:----------------------:|:--------------:|:------------:|:---------------:|:-------------:|
-| 11841.1211    | 0.54  | 500  | 4375.6743       | -0.7240        | -0.8064          | 0.6440             | 0.0823          | -0.7177              | -0.7186                | -173.1074      | -202.8401    | -0.0312         | -0.6399       |
 ### Framework versions

 This model was trained from scratch on the None dataset.
 It achieves the following results on the evaluation set:
+- Loss: 0.3423
+- Rewards/chosen: -1.1017
+- Rewards/rejected: -1.4092
+- Rewards/accuracies: 0.7239
+- Rewards/margins: 0.3075
+- Rewards/safe Rewards: -1.0907
+- Rewards/unsafe Rewards: -1.1124
+- Logps/rejected: -233.3909
+- Logps/chosen: -240.6112
+- Logits/rejected: 1.7395
+- Logits/chosen: 0.4896
 ## Model description
 | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Rewards/safe Rewards | Rewards/unsafe Rewards | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
 |:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------------:|:----------------------:|:--------------:|:------------:|:---------------:|:-------------:|
+| 0.9949        | 0.54  | 500  | 0.3472          | -1.0469        | -1.3054          | 0.7270             | 0.2585          | -1.0403              | -1.0519                | -223.0117      | -235.1269    | 2.0966          | 0.6950        |
 ### Framework versions

all_results.json CHANGED Viewed

@@ -1,23 +1,23 @@
 {
     "epoch": 1.0,
-    "eval_logits/chosen": -0.6150549054145813,
-    "eval_logits/rejected": 0.035322826355695724,
-    "eval_logps/chosen": -206.2398223876953,
-    "eval_logps/rejected": -174.71498107910156,
-    "eval_loss": 4336.228515625,
-    "eval_rewards/accuracies": 0.6084221005439758,
-    "eval_rewards/chosen": -0.7580091953277588,
-    "eval_rewards/margins": 0.06442829966545105,
-    "eval_rewards/rejected": -0.8224374651908875,
-    "eval_rewards/safe_rewards": -0.7526479363441467,
-    "eval_rewards/unsafe_rewards": -0.755587637424469,
-    "eval_runtime": 1067.1431,
     "eval_samples": 33044,
-    "eval_samples_per_second": 30.965,
-    "eval_steps_per_second": 0.968,
-    "train_loss": 11837.785513657158,
-    "train_runtime": 19026.7097,
     "train_samples": 59478,
-    "train_samples_per_second": 3.126,
-    "train_steps_per_second": 0.049
 }

 {
     "epoch": 1.0,
+    "eval_logits/chosen": 0.48959165811538696,
+    "eval_logits/rejected": 1.7394596338272095,
+    "eval_logps/chosen": -240.6112060546875,
+    "eval_logps/rejected": -233.39085388183594,
+    "eval_loss": 0.34227725863456726,
+    "eval_rewards/accuracies": 0.7238625288009644,
+    "eval_rewards/chosen": -1.1017229557037354,
+    "eval_rewards/margins": 0.3074730932712555,
+    "eval_rewards/rejected": -1.4091962575912476,
+    "eval_rewards/safe_rewards": -1.090682029724121,
+    "eval_rewards/unsafe_rewards": -1.1124191284179688,
+    "eval_runtime": 1046.4049,
     "eval_samples": 33044,
+    "eval_samples_per_second": 31.579,
+    "eval_steps_per_second": 0.987,
+    "train_loss": 1.0336197533828193,
+    "train_runtime": 18653.3333,
     "train_samples": 59478,
+    "train_samples_per_second": 3.189,
+    "train_steps_per_second": 0.05
 }

eval_results.json CHANGED Viewed

@@ -1,18 +1,18 @@
 {
     "epoch": 1.0,
-    "eval_logits/chosen": -0.6150549054145813,
-    "eval_logits/rejected": 0.035322826355695724,
-    "eval_logps/chosen": -206.2398223876953,
-    "eval_logps/rejected": -174.71498107910156,
-    "eval_loss": 4336.228515625,
-    "eval_rewards/accuracies": 0.6084221005439758,
-    "eval_rewards/chosen": -0.7580091953277588,
-    "eval_rewards/margins": 0.06442829966545105,
-    "eval_rewards/rejected": -0.8224374651908875,
-    "eval_rewards/safe_rewards": -0.7526479363441467,
-    "eval_rewards/unsafe_rewards": -0.755587637424469,
-    "eval_runtime": 1067.1431,
     "eval_samples": 33044,
-    "eval_samples_per_second": 30.965,
-    "eval_steps_per_second": 0.968
 }

 {
     "epoch": 1.0,
+    "eval_logits/chosen": 0.48959165811538696,
+    "eval_logits/rejected": 1.7394596338272095,
+    "eval_logps/chosen": -240.6112060546875,
+    "eval_logps/rejected": -233.39085388183594,
+    "eval_loss": 0.34227725863456726,
+    "eval_rewards/accuracies": 0.7238625288009644,
+    "eval_rewards/chosen": -1.1017229557037354,
+    "eval_rewards/margins": 0.3074730932712555,
+    "eval_rewards/rejected": -1.4091962575912476,
+    "eval_rewards/safe_rewards": -1.090682029724121,
+    "eval_rewards/unsafe_rewards": -1.1124191284179688,
+    "eval_runtime": 1046.4049,
     "eval_samples": 33044,
+    "eval_samples_per_second": 31.579,
+    "eval_steps_per_second": 0.987
 }

model-00001-of-00003.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a91d9ba47e204876c0ac101e2ef27a0c27d68c135aebaa5c8a60a7e531bd0cfb
 size 4943162336

 version https://git-lfs.github.com/spec/v1
+oid sha256:c0d3ffbcb96de65352cd9800707c3887845c0aad0360888d178eaa584ba7c742
 size 4943162336

model-00002-of-00003.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3e0125b6268e71d418c96272816b306e8a8ecbeb69a9581073acf5bc4a717edf
 size 4999819336

 version https://git-lfs.github.com/spec/v1
+oid sha256:1154dc5968a3ed471182d5c8ee8642c6e4be50e1ea4582bd84b69b7177df4109
 size 4999819336

model-00003-of-00003.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bddfd7df2b88d22d9d6b171d69917106ac0155c7c988d14c73740f38f92e0c13
 size 4540516344

 version https://git-lfs.github.com/spec/v1
+oid sha256:eaac8c80e03387c80e37d2530add401bd8d45cf763c934f594c7221974f5dc16
 size 4540516344

train_results.json CHANGED Viewed

@@ -1,8 +1,8 @@
 {
     "epoch": 1.0,
-    "train_loss": 11837.785513657158,
-    "train_runtime": 19026.7097,
     "train_samples": 59478,
-    "train_samples_per_second": 3.126,
-    "train_steps_per_second": 0.049
 }

 {
     "epoch": 1.0,
+    "train_loss": 1.0336197533828193,
+    "train_runtime": 18653.3333,
     "train_samples": 59478,
+    "train_samples_per_second": 3.189,
+    "train_steps_per_second": 0.05
 }

trainer_state.json CHANGED Viewed

The diff for this file is too large to render. See raw diff

training_args.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b20079996080a10e131b97c0bf9ebda22a6be065a66ab7caf122e5398a91618b
 size 6136

 version https://git-lfs.github.com/spec/v1
+oid sha256:e7ca9d4cd41515dc4e111e1ebc0b1c08a568c970597ec8bd9e83a31985bda8c0
 size 6136