Instructions to use tiiuae/Falcon-H1-0.5B-Instruct-GPTQ-Int4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tiiuae/Falcon-H1-0.5B-Instruct-GPTQ-Int4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="tiiuae/Falcon-H1-0.5B-Instruct-GPTQ-Int4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tiiuae/Falcon-H1-0.5B-Instruct-GPTQ-Int4")
model = AutoModelForCausalLM.from_pretrained("tiiuae/Falcon-H1-0.5B-Instruct-GPTQ-Int4", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tiiuae/Falcon-H1-0.5B-Instruct-GPTQ-Int4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tiiuae/Falcon-H1-0.5B-Instruct-GPTQ-Int4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiiuae/Falcon-H1-0.5B-Instruct-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/tiiuae/Falcon-H1-0.5B-Instruct-GPTQ-Int4

SGLang

How to use tiiuae/Falcon-H1-0.5B-Instruct-GPTQ-Int4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tiiuae/Falcon-H1-0.5B-Instruct-GPTQ-Int4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiiuae/Falcon-H1-0.5B-Instruct-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tiiuae/Falcon-H1-0.5B-Instruct-GPTQ-Int4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiiuae/Falcon-H1-0.5B-Instruct-GPTQ-Int4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use tiiuae/Falcon-H1-0.5B-Instruct-GPTQ-Int4 with Docker Model Runner:
```
docker model run hf.co/tiiuae/Falcon-H1-0.5B-Instruct-GPTQ-Int4
```

Falcon-H1-0.5B-Instruct-GPTQ-Int4 / quant_log.csv

HamzaYous89

Upload folder using huggingface_hub

6a19d8b verified over 1 year ago

Raw

History Blame Contribute Delete

5.71 kB

	layer,module,loss,samples,damp,time
	0,feed_forward.gate_proj,51.60443878,0.01000,4.074
	0,feed_forward.up_proj,103.42449951,0.01000,0.249
	0,feed_forward.down_proj,68.90666962,0.01000,0.498
	1,feed_forward.gate_proj,51.82374954,0.01000,0.249
	1,feed_forward.up_proj,87.18881226,0.01000,0.248
	1,feed_forward.down_proj,41.76483917,0.01000,0.494
	2,feed_forward.gate_proj,49.53681183,0.01000,0.249
	2,feed_forward.up_proj,78.44450378,0.01000,0.249
	2,feed_forward.down_proj,37.85085297,0.01000,0.494
	3,feed_forward.gate_proj,47.60984802,0.01000,0.249
	3,feed_forward.up_proj,74.64793396,0.01000,0.249
	3,feed_forward.down_proj,41.21762466,0.01000,0.496
	4,feed_forward.gate_proj,43.19258118,0.01000,0.249
	4,feed_forward.up_proj,69.82568359,0.01000,0.248
	4,feed_forward.down_proj,39.35258102,0.01000,0.494
	5,feed_forward.gate_proj,33.81882477,0.01000,0.249
	5,feed_forward.up_proj,56.53881836,0.01000,0.247
	5,feed_forward.down_proj,58.27473068,0.01000,0.494
	6,feed_forward.gate_proj,29.93852806,0.01000,0.249
	6,feed_forward.up_proj,48.31396484,0.01000,0.247
	6,feed_forward.down_proj,25.17117882,0.01000,0.493
	7,feed_forward.gate_proj,30.12131500,0.01000,0.248
	7,feed_forward.up_proj,50.13136292,0.01000,0.248
	7,feed_forward.down_proj,28.20563889,0.01000,0.493
	8,feed_forward.gate_proj,26.47483444,0.01000,0.248
	8,feed_forward.up_proj,43.44846725,0.01000,0.248
	8,feed_forward.down_proj,23.23403168,0.01000,0.495
	9,feed_forward.gate_proj,27.98056030,0.01000,0.248
	9,feed_forward.up_proj,48.01409149,0.01000,0.247
	9,feed_forward.down_proj,28.92861557,0.01000,0.493
	10,feed_forward.gate_proj,26.06900978,0.01000,0.248
	10,feed_forward.up_proj,43.19483185,0.01000,0.248
	10,feed_forward.down_proj,23.14476013,0.01000,0.494
	11,feed_forward.gate_proj,25.11003113,0.01000,0.247
	11,feed_forward.up_proj,40.92606354,0.01000,0.247
	11,feed_forward.down_proj,22.58832169,0.01000,0.494
	12,feed_forward.gate_proj,29.78376770,0.01000,0.249
	12,feed_forward.up_proj,45.79646301,0.01000,0.248
	12,feed_forward.down_proj,23.55985260,0.01000,0.493
	13,feed_forward.gate_proj,29.96887207,0.01000,0.248
	13,feed_forward.up_proj,48.22689056,0.01000,0.248
	13,feed_forward.down_proj,26.12702370,0.01000,0.494
	14,feed_forward.gate_proj,31.71737671,0.01000,0.248
	14,feed_forward.up_proj,50.67276001,0.01000,0.248
	14,feed_forward.down_proj,29.16994858,0.01000,0.493
	15,feed_forward.gate_proj,32.26173401,0.01000,0.248
	15,feed_forward.up_proj,51.04521179,0.01000,0.248
	15,feed_forward.down_proj,36.22430420,0.01000,0.495
	16,feed_forward.gate_proj,37.22601318,0.01000,0.248
	16,feed_forward.up_proj,57.67141342,0.01000,0.247
	16,feed_forward.down_proj,41.48391724,0.01000,0.492
	17,feed_forward.gate_proj,40.72809601,0.01000,0.247
	17,feed_forward.up_proj,63.36564636,0.01000,0.247
	17,feed_forward.down_proj,53.26121140,0.01000,0.495
	18,feed_forward.gate_proj,46.48646545,0.01000,0.249
	18,feed_forward.up_proj,73.20413208,0.01000,0.248
	18,feed_forward.down_proj,63.27213287,0.01000,0.494
	19,feed_forward.gate_proj,52.26978302,0.01000,0.249
	19,feed_forward.up_proj,82.46023560,0.01000,0.248
	19,feed_forward.down_proj,92.39147186,0.01000,0.494
	20,feed_forward.gate_proj,63.50267410,0.01000,0.248
	20,feed_forward.up_proj,99.62985992,0.01000,0.248
	20,feed_forward.down_proj,125.13226318,0.01000,0.494
	21,feed_forward.gate_proj,73.24076843,0.01000,0.248
	21,feed_forward.up_proj,112.58317566,0.01000,0.248
	21,feed_forward.down_proj,159.46179199,0.01000,0.494
	22,feed_forward.gate_proj,83.06292725,0.01000,0.248
	22,feed_forward.up_proj,123.96360779,0.01000,0.248
	22,feed_forward.down_proj,212.55529785,0.01000,0.493
	23,feed_forward.gate_proj,91.74035645,0.01000,0.248
	23,feed_forward.up_proj,147.29116821,0.01000,0.248
	23,feed_forward.down_proj,290.64556885,0.01000,0.495
	24,feed_forward.gate_proj,99.30014801,0.01000,0.248
	24,feed_forward.up_proj,163.34786987,0.01000,0.248
	24,feed_forward.down_proj,380.25775146,0.01000,0.492
	25,feed_forward.gate_proj,110.84506226,0.01000,0.248
	25,feed_forward.up_proj,182.65046692,0.01000,0.248
	25,feed_forward.down_proj,392.83627319,0.01000,0.495
	26,feed_forward.gate_proj,127.58309937,0.01000,0.248
	26,feed_forward.up_proj,212.37713623,0.01000,0.247
	26,feed_forward.down_proj,421.52368164,0.01000,0.492
	27,feed_forward.gate_proj,140.92987061,0.01000,0.249
	27,feed_forward.up_proj,235.86401367,0.01000,0.248
	27,feed_forward.down_proj,514.17376709,0.01000,0.494
	28,feed_forward.gate_proj,153.60296631,0.01000,0.249
	28,feed_forward.up_proj,257.42593384,0.01000,0.248
	28,feed_forward.down_proj,638.19470215,0.01000,0.494
	29,feed_forward.gate_proj,162.86459351,0.01000,0.249
	29,feed_forward.up_proj,279.35134888,0.01000,0.248
	29,feed_forward.down_proj,741.01318359,0.01000,0.495
	30,feed_forward.gate_proj,177.63189697,0.01000,0.249
	30,feed_forward.up_proj,304.95382690,0.01000,0.247
	30,feed_forward.down_proj,882.69201660,0.01000,0.493
	31,feed_forward.gate_proj,191.00192261,0.01000,0.249
	31,feed_forward.up_proj,326.85534668,0.01000,0.249
	31,feed_forward.down_proj,1015.30139160,0.01000,0.493
	32,feed_forward.gate_proj,194.33041382,0.01000,0.248
	32,feed_forward.up_proj,333.57369995,0.01000,0.248
	32,feed_forward.down_proj,1148.42822266,0.01000,0.493
	33,feed_forward.gate_proj,217.31362915,0.01000,0.249
	33,feed_forward.up_proj,358.10684204,0.01000,0.248
	33,feed_forward.down_proj,1245.03076172,0.01000,0.493
	34,feed_forward.gate_proj,242.25756836,0.01000,0.247
	34,feed_forward.up_proj,369.35974121,0.01000,0.247
	34,feed_forward.down_proj,1414.03002930,0.01000,0.494
	35,feed_forward.gate_proj,329.11608887,0.01000,0.248
	35,feed_forward.up_proj,436.05023193,0.01000,0.247
	35,feed_forward.down_proj,1798.30822754,0.01000,0.494