Instructions to use bartowski/gemma-2-9b-it-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use bartowski/gemma-2-9b-it-GGUF with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="bartowski/gemma-2-9b-it-GGUF")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("bartowski/gemma-2-9b-it-GGUF", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use bartowski/gemma-2-9b-it-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf bartowski/gemma-2-9b-it-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf bartowski/gemma-2-9b-it-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf bartowski/gemma-2-9b-it-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf bartowski/gemma-2-9b-it-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf bartowski/gemma-2-9b-it-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf bartowski/gemma-2-9b-it-GGUF:Q4_K_M
Use Docker
docker model run hf.co/bartowski/gemma-2-9b-it-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use bartowski/gemma-2-9b-it-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "bartowski/gemma-2-9b-it-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "bartowski/gemma-2-9b-it-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/bartowski/gemma-2-9b-it-GGUF:Q4_K_M
- SGLang
How to use bartowski/gemma-2-9b-it-GGUF with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "bartowski/gemma-2-9b-it-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "bartowski/gemma-2-9b-it-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "bartowski/gemma-2-9b-it-GGUF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "bartowski/gemma-2-9b-it-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Ollama
How to use bartowski/gemma-2-9b-it-GGUF with Ollama:
ollama run hf.co/bartowski/gemma-2-9b-it-GGUF:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use bartowski/gemma-2-9b-it-GGUF with Docker Model Runner:
docker model run hf.co/bartowski/gemma-2-9b-it-GGUF:Q4_K_M
- Lemonade
How to use bartowski/gemma-2-9b-it-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull bartowski/gemma-2-9b-it-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.gemma-2-9b-it-GGUF-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Llamacpp quants
Browse files- README.md +4 -3
- gemma-2-9b-it-IQ2_M.gguf +2 -2
- gemma-2-9b-it-IQ2_S.gguf +2 -2
- gemma-2-9b-it-IQ2_XS.gguf +2 -2
- gemma-2-9b-it-IQ3_M.gguf +2 -2
- gemma-2-9b-it-IQ3_XS.gguf +2 -2
- gemma-2-9b-it-IQ3_XXS.gguf +2 -2
- gemma-2-9b-it-IQ4_XS.gguf +2 -2
- gemma-2-9b-it-Q2_K.gguf +2 -2
- gemma-2-9b-it-Q2_K_L.gguf +2 -2
- gemma-2-9b-it-Q3_K_L.gguf +2 -2
- gemma-2-9b-it-Q3_K_M.gguf +2 -2
- gemma-2-9b-it-Q3_K_S.gguf +2 -2
- gemma-2-9b-it-Q3_K_XL.gguf +2 -2
- gemma-2-9b-it-Q4_K_L.gguf +2 -2
- gemma-2-9b-it-Q4_K_M.gguf +2 -2
- gemma-2-9b-it-Q4_K_S.gguf +2 -2
- gemma-2-9b-it-Q5_K_L.gguf +2 -2
- gemma-2-9b-it-Q5_K_M.gguf +2 -2
- gemma-2-9b-it-Q5_K_S.gguf +2 -2
- gemma-2-9b-it-Q6_K.gguf +2 -2
- gemma-2-9b-it-Q6_K_L.gguf +2 -2
- gemma-2-9b-it-Q8_0.gguf +2 -2
- gemma-2-9b-it-Q8_0_L.gguf +2 -2
- gemma-2-9b-it-f32.gguf +2 -2
- gemma-2-9b-it.imatrix +1 -1
README.md
CHANGED
|
@@ -11,12 +11,11 @@ extra_gated_button_content: Acknowledge license
|
|
| 11 |
tags:
|
| 12 |
- conversational
|
| 13 |
quantized_by: bartowski
|
| 14 |
-
base_model: google/gemma-2-9b-it
|
| 15 |
---
|
| 16 |
|
| 17 |
## Llamacpp imatrix Quantizations of gemma-2-9b-it
|
| 18 |
|
| 19 |
-
Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/
|
| 20 |
|
| 21 |
Original model: https://huggingface.co/google/gemma-2-9b-it
|
| 22 |
|
|
@@ -25,9 +24,11 @@ All quants made using imatrix option with dataset from [here](https://gist.githu
|
|
| 25 |
## Prompt format
|
| 26 |
|
| 27 |
```
|
| 28 |
-
<start_of_turn>user
|
| 29 |
{prompt}<end_of_turn>
|
| 30 |
<start_of_turn>model
|
|
|
|
|
|
|
| 31 |
|
| 32 |
```
|
| 33 |
|
|
|
|
| 11 |
tags:
|
| 12 |
- conversational
|
| 13 |
quantized_by: bartowski
|
|
|
|
| 14 |
---
|
| 15 |
|
| 16 |
## Llamacpp imatrix Quantizations of gemma-2-9b-it
|
| 17 |
|
| 18 |
+
Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b3274">b3274</a> for quantization.
|
| 19 |
|
| 20 |
Original model: https://huggingface.co/google/gemma-2-9b-it
|
| 21 |
|
|
|
|
| 24 |
## Prompt format
|
| 25 |
|
| 26 |
```
|
| 27 |
+
<bos><start_of_turn>user
|
| 28 |
{prompt}<end_of_turn>
|
| 29 |
<start_of_turn>model
|
| 30 |
+
<end_of_turn>
|
| 31 |
+
<start_of_turn>model
|
| 32 |
|
| 33 |
```
|
| 34 |
|
gemma-2-9b-it-IQ2_M.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3f9886a5f8d9d8e2c1c3d5a3e5da9f96e0ce6f4fdc0480a960c126fa5f18fb36
|
| 3 |
+
size 3434669952
|
gemma-2-9b-it-IQ2_S.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:13cd744736288aa0285b14870800389831c363ba044593d13705a47d7d1c8f88
|
| 3 |
+
size 3211487104
|
gemma-2-9b-it-IQ2_XS.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:fbdeb1d1423204096b4dc28e67d5d7e2ec11b8913b51c9ec02eb01b65b56059a
|
| 3 |
+
size 3067381632
|
gemma-2-9b-it-IQ3_M.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:58211a13b5bcb5982f6d43d6fb2e91608330a6da15f0afc4801c52198c2a3d90
|
| 3 |
+
size 4494616448
|
gemma-2-9b-it-IQ3_XS.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:90e57856b6242c7c0d4d58c2d299055e6d15db0248394e44b63429319b1d7c2c
|
| 3 |
+
size 4144990080
|
gemma-2-9b-it-IQ3_XXS.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ad2b0263ebff9cb8d22986d7a1d462ced0c83a95636268282087257504f91f9f
|
| 3 |
+
size 3796739968
|
gemma-2-9b-it-IQ4_XS.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5db6f9a4f4c63b2d526106ecf3e83892fe4451d85a4fec5066e5136eaf944b76
|
| 3 |
+
size 5183031168
|
gemma-2-9b-it-Q2_K.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e4fb2ad3d575c3c54f6ba4d8eb4bb0ba4864b37b1e561139bf89068f69439a81
|
| 3 |
+
size 3805398912
|
gemma-2-9b-it-Q2_K_L.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7723971dbbfe877c397732816422b2433cb5fff3425f43b4cb114860edff5737
|
| 3 |
+
size 4887766912
|
gemma-2-9b-it-Q3_K_L.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9aeb70915c136c0f4a77055198629d958d87ba771589364c0bdea126c1584360
|
| 3 |
+
size 5132453760
|
gemma-2-9b-it-Q3_K_M.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:5e824f6f0c9cde71d27e6ca2893c26caff96482adb3d3b48560bc8d9dd84a6d1
|
| 3 |
+
size 4761782144
|
gemma-2-9b-it-Q3_K_S.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:06cc0f8133631e4ba203214cbca28754646e4af4bfc89d6cb6503259e5dac5ef
|
| 3 |
+
size 4337665920
|
gemma-2-9b-it-Q3_K_XL.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:466fed80bc4a3ede2a1eaedfa4eae1fd2c74229f09fb7debc0e18df3e8a99f73
|
| 3 |
+
size 6214821760
|
gemma-2-9b-it-Q4_K_L.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e88ec9e7009e4448231855119efd408e8e296b7ff2ee2fb8893fe530b49acc17
|
| 3 |
+
size 6843426688
|
gemma-2-9b-it-Q4_K_M.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:69c8e8f0cbd77d37c2861f72e6c11ae33a0da5f5f097f93e7ba90562269c8248
|
| 3 |
+
size 5761058688
|
gemma-2-9b-it-Q4_K_S.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9af9c9340359c12b28a15fbb6295fc44cbe45e72cd147b39a459190697dafb35
|
| 3 |
+
size 5478926208
|
gemma-2-9b-it-Q5_K_L.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:65eb6343a9a249ee63a9feb05bfc7949107e9e90a2bab03b61be5d968cc397e1
|
| 3 |
+
size 7729735552
|
gemma-2-9b-it-Q5_K_M.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:69acf0e31f3e7c49771af9d443f3d98eb1e0cacaed473b523bd96ddb425cb502
|
| 3 |
+
size 6647367552
|
gemma-2-9b-it-Q5_K_S.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c8e8ce55de8f7db08a9de158c3448845b077fb5eb417d4991ed903645564a0cc
|
| 3 |
+
size 6483593088
|
gemma-2-9b-it-Q6_K.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:19d6c53fd0a2b3ad0f550ec880f39aa2f1904e377456e84cff735c594e91b169
|
| 3 |
+
size 7589070720
|
gemma-2-9b-it-Q6_K_L.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0c044c1eff0fcaa4b45f4414c9317e56469492932ef20667924446e464312c8d
|
| 3 |
+
size 8671438720
|
gemma-2-9b-it-Q8_0.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:924a339cd17640b1c54fb648835bc33a441eb77231ef74e417b47c839ad2c4dd
|
| 3 |
+
size 9827149696
|
gemma-2-9b-it-Q8_0_L.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f17c9e0f73a61a85cfaa621a249f9e83ec14cce5f33c7e3b2362881f6a35ec17
|
| 3 |
+
size 10687309696
|
gemma-2-9b-it-f32.gguf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:da30533a73c2d680e7cd2de47d6ca57744eda1a6060e4a1cc7d476cccfb992dd
|
| 3 |
+
size 36972881536
|
gemma-2-9b-it.imatrix
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
size 6116901
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e09ef8351870c9f3778d42176021611d5b8ca52b393352b2a55dfd9c8ab52de3
|
| 3 |
size 6116901
|