Instructions to use ubergarm/DeepSeek-R1T-Chimera-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use ubergarm/DeepSeek-R1T-Chimera-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ubergarm/DeepSeek-R1T-Chimera-GGUF
# Run inference directly in the terminal:
llama cli -hf ubergarm/DeepSeek-R1T-Chimera-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ubergarm/DeepSeek-R1T-Chimera-GGUF
# Run inference directly in the terminal:
llama cli -hf ubergarm/DeepSeek-R1T-Chimera-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ubergarm/DeepSeek-R1T-Chimera-GGUF
# Run inference directly in the terminal:
./llama-cli -hf ubergarm/DeepSeek-R1T-Chimera-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ubergarm/DeepSeek-R1T-Chimera-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ubergarm/DeepSeek-R1T-Chimera-GGUF

Use Docker

docker model run hf.co/ubergarm/DeepSeek-R1T-Chimera-GGUF

LM Studio
Jan

vLLM

How to use ubergarm/DeepSeek-R1T-Chimera-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ubergarm/DeepSeek-R1T-Chimera-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ubergarm/DeepSeek-R1T-Chimera-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ubergarm/DeepSeek-R1T-Chimera-GGUF

Ollama
How to use ubergarm/DeepSeek-R1T-Chimera-GGUF with Ollama:
```
ollama run hf.co/ubergarm/DeepSeek-R1T-Chimera-GGUF
```

Unsloth Studio

How to use ubergarm/DeepSeek-R1T-Chimera-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ubergarm/DeepSeek-R1T-Chimera-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for ubergarm/DeepSeek-R1T-Chimera-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for ubergarm/DeepSeek-R1T-Chimera-GGUF to start chatting

Docker Model Runner
How to use ubergarm/DeepSeek-R1T-Chimera-GGUF with Docker Model Runner:
```
docker model run hf.co/ubergarm/DeepSeek-R1T-Chimera-GGUF
```

Lemonade

How to use ubergarm/DeepSeek-R1T-Chimera-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ubergarm/DeepSeek-R1T-Chimera-GGUF

Run and chat with the model

lemonade run user.DeepSeek-R1T-Chimera-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Atomic Chat

Any chance for IQ3_XXS/IQ3_XS or similar size?

by Panchovix - opened May 18, 2025

Discussion

Panchovix

May 18, 2025

Hi there, thanks for the quant! I was wondering if it was possible to get a quant of ~300GB size or so, as I have 344GB memory (between VRAM + RAM), so can't load IQ4 :(

For example, I can load https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF-UD/tree/main/UD-Q3_K_XL which is ~276GB.

gghfez

May 18, 2025

@Panchovix I think the guy who quantized that pruned coder variant of V3-0324 has done it?

DevQuasar/tngtech.DeepSeek-R1T-Chimera-GGUF

I haven't tested them myself because the I only have 248GB combined VRAM/RAM

ubergarm

Owner May 18, 2025

Yeah I realize this quant weighs in a little heavy at 339G which is a little tight even for 256GB RAM + 96GB VRAM.... Honestly I'm not sure it will finish uploading even... :fingers_crossed:

This one has a lot of iq4_ks layers which is pretty fast on CUDA, but yeah I don't have two RTX PRO 6000s myself either hah...

Panchovix

May 18, 2025

Oh I think I can't fit Q3_K_M (or near to the limit), but got Q3_K_S from here and it works.

https://huggingface.co/bullerwins/DeepSeek-R1T-Chimera-GGUF/tree/main/DeepSeek-R1T-Chimera-Q3_K_S

But I feel the quants of @ubergarm could have better quality with the imatrix.

ubergarm

Owner Jul 8, 2025

@Panchovix

I'm working on the updated one right now which might be a good size for you given ik's recent IQ3_KS is now available:

llm_load_print_meta: model type = 671B
llm_load_print_meta: model ftype = IQ3_KS - 3.1875 bpw
llm_load_print_meta: model params = 672.050 B
llm_load_print_meta: model size = 281.463 GiB (3.598 BPW)
llm_load_print_meta: repeating layers = 280.155 GiB (3.591 BPW, 670.196 B parameters)
llm_load_print_meta: general.name = DeepSeek TNG R1T2 Chimera

Final estimate: PPL = 3.3167 +/- 0.01789

Hope to have it up in the next 12 hours depending on how upload goes hah (this one will be much faster lol): https://huggingface.co/ubergarm/DeepSeek-TNG-R1T2-Chimera-GGUF

Panchovix

Jul 8, 2025

@ubergarm it looks amazing! For sure will try it as soon it is up.

ubergarm

Owner Jul 8, 2025

Okay, ready to go!

Panchovix

Jul 9, 2025

Downloading!

Panchovix

Jul 9, 2025

Posted some metrics on https://huggingface.co/ubergarm/DeepSeek-TNG-R1T2-Chimera-GGUF/discussions/2

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment