Instructions to use ChrisGVE/gte-modernbert-base-Q8_0-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use ChrisGVE/gte-modernbert-base-Q8_0-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible embeddings server: llama serve -hf ChrisGVE/gte-modernbert-base-Q8_0-GGUF:Q8_0 --embeddings
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible embeddings server: llama serve -hf ChrisGVE/gte-modernbert-base-Q8_0-GGUF:Q8_0 --embeddings
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible embeddings server: ./build/bin/llama serve -hf ChrisGVE/gte-modernbert-base-Q8_0-GGUF:Q8_0 --embeddings
- Llama new
- LM Studio
- Jan
- Ollama
How to use ChrisGVE/gte-modernbert-base-Q8_0-GGUF with Ollama:
ollama run hf.co/ChrisGVE/gte-modernbert-base-Q8_0-GGUF:Q8_0
- Unsloth Desktop
- Docker Model Runner
How to use ChrisGVE/gte-modernbert-base-Q8_0-GGUF with Docker Model Runner:
docker model run hf.co/ChrisGVE/gte-modernbert-base-Q8_0-GGUF:Q8_0
- Lemonade
How to use ChrisGVE/gte-modernbert-base-Q8_0-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull ChrisGVE/gte-modernbert-base-Q8_0-GGUF:Q8_0
Run and chat with the model
lemonade run user.gte-modernbert-base-Q8_0-GGUF-Q8_0
List all available models
lemonade list
- Atomic Chat
gte-modernbert-base β GGUF Q8_0
Alibaba-NLP/gte-modernbert-base converted to GGUF at
Q8_0 for llama.cpp's llama-server --embeddings, with the ModernBERT architecture recorded correctly.
| File | gte-modernbert-base-Q8_0.gguf (160 MB) |
| SHA-256 | ddae471b78b3380e3e0db99b14ca588428b6592d3df9f6b3d571359087ae8c1f |
| Source revision | e7f32e3c00f91d699e8c43b53106206bcc72bb22 (model.safetensors sha256 3e85899dβ¦) |
| Architecture | modern-bert |
| Context | 8192 tokens |
| Pooling | CLS (stored in the file, as the model card prescribes) |
| Dimensions | 768 |
llama-server -m gte-modernbert-base-Q8_0.gguf --embeddings --port 8080
curl -s localhost:8080/v1/embeddings -H 'Content-Type: application/json' -d '{"input":["def f(): pass"]}'
Why this conversion
A GGUF of this model we had downloaded elsewhere was labelled bert with no bert.context_length, and
llama.cpp refused it. This one was made with llama.cpp's own convert_hf_to_gguf.py (2026-10-01, --outtype q8_0), which maps the model to llama.cpp's modern-bert architecture; it needs a llama.cpp build that has that
architecture.
Verification
Embedded short code and prose inputs with this file on llama-server and with the original model in PyTorch (CLS vector of the last hidden state): cosine similarity 0.9997β0.9998.
The weights and their licence (Apache-2.0) are Alibaba-NLP's.
- Downloads last month
- 40
8-bit
Model tree for ChrisGVE/gte-modernbert-base-Q8_0-GGUF
Base model
answerdotai/ModernBERT-base