---
license: apache-2.0
tags:
  - gguf
  - ColBERT
  - modernbert
  - code-search
  - late-interaction
  - feature-extraction
base_model: lightonai/LateOn-Code-edge
---

# LateOn-Code-edge (GGUF Q8_0 + Projection)

Quantized GGUF conversion of [lightonai/LateOn-Code-edge](https://huggingface.co/lightonai/LateOn-Code-edge) for use with [litembeddings](https://github.com/alexandernicholson/litembeddings).

## Model Details

| Property | Value |
|----------|-------|
| Base model | [lightonai/LateOn-Code-edge](https://huggingface.co/lightonai/LateOn-Code-edge) |
| Architecture | ModernBERT (17M params) |
| Output dimensions | 48 (after projection) |
| Context length | 8,192 tokens |
| Quantization | Q8_0 |
| GGUF size | 19 MB |
| Projection | 256 → 48 (composed from two PyLate Dense layers: 256→512→48) |
| Use case | Fast, CPU-friendly code search with late interaction (ColBERT-style) |

## Variants

| Variant | Size | Quality |
|---------|------|---------|
| [f16](https://huggingface.co/embedme/lightonai-lateon-code-edge-f16) | 34 MB | Lossless — 100% top-1 agreement, 240/300 weighted |
| [f32](https://huggingface.co/embedme/lightonai-lateon-code-edge-f32) | 66 MB | Original precision (lossless) |
| **Q8_0** (this repo) | 19 MB | 79% weighted score, 96-100% top-1 agreement, 3.5× smaller |

## Files

| File | Size | Description |
|------|------|-------------|
| `lightonai-lateon-code-edge-Q8_0.gguf` | 19 MB | ModernBERT encoder in GGUF Q8_0 format |
| `lightonai-lateon-code-edge-Q8_0.projection` | 49 KB | Composed projection matrix (48×256, float32) |

## Usage with litembeddings

```sql
.load ./build/litembeddings

-- Load model with projection
SELECT lembed_model('lightonai-lateon-code-edge-Q8_0.gguf',
    '{"colbert_projection": "lightonai-lateon-code-edge-Q8_0.projection"}');

-- Generate token embeddings for code
SELECT lembed_tokens('async fn get_connection(pool: &Pool) -> Result<Connection>');

-- Code search with MaxSim
SELECT
    id, code,
    lembed_maxsim(lembed_tokens('database connection pool'), token_emb) AS score
FROM code_embeddings
ORDER BY score DESC
LIMIT 10;
```




## Quantization Quality Benchmark

Tested across 3 codebases (jq/C, Rails/Ruby, FastAPI/Python) with 150 questions total (15 easy + 20 medium + 15 hard per codebase). Weighted scoring: easy×1, medium×2, hard×3 = 100 points per codebase, 300 total.

### Aggregate Weighted Scores

| Variant | Weighted Score | Percentage |
|---------|---------------|------------|
| f32     | 240 / 300     | **80.0%**  |
| f16     | 240 / 300     | **80.0%**  |
| Q8_0    | 237 / 300     | **79.0%**  |

### Per-Corpus Scores

| Corpus  | f32     | f16     | Q8_0    |
|---------|---------|---------|---------|
| jq (C)  | 66/100  | 66/100  | 63/100  |
| Rails (Ruby) | 79/100 | 79/100 | 79/100 |
| FastAPI (Python) | 95/100 | 95/100 | 95/100 |

### Quantization Quality (Top-1 Agreement vs f32)

| Corpus  | f16    | Q8_0   |
|---------|--------|--------|
| jq      | 100.0% | 96.0%  |
| Rails   | 100.0% | 100.0% |
| FastAPI | 100.0% | 98.0%  |

### Key Findings

- **f16 is lossless** — identical weighted score (240/300) and 100% top-1 agreement across all codebases
- **Q8_0 loses only 1%** — 237/300 vs 240/300, drops only on hard queries in jq corpus
- **Q8_0 is fastest** — 2.5s avg query vs 3.4s f32 vs 13.4s f16 (CPU without FP16 hardware)
- Easy/medium questions show zero quality difference between all variants


## Conversion

Converted using litembeddings' ColBERT converter with PyLate projection support:

```bash
python scripts/convert_colbert_to_gguf.py lightonai/LateOn-Code-edge ./models \
    --name lightonai-lateon-code-edge-Q8_0 --quantize q8_0
```