File size: 3,314 Bytes
beb5dbc fb0c855 c588da6 29d5f36 c588da6 29d5f36 beb5dbc 0852509 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 |
# GGUF Header Edit Benchmark
Benchmark script for measuring how long it takes to **edit GGUF headers in-place** on Hugging Face with streaming blobs (xet) and create a **pull request** per file.
It fetches metadata, rebuilds the header with a small change, commits an edit (header slice only), and records timings to a CSV.
## Result from [benchmark.ts](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/benchmark.ts)
<div class="flex">
<div style="max-width: 600px">
<img src="https://huggingface.co/mishig/xet-gguf-edit-test/resolve/main/output.png">
> [!note]
> **Rule of thumb (linear fit):**
> time_minutes ≈ 0.36 × size_GB + 0.25
</div>
| Model Size (GB) | Time (minutes) |
|------------------|----------------|
| [0.5](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/500mb.gguf) | 0.28 |
| [1.0](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/1000mb.gguf) | 0.47 |
| [1.5](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/1500mb.gguf) | 0.24 |
| [2.0](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/2000mb.gguf) | 1.06 |
| [2.5](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/2500mb.gguf) | 1.29 |
| [3.0](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/3000mb.gguf) | 1.43 |
| [3.5](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/3500mb.gguf) | 1.59 |
| [4.0](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/4000mb.gguf) | 1.61 |
| [4.5](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/4500mb.gguf) | 1.82 |
| [5.0](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/5000mb.gguf) | 1.98 |
| [5.5](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/5500mb.gguf) | 2.10 |
| [6.0](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/6000mb.gguf) | 2.18 |
| [6.5](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/6500mb.gguf) | 2.14 |
| [7.0](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/7000mb.gguf) | 4.73 |
| [7.5](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/7500mb.gguf) | 5.04 |
| [8.0](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/8000mb.gguf) | 2.71 |
| [8.5](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/8500mb.gguf) | 2.75 |
| [9.0](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/9000mb.gguf) | 3.03 |
| [9.5](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/9500mb.gguf) | 3.11 |
| [10.0](https://huggingface.co/mishig/xet-gguf-edit-test/blob/main/10000mb.gguf) | 3.24 |
</div>
---
## ✨ What this does
For each `*.gguf` file in a model repo:
1. **Discover files** via the Hugging Face model tree API.
2. **Fetch GGUF + typed metadata** with `@huggingface/gguf`.
3. **Rebuild the header** using `buildGgufHeader` (preserving endianness, alignment, and tensor info range).
4. **Commit a slice edit** (header bytes only) using `commitIter` with `useXet: true` to avoid full re-uploads.
5. **Create a PR** titled `benchmark`.
6. **Record timing** (wall-clock) to `benchmark-results.csv`.
---
## 🧱 Requirements
- Node 18+
- A Hugging Face token with **read + write** on the target repo: `HF_TOKEN`
- NPM packages:
- `@huggingface/gguf`
- `@huggingface/hub`
- Network access to `huggingface.co`
---
## 🔧 Setup
```bash
npm i
npm run benchmark
```
|