# 8× RTX PRO 6000 profile

Back to the [model card](../README.md). Speed tables for this profile are in [SPEED.md](SPEED.md#8x-profile-one-replica).

The same pack also serves on an 8-GPU PCIe node (8× RTX PRO 6000 Blackwell Server Edition, 96 GB each, no NVLink) with
`PROFILE=8x`. The node runs two independent replicas, one per 4 GPUs (GPUs 0-3 and 4-7, each pinned to its NUMA node). Each
replica uses tensor-parallel 4 for attention and expert-parallel 4 for the routed experts, so every expert is in VRAM and the
host-RAM expert machinery is off. Put a router or load balancer in front of the two ports. `recipe/serve/serve-8x.sh` starts
both; `PROFILE=8x REPLICA=0|1 bash recipe/serve/serve.sh` starts one.

What the profile adds to the 2x stack (all in `recipe/`, the settings in `serve/profile-8x.env`):

- **DSpark k=7 with the code-tuned head** (`drafter-code-k7/`, [DRAFTER.md](DRAFTER.md); `DS41_DRAFTER=released` serves the pack's head).
- **`xmoe_p8k`**, a build of the MoE kernel with a larger job table: the flat decode path covers up to 1,152 rows
  (`XMOE_FLAT_MAX_BSZ=1152`), so 128 sequences × 8 tokens per verify step stay on it. Without it, 128 code streams at k=7
  decode at 1,942 tok/s per replica instead of 4,469.
- **A chunked indexer** for decode (`DS41_IDX_CHUNK=1`, `patches/vllm/v41-indexer-chunked-decode.diff`, built into the
  image). The sparse indexer's decode logits are sized by the context window, not by the actual contexts: at 1,024 rows
  and a 1M window, a 2.56 GiB fp32 tensor allocated in one piece, which ran out of memory. The patch computes logits,
  candidate mask and top-k in chunks of whole sequences. It is exact: all 1,200 chunked calls we checked against the single
  call matched bit for bit.
- **8-bit all-reduce at TP4** (`kernels/ar8tp4`, `kernels/fp8arg`): two stages of the 2x stack's 8-bit pair exchange
  ({0,1}/{2,3}, then {0,2}/{1,3}), bit-identical on all four ranks, for prefill chunks and, captured in the CUDA graphs, for
  decode batches of 64 rows or more. Smaller decode batches use FlashInfer's PCIe IPC all-reduce.
- **Fewer collectives, exact:** the lm_head is gathered once before graph capture (no per-step logits gather), Engram rows
  are all-gathered as raw int4 plus scales, and prefill chunks of 1,024 rows or more use a replicated Engram `wkv`.
- **Length-aware indexer top-k** (`DS41_ROWTOPK=1`, the new default on both profiles).
- A 150 GB CPU KV offload tier per replica (more prefix-cache hits on long multi-turn agentic workloads; no decode effect).

**Limits, per replica**

| Setting | Value |
|---|---|
| Context window (`MAX_MODEL_LEN`) | 1,048,576 tokens |
| KV cache (`KV_MEM`) | 18 GiB fp8 per GPU: 4,538,321 tokens per replica, 4.33 full-length requests |
| Sequences (`MAX_NUM_SEQS`) | 128 |
| Prefill chunk (`MAX_BATCHED`) | 16,384 tokens |
| CUDA graph sizes | 1, 8, 16, 32, 64, 128, 256, 512, 1,024 (128 × (1 + 7)) |
| CPU KV offload (`KV_OFFLOAD_GB`) | 150 GB |
| VRAM in use after boot | 91,629 of 97,887 MiB per GPU |
| Host RAM | each replica's container is capped at 400 GiB (`CONTAINER_MEM`); the tested node had 1.4 TB |

Without the chunked indexer (`DS41_IDX_CHUNK=0`), use `MAX_MODEL_LEN=524288`: 128 sequences × k=7 at a 1M window run out
of memory there.

## Quick start (8x)

On an 8x RTX PRO 6000 node (8 sm_120 GPUs with 96 GB, two NUMA nodes; up to 400 GiB of host RAM per replica), with the
whole repo downloaded as in the [model card's quick start](../README.md#quick-start), including `drafter-code-k7/`:

```bash
bash docker/build.sh                      # the same image, now with the chunked-indexer patch
bash kernels/build.sh 8x                  # xmoe_p8k, xmoe_do_i8, fp8ar, fp8arg and rowtopk
API_KEY=change-me bash serve/serve-8x.sh  # replicas on :8000 (GPUs 0-3) and :8001 (GPUs 4-7); route between them
```

`recipe/serve/serve.sh` documents every setting, `recipe/serve/profile-8x.env` the 8x profile's, and `recipe/README.md`
explains the stack.

## Tested on

- The 8x profile was tested on one cloud VM (8× RTX PRO 6000 Blackwell Server Edition, AMD EPYC 9B45, two NUMA nodes), one
  replica at a time for the single-replica table (our development tree on that node, with the same image, kernels and
  hooks). The recipe's own build path was then checked on the same VM with both replicas serving: the same KV pool,
  speed within run-to-run noise, GSM8K-200 99.0% / HumanEval 97.0%, and the both-replicas table in [SPEED.md](SPEED.md#both-replicas-under-load-at-once).
- The 8x profile's k=7 is tuned for code: for prose-heavy bulk generation `SPEC_K=3` is 8-10% faster.