litert-community/Nemotron-H-4B-Instruct-128K

LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).

Nemotron-H-4B-Instruct — LiteRT-LM

nvidia/Nemotron-H-4B-Instruct-128K converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15. To our knowledge this is the first Nemotron-H in LiteRT form — a three-kind hybrid: 24 Mamba2 selective-scan layers + 24 plain MLP layers + 4 grouped-query attention layers (52 layers total), running fully delegated on the GPU.

The mamba layers carry constant-size conv + SSM recurrent state, only the 4 attention layers keep KV (4096-token budget here), and the MLP layers carry no state at all — memory stays nearly flat with context length.

File	Recipe	Size
Nemotron-H-4B-Instruct-128K_int8.litertlm	int8 dynamic on linears + embedding (convs and the scan stay float); fp32 activations declared for GPU	4.67 GB

Correctness

• Logits parity vs PyTorch: the float export matches the HF model teacher-forced across 8 decode positions — max|logit diff| 5.8e-05, correlation 1.000000, top-1 and top-5 identical at every position.
• 8-question sanity gate: 8/8 on CPU and 8/8 on GPU (Mac, litert-lm 0.16.0), answers near-verbatim identical across backends.
• Prompt-length robustness: hermetic prefill-chunk sweep (fresh engine per length) — all lengths clean at the ship shape.
• iPhone 17 Pro (Metal): runs on GPU and CPU with identical answers on the composite probe (7/8-class; one arithmetic near-miss appears identically on BOTH backends — a quantization-level composite-prompt effect, not a backend bug).

Usage

litert-lm run ./Nemotron-H-4B-Instruct-128K_int8.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./Nemotron-H-4B-Instruct-128K_int8.litertlm --backend gpu --cache no --prompt "..."

Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks. The bundle carries the tokenizer and the stock Nemotron-H chat template.

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:

Backend	Prefill (256)	Decode	TTFT
GPU	724 tok/s	75.0 tok/s	0.37 s
CPU	99 tok/s	20.1 tok/s	2.63 s

On device (cold start, single runs, 131-token composite prompt, quality harness):

Device	Backend	Prefill	Decode	TTFT	Peak memory
iPhone 17 Pro	GPU (Metal)	59.8 tok/s	10.7 tok/s	2.48 s	4.02 GB

Honest notes:

• Where GPU execution is verified. macOS (Metal), iPhone 17 Pro (Metal) and Pixel 8a (Arm Mali, OpenCL). Qualcomm Adreno is now measured too, and this bundle does not run there — see the Galaxy S26 section below. On a Snapdragon phone, use the CPU backend.
• On low-end Android the GPU buys prefill and time-to-first-token, not decode (decode is memory-bandwidth-bound there; the CPU path reads int8 weights while the fp32-activation GPU path reads expanded ones). Pick the backend for your workload: long prompts favour the GPU, long answers favour the CPU. On Apple hardware the GPU wins across the board.
• A 4B does not fit an 8 GB Android phone: on a Pixel 8a, engine creation aborts on both backends (4.67 GB weights plus the multi-signature arena exceed the ~3.8 GB available). Android needs higher-RAM devices; the iPhone rows above used the increased-memory entitlement.
• GPU runs with fp32 activations (declared in the bundle).
• On composite many-question prompts, int8 costs borderline arithmetic items (identically on every backend). Per-question use is clean (8/8).

Galaxy S26 — GPU backend

Nemotron-H-4B-Instruct-128K_int8.litertlm delegates fully — 3246 of 3246 operations — and then fails during the run, producing no usable output.

file	GPU backend	delegation	peak
Nemotron-H-4B-Instruct-128K_int8.litertlm	does not run	3246 / 3246 ops on LiteRT GPU	1456 MB

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. Gated 2026-08-24.

GPU wiring, including the Gallery import toggle: GPU guide.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm nemotron_h_work/):

• Folded selective scan: the Mamba2 scan is re-expressed as batched matmuls with chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no BROADCAST_TO, no int64 index math) — this is what makes the graph fully delegable on GPU.
• Cache-less MLP layer type: NemotronH interleaves plain MLP blocks; a dedicated no-state cache layer keeps absolute layer indexing without phantom KV buffers (24 of them would otherwise be allocated and paid for in RAM).
• Min-only dt clamp handled exactly: NemotronH clamps dt at time_step_min with no upper bound; engine pad steps are forced to exact identity (dt = 0 post-clamp) so partially-filled prefill chunks cannot decay the state.
• Class-registry patching: NemotronH constructs its mixers from an import-time class registry — the export patch swaps the registry entry (module-attribute swapping alone silently exports the unrewritten reference scan; a loud guard now prevents that).
• Quantization: post-hoc dynamic int8 over linears + embedding only; convs and the scan stay float.

2026-08-28 — start_token fix (weights unchanged)

The LiteRT-LM engine prepends the metadata start_token to every prompt, but this model's reference prompt has no leading BOS at all — the bundle was feeding an extra <s> the model was never trained on. The start token has been removed.

Metadata-only change: every section of the bundle except the LlmMetadata block is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the tokenizer are unchanged and the speed and memory numbers on this card still describe exactly this file — only the file's own sha256 differs. What changed is the input: the token stream the model reads for a given conversation can differ from the previous file's, and it now matches this model's own reference chat stream. Greedy decoding can turn on a single token, so an individual answer can differ from the previous file in either direction. Unless a row says otherwise, the accuracy figures on this card were measured on the previous file and have not been re-measured on this one. If you downloaded before 2026-08-28, re-download.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

File	Prefill (tok/s)	Decode (tok/s)	TTFT	Peak RSS
Nemotron-H-4B-Instruct-128K_int8.litertlm	20.9 (19.4–20.9)	2.0 (1.9–2.0)	12.8 s	5.7 GB