Measured on device (edge-compat, granite-4.0-h-350m-fp16): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · did not run: engine_create_failed (2026-08-24); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 9.5 tok/s · prefill 9 tok/s · TTFT 25.05 s (2026-09-05); Raspberry Pi 5 · LiteRT-LM 0.16.1 · CPU, 4 threads · decode 3.8 tok/s · prefill 8 tok/s · TTFT 30.43 s (2026-09-02). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/granite-4.0-h-350m-fp16/CARD.md

Measured on device (edge-compat, granite-4.0-h-350m-int8): Galaxy S26 · LiteRT-LM 0.16.0 · GPU · did not run: engine_create_failed (2026-08-24); Galaxy S26 · LiteRT-LM 0.16.0 · CPU · decode 11.5 tok/s · prefill 11 tok/s · TTFT 21.39 s (2026-09-05); Pixel 8a · LiteRT-LM 0.16.0 · CPU · decode 8.9 tok/s · prefill 8 tok/s · TTFT 28.01 s (2026-09-05). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/granite-4.0-h-350m-int8/CARD.md

granite-4.0-h-350m — LiteRT-LM

ibm-granite/granite-4.0-h-350m converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15 — the Mamba2 conv/SSM state buffers are bound through 0.15's generalized state binding (ExecutorMetadata); the 0.14 engine cannot bind them. Companion to litert-community/granite-4.0-h-1b.

Granite 4.0-h is IBM's hybrid architecture: 28 Mamba2 (selective-scan) blocks interleaved with 4 grouped-query attention blocks. The Mamba2 blocks carry per-layer conv + SSM recurrent state instead of a growing KV cache, so memory stays nearly flat with context length — only the 4 attention layers keep KV.

File	Recipe	Size
granite-4.0-h-350m_fp16.litertlm	fp16 weights (float-casting on linears + embedding; convs/SSM fp32)	723 MB
granite-4.0-h-350m_int8.litertlm	int8 dynamic weight-only (linears + embedding; convs/SSM float)	436 MB
granite-4.0-h-350m_int8_gpu.litertlm	the same int8 weights, re-exported with a rank-≤4 selective scan and fp32 activations declared — the GPU file	481 MB

For GPU, pick _int8_gpu. It carries the same int8 weights as _int8, re-exported with the Mamba2 selective scan written as rank-≤4 batched matmuls and with prefer_activation_type = "fp32" declared. The two older files are unchanged. It is 45 MB larger because the rewrite pads each short prefill signature up to a full 256-token chunk with a stored constant instead of a PAD op — that is the trade that lets the graph run on a GPU at all.

Pick fp16 for reference-faithful quality (greedy outputs match the PyTorch model token-for-token on our probes), int8 for size and speed. On phones we recommend int8: the CPU runtime unpacks fp16 weights to fp32 in RAM (~3.7 GB peak on iPhone vs ~2.1 GB for int8), so fp16 is best treated as the desktop/quality variant.

Correctness

The converted graph is verified against the PyTorch reference at the logits level: a float export matches the HF model exactly at every decode position (8-step teacher-forced comparison, per-position max|logit diff| ≤ 3.9e-4, correlation 1.000000, top-1 identical at all positions) — both the chunked prefill path and the single-step decode path (rolled conv window + SSM state recurrence) reproduce the reference math. Both files pass an 8-question sanity gate at 8/8, matching the HF reference, and a prompt-length sweep (12–200 tokens through the engine's real prefill chunking) generates cleanly — with one engine-reuse caveat for int8 noted below.

Unlike the HF generate() default, the bundle does not prepend a BOS token — this matches Granite's official chat template, and at 350M scale it matters: a prepended <|end_of_text|> measurably degrades greedy answers on this model.

Known limitation — engine state reuse, not quantization: if one Engine object is reused across many conversations whose prompts share a growing common prefix, a reply can stop after a few tokens; on the int8 file this appears at chat-templated lengths 33–37, and on _int8_gpu at 37–41 (the folded scan changes the graph, so the band moves with it; a fresh Engine is clean at every length tested). Re-measured 2026-08-11 on litert-lm 0.15.0: a fresh Engine per conversation is clean at every tested length, and so is a reused engine that has not built up that prefix history. The cause is the runtime carrying recurrent state across conversations rather than anything in the weights (LiteRT-LM#3165); the fp16 file tolerates it, the int8 file does not. Ordinary single-conversation use is unaffected — if you sweep prompts programmatically, create a fresh Engine per prompt. A fix landed on LiteRT-LM main on 2026-11-03 (e1f86b05, the internal CL that continued PR #3945). No release carries it yet (v0.19.1 does not); this note will be updated when one does.

fp16 file on the Mac GPU. The _fp16 file is CPU-only on the Mac GPU backend of litert-lm 0.18.0: the WebGPU delegate rejects DEQUANTIZE and EMBEDDING_LOOKUP (Empty quantization params) and rank-5 tensors, and no engine is created (measured 2026-10-12; the _int8_gpu file is the GPU file). Android GPU on 0.18.0: not tried; on litert-lm 0.16.0 the Galaxy S26 GPU also refused this fp16 file (device rows below).

Tool calls on litert-lm 0.18.0. Since the 2026-10-14 re-pack of the three files (metadata only: llm_model_type qwen2p5, and the chat template now renders tool results), the runtime parses the tool call and the tool result reaches the model; the earlier files returned the call as message text (<tool_call>{"name": "get_weather", "arguments": {"location": "Paris"}}</tool_call>). For _int8_gpu, the one GPU file of the three: the 2026-10-14 re-pack changes the model type, and with it the runtime's GPU executor defaults (every non-generic LiteRT-LM model type shares them: allow_src_quantized_fc_conv_ops on, OpenCL hint_waiting_for_completion off, kernel batch size unset); the phone GPU rows on this card were measured with the previous file and have not been re-run — the Mac GPU digit check passed on 0.18.0.

Usage

litert-lm run ./granite-4.0-h-350m_fp16.litertlm --prompt "What is the capital of France? Answer in one sentence."

The bundle carries the tokenizer and Granite's full chat template. Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks; the KV budget for the attention layers allows up to 4096 tokens. _fp16 and _int8 are CPU files: their scan graph materialises rank-5 and rank-6 intermediates and every mobile GPU delegate refuses tensors above rank 4. _int8_gpu is the same model with those contractions rewritten as batched matmuls, and it delegates in full on Adreno and on Apple GPUs.

Performance

Device	File	Prefill	Decode	TTFT	Peak RSS
MacBook Pro (M4 Max), CPU	int8	130 tok/s (256) · 130 tok/s (1024)	26.3 tok/s	2.01 s	—
MacBook Pro (M4 Max), CPU	fp16	114 tok/s (256)	22.0 tok/s	2.3 s	—
iPhone 17 Pro, CPU	int8	~10 tok/s (41-tok prompt)	18–21 tok/s	~4 s	~2.1 GB
iPhone 17 Pro, CPU	fp16	~6 tok/s (41-tok prompt)	7.8 tok/s	~10 s	~3.7 GB

(The int8 Mac row was re-measured with litert-lm benchmark 0.15.0, -d 256 --runs 3, on an otherwise idle machine; the fp16 Mac row and both iPhone rows are the original cold-start single runs. Decode is dominated by the selective-scan running as generic float ops; per-parameter it is slower than a plain transformer of this size.)

Galaxy S26 — GPU backend

granite-4.0-h-350m_int8_gpu.litertlm is the GPU file. The other two are CPU files, and the reason is in the graph rather than in the runtime: transformers writes the Mamba2 SSD contractions as broadcast-multiply-reduce, which materialises rank-5 and rank-6 intermediates, and every mobile GPU delegate refuses tensors above rank 4.

file	GPU backend	delegation	peak
granite-4.0-h-350m_int8_gpu.litertlm	runs	38313 / 38313 ops across 12 subgraphs on LiteRT GPU	2231 MB
granite-4.0-h-350m_int8.litertlm	does not run	109 / 3673 ops in the first subgraph reached	435 MB
granite-4.0-h-350m_fp16.litertlm	does not run	109 / 3807 ops in the first subgraph reached	375 MB

Measured on a Samsung Galaxy S26 (SM-S942Q / SM8850, Android 16) with litert_lm_advanced_main from litert-lm 0.16.0, --backend=gpu --sampler_backend=cpu, prompt What is the capital of France?. Peak is the process high-water mark (VmHWM) sampled during that same run. The two CPU files were gated 2026-08-24 and _int8_gpu 2026-08-27.

LiteRT-LM has no partial-GPU state. Anything short of full delegation is refused at engine creation, so a row above that says does not run is a refusal, not a slower path.

Speed on that phone, with the CPU control beside it

backend	prefill	decode	peak
GPU · LiteRT GPU (LITERT_CL)	486 tok/s	48.8 tok/s	2131 MB
CPU	306 tok/s	97.6 tok/s	1105 MB

Both rows n=2 and within ±3% (GPU 486.5 / 484.0 prefill and 48.8 / 47.9 decode; CPU 305.6 / 296.8 and 97.6 / 95.2), 205-token prompt, same handset and binary, CPU taken immediately after the GPU runs with the phone already warm. Engine init is 8.6–9.6 s on GPU and is not counted in TTFT.

The GPU is a prefill win and a decode loss here. Use the GPU backend when you feed long prompts and read short answers; use CPU when you generate a lot of text. Both backends run the same _int8_gpu file.

The two CPU files are also slow on this phone's CPU. Same binary, prompt and protocol, 2026-09-05, two cold runs each with the phone at thermal status ≤ 1: _int8 10.6 tok/s prefill / 10.7–11.5 decode at a 2.9 GB peak; _fp16 9.0–13.7 / 9.5–11.3 at 5.4–6.5 GB. The rank-5/6 scan graph that keeps them off the GPU also falls outside XNNPACK's fast path, so on Android _int8_gpu is the file for both backends (97.6 tok/s decode on this CPU, above). The desktop and iPhone CPU figures for _int8 / _fp16 in the table above are unaffected.

Pixel 8a (Tensor G3, Mali-G715, 8 GB), same binary and protocol, 2026-09-07. _int8_gpu delegates in full on this GPU as well (38313 / 38313 ops) and answers the probe, and the order is the one seen on the S26: GPU 180 tok/s prefill / 20.4–20.7 decode with a 37 s engine init and a 4.3 GB peak; CPU 103–111 / 40.1–40.7 with a 3.1 s init and a 1.1 GB peak (two cold runs per backend, thermal status 0 before every run). The _int8 file on the same CPU decodes 7.0–8.9 tok/s (2026-09-05). On an 8 GB-class phone run _int8_gpu on the CPU; this repo's litertlm_manifest.json recommends cpu for both Android device classes on these pairs.

iPhone 17 Pro

_int8_gpu runs on both backends: 7/8 on an 8-question on-device check over Metal and 7/8 on CPU, with byte-identical greedy output between them and matching per-question verdicts (the miss is a rhyme question, not arithmetic or fact). Engine init 13.6 s on Metal, 1.0 s on CPU. That check is a different harness from the desktop one above and its questions differ, so the two scores are not comparable to each other.

Reproduce

Conversion script, litert-torch patch (Mamba2 export-cache layer + decode-state continuation + prefill pad guard), and gate harnesses: https://github.com/john-rocky/hf-to-litertlm (granite_work/).

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

File	Prefill (tok/s)	Decode (tok/s)	TTFT	Peak RSS
granite-4.0-h-350m_fp16.litertlm	8.5 (8.4–8.5)	3.8 (3.7–3.8)	30.4 s	5.8 GB
granite-4.0-h-350m_int8.litertlm	8.8 (8.4–8.8)	5.6 (4.8–5.7)	29.1 s	2.6 GB
granite-4.0-h-350m_int8_gpu.litertlm	88.4 (88.2–88.5)	14.0 (14.0–14.2)	3.0 s	1.1 GB

Known runtime issues

External weights (since 2026-11-09). In granite-4.0-h-350m_int8.litertlm, granite-4.0-h-350m_int8_gpu.litertlm and granite-4.0-h-350m_fp16.litertlm, the weights sit in a TFLiteWeights section at the end of the file (LiteRT's external weights) instead of inside the model graph. On Android, the GPU path needs LiteRT-LM 0.19.0 or later: the GPU engine of the v0.18.0 and v0.18.1 Android builds crashes (SIGSEGV) on such a file (LiteRT-LM#3986). The earlier files with the weights inside the graph stay at revision 1290797. Some external weights of granite-4.0-h-350m_fp16.litertlm are float (FLOAT16 or FLOAT32), and LiteRT's WebGPU streamed loading takes only quantized (INT8, INT4, INT2) external weights: that file is for native runtimes, not for streamed loading in a browser.

From a conversation preface (the system message, initial messages, tools and extra context a conversation is created with) that has tools and no messages, LiteRT-LM renders these files' chat template with an empty message list, and the template reads messages[0] without checking that the list is non-empty, so the render fails with Failed to apply template: undefined value. This happens in two places: when the preface is prefilled at creation (prefillPrefaceOnInit = true in Kotlin, prefillPrefaceOnInit: true on the web, SetPrefillPrefaceOnInit(true) in C++; the default is false), where creating the conversation fails with that error, and in the preface render (renderPrefaceIntoString() in Kotlin and Swift, RenderPrefaceIntoString() in C++, litert_lm_conversation_render_preface_to_string in the C API). The same happens with extra context (such as enable_thinking) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2). Read in the source of LiteRT-LM 0.17.0, 0.18.1 and main; measured with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) through its C API on these files' template text. The default path, where the first message is rendered together with the preface, is not affected: the Python API and the CLI of 0.18.1 expose neither the flag nor the preface render, and Google AI Edge Gallery sets neither (its source as read on 2026-10-25). A system message or any initial message in the conversation config avoids it.