Molmo2 H3 Encoder + MolmoPoint Tracking

Community packages for two separate jobs: Molmo2-8B INT8 ConvRot as an experimental MiniMax H3 encoder, and MolmoPoint-8B point tracking for video segmentation. This is not an official Ai2 or MiniMax release.

The single weight file molmo2_8b_int8_convrot.safetensors totals 9.29 GB (decimal); 302 eligible linear layers are quantized. The complete package including the projector is about 9.65 GB.

The full model is included: language and vision weights, multimodal adapter, tokenizer, processor and original model code. Eligible linear layers use symmetric per-output-channel INT8 weights with regular block-Hadamard ConvRot rotations (blocks 256, 64 or 16) and dynamic per-token INT8 activations. Embeddings, normalization, biases and unsupported shapes retain BF16. The separate H3 projector retains its original precision. No original BF16 weight download is required.

Which model to use

Task Package Node Output
H3 text/visual conditioning Root Molmo2-8B-Minimax-H3 INT8 ConvRot + projections/ Auto Download & Load Molmo2 (H3 Encoder) H3-compatible CLIP
Locate and track subjects molmopoint_8b_int8_convrot/ Auto Download & Load MolmoPoint-8B Native point tokens and object IDs
Lower-memory point tracking on Blackwell molmopoint_8b_nvfp4_awq/ MolmoPoint loader, NVFP4 AWQ option Same point format

The MolmoPoint packages are separate models. They do not use the H3 projection or replace the H3 text encoder. The ComfyUI Molmo2-H3 nodes expose points for MolmoH3DrawPoints and for the BFS Video Segment Picker. In the picker, connect point_model, choose MolmoPoint-8B as propagation_method, select a subject, and use Apply fix. MolmoPoint supplies sparse, identity-aware points; SAM 3.1 turns selected points/masks into per-frame segmentation. Clear subject descriptions and correct anchors still matter, especially across camera cuts.

The INT8 ConvRot point package quantizes 144 language linear layers and retains the visual backbone and point prediction head in BF16. The NVFP4 AWQ point package uses ModelOpt activation-aware calibration on video prompts for the same language layers, with native NVFP4 weights and AWQ pre-quantization scales. Both need their included loaders; standard Transformers from_pretrained does not interpret these custom quantized weights.

Tracking evidence from Ai2

The original MolmoPoint-8B generates point tokens tied to visual patches and decodes them with pointing metadata, a constrained logits processor, and extract_video_points. Ai2 reports +5.7 J&F on Molmo2-Track versus Molmo 2 (8B), and state-of-the-art results on MeViS. These are upstream full-precision results, not scores for the INT8 or NVFP4 packages here. See the Ai2 technical write-up.

Ai2 MolmoPoint benchmark figure

Local validation of the quantized trackers

Both packages loaded and generated the same five tracking points with a consistent object ID on an eight-frame penguin clip. NVFP4 also passed inference through 144 ComfyUI native quantized layers on RTX PRO 5000 Blackwell. This is a smoke test, not a tracking accuracy benchmark. Approximate package sizes: INT8 9.8 GiB; NVFP4 AWQ 6.9 GiB. The NVFP4 loader adapts ModelOpt nibble order and block scales to ComfyUI native layout at load time. Execution uses the current ComfyUI process and model manager.

Current H3 projection limitation

The Molmo2 β†’ H3 projection can be used experimentally for fl2va and t2v. It sometimes works for ref2va, but identity and video quality are not yet consistent or as good as desired. The projection needs further task-specific training, particularly with multiple reference images and paired target videos. This limitation concerns the H3 projection; it is separate from MolmoPoint's pointing and SAM 3.1 segmentation roles.

Load the complete quantized model

Use Python with PyTorch supporting your NVIDIA CUDA GPU, transformers 4.57.1 or later (checked with 4.57.1, 5.17 and 5.22.1), accelerate, safetensors, pillow, numpy, einops and molmo_utils. Video processing may require the upstream video dependencies.

from huggingface_hub import snapshot_download
import importlib.util
from transformers import AutoProcessor

folder = snapshot_download("Alissonerdx/Molmo2-8B-Minimax-H3", ignore_patterns=["molmopoint_*/**"])
spec = importlib.util.spec_from_file_location("convrot_runtime", folder + "/convrot_runtime.py")
runtime = importlib.util.module_from_spec(spec)
spec.loader.exec_module(runtime)
model = runtime.load_model(folder, device="cuda")
processor = AutoProcessor.from_pretrained(folder, trust_remote_code=True)

This package uses its included loader; ordinary Transformers from_pretrained does not interpret this INT8 weight file.

Use in ComfyUI

Install the node from GitHub: alisson-anjos/ComfyUI-Molmo2-H3. Its Molmo2 as H3 encoder (projection, experimental) loader has no inputs: on first use it downloads this package into ComfyUI/models/text_encoders/Molmo2-8B-Minimax-H3/ (keeping files already present), finds the projector under projections/, and returns a CLIP. Connect that CLIP where an H3 workflow uses its text-encoder loader. Molmo runs inside the ComfyUI process, with ComfyUI's own Python, and is placed and offloaded by ComfyUI's model manager like other text encoders; no separate environment or worker process is needed. MiniMax H3 and task LoRAs are separate models.

Experimental H3 projection

molmo2-8b-h3-tap24-phaseb-pilot.safetensors maps layer-24 Molmo hidden states into H3 conditioning. The projector was calibrated with the original floating-point Molmo; quantization compatibility requires evaluation. It does not establish reliable photo-controlled selection of an arbitrary person in a multi-person video. No claim of preserved tracking accuracy or successful identity replacement is made.

Reproducibility and validation

Native tracking comparison: the short target description failed selection, while explicit shorts/gloves descriptions selected each requested performer in both BF16 and INT8 in this clip. This small check does not establish general tracking equivalence. See the explicit-prompt follow-up alongside the original results.

See provenance.json for original shard SHA256 verification, convrot_config.json for the exact module map, and validation.json for completed inference checks. Basic inference checks do not substitute for a full tracking or downstream video benchmark. Export and validation scripts are provided in quantization/.

The runtime is a compact eager PyTorch implementation, not a fused performance-optimized engine. Hardware INT8 GEMM is used on CUDA; no throughput improvement is promised. Source files are from allenai/Molmo2-8B under Apache 2.0. Modified distribution: quantized weights, standalone loader and experimental H3 integration. See LICENSE and NOTICE.

Single-file MolmoPoint packages

Each MolmoPoint variant is distributed as one .safetensors file. Configuration, tokenizer, preprocessing settings, chat template, and the quantization manifest are embedded in its metadata. The updated ComfyUI nodes download only that checkpoint and materialize small assets in an automatic cache. Model Python code is shipped with the nodes. Use the updated loader; older loaders expecting adjacent JSON/Python files are incompatible with this packaging.

For NVIDIA Blackwell GPUs (including RTX 50-series and RTX PRO Blackwell), choose NVFP4 AWQ with a compatible ComfyUI/PyTorch installation for the smaller checkpoint and native FP4 operations. INT8 ConvRot remains the default for broader compatibility. Both are managed by ComfyUI in the current process, with no additional worker thread.

Downloads last month
18,564
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Alissonerdx/Molmo2-8B-Minimax-H3

Finetuned
Qwen/Qwen3-8B
Finetuned
(6)
this model