Isaac 0.5 for SO-100 and SO-101

We fine-tuned Isaac 0.5 on the MolmoAct2 SO-100 and SO-101 demonstrations. The policy reads RGB images, a language instruction, and the robot鈥檚 joint state. It predicts absolute joint and gripper targets for one arm.

This checkpoint was saved at step 125,000 of fine-tuning on 32 GPUs with a global batch of 256 sequences and a sequence length of 2,048 tokens.

Quick Links

Isaac 0.5 base model 路 MolmoAct2 SO-100/SO-101 dataset 路 Code 路 Blog

Get started

Clone Isaac with its pinned LeRobot submodule. Isaac on GitHub

git clone --branch main --recurse-submodules https://github.com/perceptron-ai-inc/isaac.git
cd isaac/lerobot
uv venv --python 3.12
source .venv/bin/activate
uv pip install -c constraints/perceptron_isaac-cu128.txt \
  --index https://download.pytorch.org/whl/cu128 --index-strategy unsafe-best-match \
  '.[perceptron_isaac,training]'

hf download PerceptronAI/Isaac-0.5-SO100_101 --local-dir ./Isaac-0.5-SO100_101

Run inference

Wrist RGB Side RGB
Wrist RGB Side RGB

This recording uses wrist and side cameras. The sample feeds the wrist image into the policy鈥檚 top slot and includes joint values converted to the checkpoint鈥檚 training frame.

import json
from pathlib import Path

import torch
from lerobot.policies.factory import make_pre_post_processors
from lerobot.policies.perceptron_isaac import PerceptronIsaacPolicy
from torchvision.io import decode_image

snapshot = Path("./Isaac-0.5-SO100_101")
assets = snapshot / "assets"
sample = json.loads((assets / "sample_observation.json").read_text())
observation = {
    key: decode_image(str(assets / file), mode="RGB").float().div(255).unsqueeze(0)
    for key, file in sample["cameras"].items()
}
observation["observation.state"] = torch.tensor([sample["state"]], dtype=torch.float32)
observation["task"] = [sample["task"]]

path = str(snapshot / "recorded_policy")
policy = PerceptronIsaacPolicy.from_pretrained(path)
preprocess, postprocess = make_pre_post_processors(policy.config, pretrained_path=path)
with torch.inference_mode():
    actions = postprocess(policy.predict_action_chunk(preprocess(observation)))

The policy returns a 60 脳 6 action chunk for each observation.

Fine-tune

From isaac/lerobot, set DATASET_REPO_ID to a compatible LeRobot dataset. This example uses two GPUs and CPU offload and saves a checkpoint at the end of training. Isaac fine-tuning guide

STEPS=10000
torchrun --standalone --nproc_per_node=2 ../scripts/finetune.py \
  --policy.path="./Isaac-0.5-SO100_101/recorded_policy" --dataset.repo_id="$DATASET_REPO_ID" \
  --steps="$STEPS" --policy.max_train_steps="$STEPS" --batch_size=1 \
  --dataset.video_backend=pyav --parallelism.dp_shard=-1 \
  --accelerator.mixed_precision=bf16 --accelerator.fsdp.cpu_offload=true \
  --accelerator.fsdp.reduce_dtype=float32 --policy.push_to_hub=false \
  --env_eval_freq=0 --eval_steps=0
Downloads last month
11
Video Preview
loading