Instructions to use PerceptronAI/Isaac-0.5-SO100_101 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LeRobot
How to use PerceptronAI/Isaac-0.5-SO100_101 with LeRobot:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Isaac 0.5 for SO-100 and SO-101
We fine-tuned Isaac 0.5 on the MolmoAct2 SO-100 and SO-101 demonstrations. The policy reads RGB images, a language instruction, and the robot鈥檚 joint state. It predicts absolute joint and gripper targets for one arm.
This checkpoint was saved at step 125,000 of fine-tuning on 32 GPUs with a global batch of 256 sequences and a sequence length of 2,048 tokens.
Quick Links
Isaac 0.5 base model 路 MolmoAct2 SO-100/SO-101 dataset 路 Code 路 Blog
Get started
Clone Isaac with its pinned LeRobot submodule. Isaac on GitHub
git clone --branch main --recurse-submodules https://github.com/perceptron-ai-inc/isaac.git
cd isaac/lerobot
uv venv --python 3.12
source .venv/bin/activate
uv pip install -c constraints/perceptron_isaac-cu128.txt \
--index https://download.pytorch.org/whl/cu128 --index-strategy unsafe-best-match \
'.[perceptron_isaac,training]'
hf download PerceptronAI/Isaac-0.5-SO100_101 --local-dir ./Isaac-0.5-SO100_101
Run inference
This recording uses wrist and side cameras. The sample feeds the wrist image into the policy鈥檚 top slot and includes joint values converted to the checkpoint鈥檚 training frame.
import json
from pathlib import Path
import torch
from lerobot.policies.factory import make_pre_post_processors
from lerobot.policies.perceptron_isaac import PerceptronIsaacPolicy
from torchvision.io import decode_image
snapshot = Path("./Isaac-0.5-SO100_101")
assets = snapshot / "assets"
sample = json.loads((assets / "sample_observation.json").read_text())
observation = {
key: decode_image(str(assets / file), mode="RGB").float().div(255).unsqueeze(0)
for key, file in sample["cameras"].items()
}
observation["observation.state"] = torch.tensor([sample["state"]], dtype=torch.float32)
observation["task"] = [sample["task"]]
path = str(snapshot / "recorded_policy")
policy = PerceptronIsaacPolicy.from_pretrained(path)
preprocess, postprocess = make_pre_post_processors(policy.config, pretrained_path=path)
with torch.inference_mode():
actions = postprocess(policy.predict_action_chunk(preprocess(observation)))
The policy returns a 60 脳 6 action chunk for each observation.
Fine-tune
From isaac/lerobot, set DATASET_REPO_ID to a compatible LeRobot dataset. This example uses two GPUs and CPU offload and saves a checkpoint at the end of training. Isaac fine-tuning guide
STEPS=10000
torchrun --standalone --nproc_per_node=2 ../scripts/finetune.py \
--policy.path="./Isaac-0.5-SO100_101/recorded_policy" --dataset.repo_id="$DATASET_REPO_ID" \
--steps="$STEPS" --policy.max_train_steps="$STEPS" --batch_size=1 \
--dataset.video_backend=pyav --parallelism.dp_shard=-1 \
--accelerator.mixed_precision=bf16 --accelerator.fsdp.cpu_offload=true \
--accelerator.fsdp.reduce_dtype=float32 --policy.push_to_hub=false \
--env_eval_freq=0 --eval_steps=0
- Downloads last month
- 11

