Cosmos3-H-Embodied-Nano / EXPLAINABILITY.md

Claude Opus 5.5

Align model card style with Cosmos3-H-Dreams

9ad2093 15 days ago

5.14 kB

# Explainability Subcard
## intended_domain
Surgical world modeling and action-conditioned generation: forward-dynamics surgical simulation, inverse-dynamics (video-to-kinematics) action labeling, surgical policy modeling, closed-loop policy evaluation, and synthetic data generation. Used directly in any of its three modes or as the initialization for further surgical fine-tuning.
## Model Type
Mixture-of-Transformers (MoT) world foundation model, post-trained jointly on three tasks (forward dynamics, inverse dynamics, policy) with text and action-history conditioning.
## Intended Users
Medical Robotics Engineers, Surgical Robotics Researchers, Healthcare AI Developers, Surgeons and Surgical Trainees
## Output
Types: A sequence of generated video frames (forward dynamics / policy) and/or a sequence of predicted action vectors, 24 steps at 10 Hz (inverse dynamics / policy). Formats: Red, Green, Blue (RGB) video; numeric action vectors.
## Describe how the model works:
The model couples video, action, and text in one multimodal sequence over a Mixture-of-Transformers with a shared Reasoner (vision-language) tower and Generator (world-simulator / diffusion) tower. It works on 2.4 s clips (25 frames and 24 actions at 10 Hz, 256-pixel resolution tier). Actions are per-arm end-effector poses relative to the current pose plus the gripper command (20 values for a two-arm robot, zero-padded to 64 channels at the model input), normalized per dataset family; up to 16 past action steps are given as clean conditioning. The three operating modes are different clean-vs-noised token masks over that single shared sequence: in **forward dynamics** the first frame and the actions are clean conditioning and the model denoises the future video (p(video | frame₀, actions)); in **inverse dynamics** the video frames are clean conditioning and the model denoises the actions (p(actions | video)); in **policy** the first frame, the task instruction and the action history are clean and the model denoises both future actions and video (p(actions, video | frame₀, text, action history)). Actions enter through embodiment-specific action-embedding projections and are predicted (in inverse dynamics / policy) through an action-prediction head. Because it is trained on all three modes jointly (mixed 0.4 / 0.2 / 0.4), one checkpoint learns both directions of the surgical world–action relationship.
## Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of:
None
## Technical Limitations & Mitigation:
The model may underperform in poor or variable lighting conditions, occlusions from instruments or blood, and specular reflections, which can degrade visual predictions. It may not perform well for out-of-distribution scenarios including novel procedures, unusual anatomies, or emergency situations not well-represented in the training data; rapid motions or long-horizon forward-dynamics rollouts chained from multiple clips, where errors accumulate; actions beyond the trained kinematic range (normalized values are clipped at ±1.5) or near joint limits; and generalization challenges across different camera placements, surgical sites, or surgeon styles. Because dataset families are sampled with unequal mixture ratios (e.g., JHU dVRK ~32%, CMR Versius clinical ~20%, several families ≤2%), per-embodiment fidelity varies. As a jointly trained model, per-mode quality may trade off slightly against a bespoke single-mode model; mode-specialized fine-tuning may improve performance on a given mode.
**Mitigation:**
To mitigate these limitations, we recommend performing data augmentation with lighting and occlusion variations; uncertainty estimation and out-of-distribution detection to flag anomalous states; limiting autoregressive rollout length with periodic ground-truth re-anchoring/reset; enforcing kinematic and safety constraints; multi-site training data collection; mode-specialized fine-tuning for the intended use; and maintaining strict human oversight with multiple safety layers.
## Verified to have met prescribed NVIDIA quality standards:
Yes
## Performance Metrics:
Per dataset family on the held-out validation split: video PSNR against ground truth (forward dynamics and policy) and action mean squared error in normalized action space (inverse dynamics and policy).
## Potential Known Risks:
The model may generate videos that contain artifacts. The model may inaccurately represent 3D space, 4D space-time, or physical laws in the generated videos, leading to artifacts such as disappearing or morphing objects, unrealistic interactions, implausible motions, and physically inconsistent outcomes. In forward-dynamics mode errors can compound over long autoregressive rollouts. Predicted actions (inverse-dynamics / policy) may be inaccurate and must not be executed on a physical system without independent validation and human oversight.
## Licensing:
Governing Terms: Use of this model is governed by the [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/).

	# Explainability Subcard
	## intended_domain
	Surgical world modeling and action-conditioned generation: forward-dynamics surgical simulation, inverse-dynamics (video-to-kinematics) action labeling, surgical policy modeling, closed-loop policy evaluation, and synthetic data generation. Used directly in any of its three modes or as the initialization for further surgical fine-tuning.

	## Model Type
	Mixture-of-Transformers (MoT) world foundation model, post-trained jointly on three tasks (forward dynamics, inverse dynamics, policy) with text and action-history conditioning.

	## Intended Users
	Medical Robotics Engineers, Surgical Robotics Researchers, Healthcare AI Developers, Surgeons and Surgical Trainees

	## Output
	Types: A sequence of generated video frames (forward dynamics / policy) and/or a sequence of predicted action vectors, 24 steps at 10 Hz (inverse dynamics / policy). Formats: Red, Green, Blue (RGB) video; numeric action vectors.

	## Describe how the model works:
	The model couples video, action, and text in one multimodal sequence over a Mixture-of-Transformers with a shared Reasoner (vision-language) tower and Generator (world-simulator / diffusion) tower. It works on 2.4 s clips (25 frames and 24 actions at 10 Hz, 256-pixel resolution tier). Actions are per-arm end-effector poses relative to the current pose plus the gripper command (20 values for a two-arm robot, zero-padded to 64 channels at the model input), normalized per dataset family; up to 16 past action steps are given as clean conditioning. The three operating modes are different clean-vs-noised token masks over that single shared sequence: in forward dynamics the first frame and the actions are clean conditioning and the model denoises the future video (p(video \| frame₀, actions)); in inverse dynamics the video frames are clean conditioning and the model denoises the actions (p(actions \| video)); in policy the first frame, the task instruction and the action history are clean and the model denoises both future actions and video (p(actions, video \| frame₀, text, action history)). Actions enter through embodiment-specific action-embedding projections and are predicted (in inverse dynamics / policy) through an action-prediction head. Because it is trained on all three modes jointly (mixed 0.4 / 0.2 / 0.4), one checkpoint learns both directions of the surgical world–action relationship.


	## Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of:
	None

	## Technical Limitations & Mitigation:
	The model may underperform in poor or variable lighting conditions, occlusions from instruments or blood, and specular reflections, which can degrade visual predictions. It may not perform well for out-of-distribution scenarios including novel procedures, unusual anatomies, or emergency situations not well-represented in the training data; rapid motions or long-horizon forward-dynamics rollouts chained from multiple clips, where errors accumulate; actions beyond the trained kinematic range (normalized values are clipped at ±1.5) or near joint limits; and generalization challenges across different camera placements, surgical sites, or surgeon styles. Because dataset families are sampled with unequal mixture ratios (e.g., JHU dVRK ~32%, CMR Versius clinical ~20%, several families ≤2%), per-embodiment fidelity varies. As a jointly trained model, per-mode quality may trade off slightly against a bespoke single-mode model; mode-specialized fine-tuning may improve performance on a given mode.

	Mitigation:
	To mitigate these limitations, we recommend performing data augmentation with lighting and occlusion variations; uncertainty estimation and out-of-distribution detection to flag anomalous states; limiting autoregressive rollout length with periodic ground-truth re-anchoring/reset; enforcing kinematic and safety constraints; multi-site training data collection; mode-specialized fine-tuning for the intended use; and maintaining strict human oversight with multiple safety layers.

	## Verified to have met prescribed NVIDIA quality standards:
	Yes

	## Performance Metrics:
	Per dataset family on the held-out validation split: video PSNR against ground truth (forward dynamics and policy) and action mean squared error in normalized action space (inverse dynamics and policy).

	## Potential Known Risks:
	The model may generate videos that contain artifacts. The model may inaccurately represent 3D space, 4D space-time, or physical laws in the generated videos, leading to artifacts such as disappearing or morphing objects, unrealistic interactions, implausible motions, and physically inconsistent outcomes. In forward-dynamics mode errors can compound over long autoregressive rollouts. Predicted actions (inverse-dynamics / policy) may be inaccurate and must not be executed on a physical system without independent validation and human oversight.

	## Licensing:
	Governing Terms: Use of this model is governed by the [NVIDIA Open Model License Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/).