EXPLAINABILITY.md

Explainability Subcard

intended_domain

Surgical world modeling and action-conditioned generation: forward-dynamics surgical simulation, inverse-dynamics (video-to-kinematics) action labeling, surgical policy modeling, closed-loop policy evaluation, and synthetic data generation. Used directly in any of its three modes or as the initialization for further surgical fine-tuning.

Model Type

Mixture-of-Transformers (MoT) world foundation model, post-trained jointly on three tasks (forward dynamics, inverse dynamics, policy) with text and action-history conditioning.

Intended Users

Medical Robotics Engineers, Surgical Robotics Researchers, Healthcare AI Developers, Surgeons and Surgical Trainees

Output

Types: A sequence of generated video frames (forward dynamics / policy) and/or a sequence of predicted action vectors, 24 steps at 10 Hz (inverse dynamics / policy). Formats: Red, Green, Blue (RGB) video; numeric action vectors.

Describe how the model works:

The model couples video, action, and text in one multimodal sequence over a Mixture-of-Transformers with a shared Reasoner (vision-language) tower and Generator (world-simulator / diffusion) tower. It works on 2.4 s clips (25 frames and 24 actions at 10 Hz, 256-pixel resolution tier). Actions are per-arm end-effector poses relative to the current pose plus the gripper command (20 values for a two-arm robot, zero-padded to 64 channels at the model input), normalized per dataset family; up to 16 past action steps are given as clean conditioning. The three operating modes are different clean-vs-noised token masks over that single shared sequence: in forward dynamics the first frame and the actions are clean conditioning and the model denoises the future video (p(video | frame₀, actions)); in inverse dynamics the video frames are clean conditioning and the model denoises the actions (p(actions | video)); in policy the first frame, the task instruction and the action history are clean and the model denoises both future actions and video (p(actions, video | frame₀, text, action history)). Actions enter through embodiment-specific action-embedding projections and are predicted (in inverse dynamics / policy) through an action-prediction head. Because it is trained on all three modes jointly (mixed 0.4 / 0.2 / 0.4), one checkpoint learns both directions of the surgical world–action relationship.

Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of:

None

Technical Limitations & Mitigation:

The model may underperform in poor or variable lighting conditions, occlusions from instruments or blood, and specular reflections, which can degrade visual predictions. It may not perform well for out-of-distribution scenarios including novel procedures, unusual anatomies, or emergency situations not well-represented in the training data; rapid motions or long-horizon forward-dynamics rollouts chained from multiple clips, where errors accumulate; actions beyond the trained kinematic range (normalized values are clipped at ±1.5) or near joint limits; and generalization challenges across different camera placements, surgical sites, or surgeon styles. Because dataset families are sampled with unequal mixture ratios (e.g., JHU dVRK ~32%, CMR Versius clinical ~20%, several families ≤2%), per-embodiment fidelity varies. As a jointly trained model, per-mode quality may trade off slightly against a bespoke single-mode model; mode-specialized fine-tuning may improve performance on a given mode.

Mitigation: To mitigate these limitations, we recommend performing data augmentation with lighting and occlusion variations; uncertainty estimation and out-of-distribution detection to flag anomalous states; limiting autoregressive rollout length with periodic ground-truth re-anchoring/reset; enforcing kinematic and safety constraints; multi-site training data collection; mode-specialized fine-tuning for the intended use; and maintaining strict human oversight with multiple safety layers.

Verified to have met prescribed NVIDIA quality standards:

Yes

Performance Metrics:

Per dataset family on the held-out validation split: video PSNR against ground truth (forward dynamics and policy) and action mean squared error in normalized action space (inverse dynamics and policy).

Potential Known Risks:

The model may generate videos that contain artifacts. The model may inaccurately represent 3D space, 4D space-time, or physical laws in the generated videos, leading to artifacts such as disappearing or morphing objects, unrealistic interactions, implausible motions, and physically inconsistent outcomes. In forward-dynamics mode errors can compound over long autoregressive rollouts. Predicted actions (inverse-dynamics / policy) may be inaccurate and must not be executed on a physical system without independent validation and human oversight.

Licensing:

Governing Terms: Use of this model is governed by the NVIDIA Open Model License Agreement.