MedVersa: A Generalist Learner for Multifaceted Medical Image Interpretation

A model implementation for our paper A Generalist Learner for Multifaceted Medical Image Interpretation.

MedVersa is a compound medical AI system that coordinates multimodal inputs, orchestrates models and tools for various medical imaging tasks, and generates multimodal outputs.

Installation

Prerequisites

• Python
• NVIDIA CUDA-compatible GPU
• Miniconda or Anaconda

Environment Setup

1. Create and activate the conda environment:

# For NVIDIA A100 GPUs
conda env create -f environment.yml
conda activate medversa

# For NVIDIA H100 GPUs (CUDA 11.8)
conda env create -f environment_cu118.yml
conda activate medversa

Troubleshooting

If you encounter dependency issues, try these solutions:

# Fix OpenCV issues
pip install opencv-contrib-python

# Fix incompatible torchvision version
pip install torchvision==0.15.2+cu118 --index-url https://download.pytorch.org/whl/cu118

Usage

Basic Inference

from utils import *
from torch import cuda

# Initialize model
device = 'cuda' if cuda.is_available() else 'cpu'
model_cls = registry.get_model_class('medomni') # medomni is the architecture name :)
model = model_cls.from_pretrained('hyzhou/MedVersa').to(device).eval()

# Define example input
example = {
    'images': ["./demo_ex/c536f749-2326f755-6a65f28f-469affd2-26392ce9.png"],
    'context': "Age:30-40.\nGender:F.\nIndication: ___-year-old female with end-stage renal disease not on dialysis presents with dyspnea. PICC line placement.\nComparison: None.",
    'prompt': "How would you characterize the findings from <img0>?",
    'modality': "cxr",
    'task': "report generation"
}

# Configure generation parameters
params = {
    'num_beams': 1,
    'do_sample': True,
    'min_length': 1,
    'top_p': 0.9,
    'repetition_penalty': 1,
    'length_penalty': 1,
    'temperature': 0.1
}

# Generate predictions
seg_mask_2d, seg_mask_3d, output_text = generate_predictions(
    model, 
    example['images'],
    example['context'],
    example['prompt'],
    example['modality'],
    example['task'],
    **params,
    device,
)
print(output_text)

For more detailed examples and usage scenarios, see inference.py.

Prompts

More prompts can be found in medomni/datasets/prompts.json.