README.md

metadata

title: ViTViz
emoji: 👁️
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 5.49.1
app_file: app.py
pinned: false

ViTViz – Vision Transformer Attention & Adversarial Attack Analyzer

Interactive tool for visualizing attention mechanisms in Vision Transformers and analyzing adversarial attacks on image classification.

Features

• Classification with top-k predictions
• Attention Visualization via Attention Rollout and per-layer/per-head views
• Adversarial Attacks: FGSM, PGD, MIM, TGR, SAGA with iteration-by-iteration analysis

Supported Models

Any timm-compatible ViT architecture (model.blocks[i].attn.qkv), including:

Model	Architecture	Dataset
ViT-B/16	Base, patch 16	ImageNet-1k
ViT-S/16	Small, patch 16	ImageNet-1k
ViT-L/16	Large, patch 16	ImageNet-1k
ViT-B/32	Base, patch 32, 384px	ImageNet-1k
ViT-S/32	Small, patch 32, 384px	ImageNet-1k
ViT-L/32	Large, patch 32, 384px	ImageNet-1k
Custom Upload	Any ViT	Any

Supported File Formats

.pth, .pt, .safetensors, .ckpt

Quick Start

# 1. Clone e instale as dependências
git clone <repo-url>
cd ViTViz
make install

# 2. Suba o app Gradio
make app

Comandos Disponíveis (make help)

Comando	Descrição
make install	Instala todas as dependências
make app	Roda o app Gradio (localhost:7860)
make quick-test	Teste rápido: ViT-S + FGSM + 1 epsilon
make experiment	Sweep completo de ataques (config padrão)
make experiment-dry	Mostra combinações sem executar nada
make attention	Gera mapas de atenção para todas as combinações
make clean	Remove resultados e cache

Para usar um config específico:

make experiment CONFIG=configs/experiments/exp_all_attacks.yaml

Pipeline de Experimentos

O projeto expõe um pipeline de linha de comando separado da UI, voltado para geração de resultados reproduzíveis para artigos científicos.

Estrutura de diretórios

configs/                  # Configurações YAML de experimentos
  default.yaml            # Config base (modelos, ataques, epsilons, seeds)
  experiments/
    exp_quick_test.yaml   # Teste rápido (1 modelo, 1 ataque)
    exp_all_attacks.yaml  # Sweep completo
data/
  sample_images/          # Imagens de entrada (.jpg/.png)
experiments/
  run_attack_sweep.py     # Sweep de ataques → CSV com métricas
  run_attention_analysis.py  # Salva rollouts de atenção (.npy + .png)
notebooks/
  01-metrics-analysis.ipynb  # Gera tabelas LaTeX e gráficos de métricas
  02-attention-figures.ipynb # Gera figuras de comparação de atenção
results/
  raw/                    # Outputs brutos (CSV, .npy, imagens)
  figures/                # Figuras geradas pelos notebooks
  tables/                 # Tabelas LaTeX

Workflow para paper

# 1. Adicione imagens em data/sample_images/
# 2. Rode os experimentos
make experiment

# 3. Gere os mapas de atenção
make attention

# 4. Abra os notebooks para gerar figuras e tabelas
cd notebooks && jupyter notebook

Métricas calculadas automaticamente

• Perturbação de imagem: L∞, PSNR, SSIM, LPIPS, Modified Pixels
• Deslocamento de atenção: Attention W1 (Rollout, padrão), Attention JSD (Rollout)
• Ataque: ASR, Confidence Drop, Top-k Overlap Drop
  • Na aba de ataque, quando Ground Truth Label é informado, as métricas de classe usam esse rótulo como referência; sem GT, usam fallback para a predição original.

Apuana Cluster Integration (UI)

The app now includes an Apuana Cluster tab to orchestrate remote SLURM jobs from the local UI.

Prerequisites

• VPN + access enabled for Apuana
• SSH configured locally (key/agent) for slurm-client1.cin.ufpe.br
• Remote project already available on cluster (example: ~/ViTViz)
• Python env available on cluster and activation script path known (example: $HOME/envs/vitviz/bin/activate)

Supported actions in the UI

• Generate a job bundle (experiment.yaml + run_job.sh)
• Build experiment config dynamically in UI (models, custom models, attacks, params, seeds, epsilons, metrics)
• Sync bundle to cluster via rsync
• Submit with sbatch and capture JOB_ID
• Check status (squeue / sacct)
• Cancel job (scancel)
• Download remote files via rsync

Recommended first run

1. Open Apuana Cluster tab.
2. Fill remote user/host and remote project dir.
3. Choose Config Source:

• Use Config File: pick an existing YAML (e.g. configs/experiments/exp_quick_test.yaml)
• Build in UI: select default models, optionally add custom models, choose attacks, edit attack params JSON, define seeds/epsilons/metrics

4. Click Generate Bundle.
5. Click Sync Bundle.
6. Click Submit Job.
7. Use Check Job Status / List My Queue to monitor.

Dynamic builder notes

• Custom model line format: name|path|img_size|num_classes|type|dataset
• Only name|path are required in custom model lines.
• Attack params are edited per attack as JSON objects.
• selected_metrics is written into evaluation and used by the sweep runner.
• CSV output schema stays fixed; metrics not selected are saved as empty cells.