| --- |
| license: mit |
| datasets: |
| - ILSVRC/imagenet-1k |
| language: |
| - en |
| tags: |
| - diffusion |
| --- |
| |
| # Transfusion - VAE |
|
|
| ## How to use with 🧨 diffusers |
|
|
| ```py |
| from diffusers.models import AutoencoderKL |
| vae = AutoencoderKL.from_pretrained("lavinal712/transfusion-vae") |
| ``` |
|
|
| ## Model |
|
|
| This model was trained for 50 (legacy: 7) epochs on ImageNet, COCO and FFHQ (legacy: ImageNet), with training parameters following the original Transfusion paper. |
|
|
| $$\mathcal{L}_{\mathrm{VAE}} = \mathcal{L}_1 + \mathcal{L}_{\mathrm{LPIPS}} + 0.5\mathcal{L}_{\mathrm{GAN}} + 0.2\mathcal{L}_{\mathrm{ID}} + 0.000001\mathcal{L}_{\mathrm{KL}}$$ |
|
|
| ## Evaluation |
|
|
| ImageNet 2012 (256x256, val, 50000 images) |
|
|
| | Model | rFID | PSNR | SSIM | LPIPS | |
| |-----------------|-------|--------|-------|-------| |
| | Transfusion-VAE | 0.408 | 28.723 | 0.845 | 0.081 | |
| | SD-VAE | 0.692 | 26.910 | 0.772 | 0.130 | |
|
|
| COCO 2017 (256x256, val, 5000 images) |
|
|
| | Model | rFID | PSNR | SSIM | LPIPS | |
| |-----------------|-------|--------|-------|-------| |
| | Transfusion-VAE | 2.749 | 28.556 | 0.855 | 0.078 | |
| | SD-VAE | 4.246 | 26.622 | 0.784 | 0.127 | |
|
|
| ## Evaluation (legacy) |
|
|
| ImageNet 2012 (256x256, val, 50000 images) |
|
|
| | Model | rFID | PSNR | SSIM | LPIPS | |
| |-----------------|-------|--------|-------|-------| |
| | Transfusion-VAE | 0.567 | 28.195 | 0.829 | 0.100 | |
| | SD-VAE | 0.692 | 26.910 | 0.772 | 0.130 | |
|
|
| Paper: [Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model](https://arxiv.org/abs/2408.11039) |
|
|
| Dataset: [ImageNet](https://image-net.org/), [COCO](https://cocodataset.org/), [FFHQ](https://github.com/NVlabs/ffhq-dataset) |
|
|
| Base Code: [lavinal712/AutoencoderKL](https://github.com/lavinal712/AutoencoderKL) |
|
|
| Training Code: [lavinal712/AutoencoderKL/tree/transfusion_vae](https://github.com/lavinal712/AutoencoderKL/tree/transfusion_vae) |
