README.md

metadata

license: apache-2.0
datasets:
  - projectlosangeles/Godzilla-MIDI-Dataset
  - asigalov61/MIDI-Loops
language:
  - en
tags:
  - Orpheus
  - MIDI
  - music-ai
  - music-transformer
  - SOTA
  - multi-instrumental
  - music
  - loops
  - bridge
  - infilling
  - inpainting
metrics:
  - accuracy

Orpheus Music Transformer

SOTA 8k multi-instrumental music transformer trained on 2.31M+ high-quality MIDIs

---

Abstract

Project Los Angeles is very proud to present Orpheus Music Transformer, an efficient, SOTA transformer model for long-form, multi-instrumental music generation. At its core lies a 479 M-parameter autoregressive transformer equipped with Rotary Positional Embeddings (RoPE) and Flash Attention, enabling sequence lengths up to 8 k tokens—sufficient to capture extended musical structures. Trained for three epochs on 2.31 million high-quality MIDI tracks from the Godzilla dataset, our model employs a compact 3-token-per-note and 7-token-per-tri-chord encoding, plus a novel duration-and-velocity-last ordering to enhance expressivity. We leverage PyTorch’s bfloat16 precision and memory-efficient sparse-dense products for accelerated inference on CUDA, and provide a top-p sampling filter with adjustable temperature.

The Gradio interface empowers users to upload seed MIDI files or generate from scratch, tune prime/generation token counts, control randomness (temperature, top-p), and optionally append drums or natural “outro” tokens. Generated outputs appear in ten parallel batches with synchronized audio previews and piano-roll plots. Users can iteratively add or remove entire batches to sculpt a final composition, which is rendered back into MIDI and audio via an integrated SoundFont pipeline. Our release demonstrates a seamless blend of state-of-the-art model performance, efficient MIDI tokenization, and user-centric design, fostering rapid exploration of algorithmic composition.

---

Models

Presented are seven models:

Base models

Orpheus Music Transformer Model

This is a base model that is capable of music generation/continuation and notes/drums inpainting

Orpheus Music Transformer Classifier Model

This is a music classification model that is was trained to distinguish between human music and music generated by Orpheus base model

Orpheus Bridge Music Transformer Model

This is an auxiliary model that is capable of seamless bridge inpainting/infilling in any music composition

Fine-tuned base models

Orpheus LRNO Segments Music Transformer Model

This is a fine-tuned version of the Orpheus base model that is capable of music segmentation and music segments generation

Orpheus Loops Music Transformer Model

This is a fine-tuned version of the Orpheus base model that is capable of generating multi-instrumental loops

Orpheus Children Songs Music Transformer Model

This is a fine-tuned version of the Orpheus base model that is great for working with children songs

Orpheus Mono Melodies Transformer Model

This is a fine-tuned version of the Orpheus base model that is great for working with compositions that have a monophonic melody

---

Live Hugging Face spaces demos

Orpheus Music Transformer

If you enjoyed any of the Orpheus Music Transformer demos, please star and duplicate. It helps a lot! 🤗

---

Inference notebooks

[NEW & SOTA] Orpheus Auto-Continuations Generator

Orpheus Drums Transformer

---

Training dataset code

Models were trained on select HQ MIDIs from Godzilla MIDI Dataset

Please check out Orpheus Taining Dataset Maker notebook for details

---

Models training code

Please check out Orpheus Music Transformer Maker code/notebook for details

---

Project Los Angeles

Tegridy Code 2025