YuE2 Music
Back to tips

How YuE2 Music Works: Behind the Scenes of Symbolic Planning & Acoustic Diffusion

An in-depth look into the architecture of YuE2 3B open-source music model. Understand two-stage symbolic planning, ABC lead sheet conditioning, lyrics token alignment, acoustic diffusion, and the role of random seeds.

Oct 10, 2026YuE2 Engineering TeamYuE2 Engineering Team

In generative AI music, most tools like Suno or Udio act as black-box systems: you provide a text prompt, press generate, and receive an uneditable finished audio track. For casual exploration, this can be fun, but for musicians and music producers seeking exact control over melody lines, chord progressions, structure, and arrangement, black-box systems quickly become frustrating lotteries.

YuE2 takes a fundamentally different approach. As an open-source 3B music foundation model, YuE2 employs a state-of-the-art two-stage architecture: Symbolic Prior Planning decoupled from Neural Acoustic Diffusion.

To help creators maximize their output quality, this guide explains the internal mechanisms of YuE2.

1. The Two-Stage Architecture

End-to-end music models attempt to map arbitrary text prompts directly into raw continuous audio waveforms. This often results in pitch drift, harmonic muddiness, or unpredictable song forms because textual language models lack precise temporal and pitch constraints.

YuE2 separates music creation into two distinct stages:

[User Prompt / Lyrics / Musical Preferences]
                      │
                      ▼
┌────────────────────────────────────────────────────────┐
│  Stage 1: Symbolic Planning                            │
│  - Key, Meter, and BPM determination                   │
│  - Melody line & Chord progression (ABC Lead Sheet)    │
│  - Lyric syllable to musical measure alignment         │
└────────────────────────────────────────────────────────┘
                      │ (ABC tokens + Structural phonemes)
                      ▼
┌────────────────────────────────────────────────────────┐
│  Stage 2: Acoustic Diffusion & Neural Vocoder          │
│  - Gaussian white noise initialized by Random Seed     │
│  - Cross-attention conditioning on ABC score & lyrics  │
│  - Iterative flow matching & denoising steps           │
│  - High-fidelity 44.1kHz stereo audio synthesis        │
└────────────────────────────────────────────────────────┘
  1. Stage 1: Symbolic Planning
    The model plans the composition's skeleton: melodic contour, harmonic chords, and segment layout, expressed in standardized ABC notation.
  2. Stage 2: Acoustic Diffusion
    The acoustic diffusion backbone uses the ABC lead sheet as a rigid geometric condition, with the genre prompt and lyrics providing vocal timbre, instrumental textures, and mix space.

2. Why ABC Notation is the Core of White-Box Controllability

ABC notation is a compact, human-readable ASCII music representation:

X: 1
T: Ocean Breeze
M: 4/4
L: 1/8
Q: 1/4=110
K: G
|: "G" G3 A B2 d2 | "Em" e4 z2 B2 | "C" c3 B A2 G2 | "D" A4 z4 :|

YuE2 relies on ABC scores for three main reasons:

  • Pitch and Metric Determinism: Every note duration and exact pitch is locked before audio generation begins. Pitch drift is eliminated.
  • Harmonic Guidance: Double-quoted chords (e.g. "Em", "Cmaj7") directly instruct the accompaniments on harmony voicing, preventing clashes.
  • Ecosystem Portability: ABC scores can be previewed as sheet music in the browser, exported as multi-track MIDI, and imported into DAWs like Logic Pro, Cubase, or Ableton Live.

3. Lyrics Tokenization and Phoneme Alignment

In YuE2, lyrics are not merely text to be spoken; they serve as structural timing anchors for the acoustic attention heads.

Structural Section Tags

YuE2 expects standardized bracket tags:

  • [intro]: Instrumental prelude.
  • [verse] / [verse 1]: Storytelling verses with dynamic subtlety.
  • [chorus]: The energetic hook with reinforced drums and full vocal harmony.
  • [bridge]: Harmonic or rhythmic transition.
  • [outro]: Outro and resolution.

Instrumental Mode Handling

If you wish to create a purely instrumental piece, the lyrics field should never be completely empty. Without tokens to anchor cross-attention, acoustic diffusion can degenerate. The YuE2 engine automatically injects structured instrumental guidance tokens:

[instrumental]
[intro]
[theme a]
[interlude]
[climax]
[outro]

This guarantees crisp, dynamic solos and accompaniment without unwanted phantom voices.

4. The Acoustic Diffusion Model and Random Seeds

Users sometimes ask: "Why did two generations with the exact same prompt produce the exact same song?"

This comes down to the mathematics of diffusion models:

  1. Initial Noise Canvas: Diffusion models start from pure Gaussian white noise and iteratively denoise it toward clean audio.
  2. Seeds Determine the Canvas: The pseudo-random number generator (PRNG) uses a seed integer to initialize this noise. If the seed is identical, the starting canvas is identical.
  3. Deterministic Denoising: When Prompt + ABC Score + Lyrics + Seed are identical, every denoising step produces identical values, resulting in byte-for-byte identical audio files.
  4. Dynamic Random Seeds: To ensure creative variety, YuE2 generates a random 31-bit seed (0 to 2,147,483,647) for each render task. Even with the same score, changing the seed yields fresh vocal interpretations, varied guitar strumming patterns, and distinct sonic colors.

5. Summary & Best Practices

  1. Plan Melody First: Use the built-in AI assistant to chat and craft an ABC score, then preview it on the piano canvas.
  2. Structure Lyrics: Tag your lyrics with standard [verse] and [chorus] brackets.
  3. Use Instrumental Mode: Keep the toggle on when you want non-vocal productions.
  4. Iterate with Seeds: Re-render with new seeds to explore fresh variations on your favorite melody lines!