how to actually successfully use the cover mode?

[TanvirHafiz]

havent found any tutorial. anyone can guide me please?

[sigalarm]

Safe to assume this did not help?

#235

If not, tell me what went wrong, I may be able to help fill in the gaps.

[TanvirHafiz]

My interface is somehow a lot different. And also I does not seem to do what it says. I was hoping cover actually keeps the music structure with a new lyrics. But I get I'm getting something entire different. Also using it breaks the system

[sigalarm]

Actually, just try this

ACE-Step Cover Mode — A Complete Guide for Musicians

How to use Cover mode to restyle, remix, and reimagine songs with ACE-Step 1.5.

---

What Is Cover Mode?

Cover mode takes an existing piece of audio and regenerates it in a different style while preserving its underlying structure. Think of it the way a band might "cover" a song — same melody and form, but a completely different sound.

You give ACE-Step a source song, describe the new style you want, and it produces a version that follows the original's blueprint (melody contour, chord progression, rhythm, song form) while changing the sonic details (instruments, production, vocal style, mix).

    YOUR SOURCE AUDIO                      YOUR NEW VERSION
    ┌──────────────────┐                   ┌──────────────────┐
    │  Country ballad  │   Cover mode +    │  Jazz lounge     │
    │  Acoustic guitar │  ──────────────▶  │  Piano & upright │
    │  Twangy vocals   │   new caption     │  bass, smooth    │
    │  Simple drums    │                   │  vocals, brushes │
    └──────────────────┘                   └──────────────────┘

    Structure preserved:  melody shape, song sections, energy arc, timing
    Style changed:        instruments, timbre, production, vocal character

---

When to Use Cover Mode (and When Not To)

Use Cover when you want to:

• Turn a song from one genre into another (pop to jazz, folk to electronic)
• Create an alternate arrangement of a song you've generated
• Remix a track — change lyrics or vocals while keeping the structure
• Produce multiple stylistic variations of the same musical idea
• Reinterpret a source recording with new instrumentation

Don't use Cover when you want to:

• Fix one bad section of an otherwise good song — use Repaint instead
• Add a new instrument layer to an existing track — use Lego instead
• Remove an instrument from a mix — use Extract instead
• Generate something completely from scratch — use Text to Music instead

---

How It Works Under the Hood

Understanding what's actually happening will help you use the feature effectively.

When you upload source audio in cover mode, ACE-Step converts it into semantic codes — a compressed representation that captures the musical structure (melody, rhythm, chords, arrangement) while discarding fine-grained sonic details like specific timbres and production effects.

These codes act as a skeleton or scaffold. The DiT (the "Studio Engineer" brain) then generates brand new audio that follows this scaffold, but fills in all the sonic detail based on your caption, lyrics, and other settings.

    ┌──────────────┐     ┌──────────────┐     ┌──────────────┐
    │ Source audio │────▶│ Semantic     │────▶│ DiT builds   │
    │ (your song)  │     │ codes        │     │ new audio on │
    │              │     │ (structure)  │     │ this skeleton│
    └──────────────┘     └──────────────┘     └──────┬───────┘
                                                      │
                         Your caption ────────────────┘
                         (describes the new style)

Important: The Language Model (Brain 1 / "The Songwriter") is skipped in cover mode. You don't need it to plan the song because the source audio already provides the plan. This means:

• Cover mode is faster than text-to-music generation
• The "Thinking" checkbox has no effect in cover mode
• You are the planner — the source audio + your caption replace what the LM would normally do

---

Step-by-Step: Your First Cover

In the Gradio Web Interface

1. Initialize the service — Make sure you've clicked "Initialize Service" and models are loaded.

2. Switch to Custom Mode — Cover works in Custom mode (the Simple mode is designed for text-to-music generation).

3. Select the "cover" task type — In the task type dropdown, choose cover. You'll see the instruction text change to "Generate audio semantic tokens based on the given conditions:" — that's normal.

4. Upload your source audio — Open the Audio Uploads accordion and upload your source file into the Source Audio slot.

   • Supported formats: WAV, MP3, FLAC, OGG, OPUS
   • The audio will be automatically converted to stereo 48kHz internally

5. Write your caption — Describe the new style you want, not the style of the source. Be specific about genre, instruments, mood, production, and vocal character. (More on this below.)

6. Optionally write new lyrics — You can provide different lyrics than the original. The cover will follow the source audio's structure but use your new words.

7. Set Audio Cover Strength — This slider appears when you select the cover task. It controls how closely the output follows the source audio's structure. (More on this below.)

8. Set duration — The duration should match your source audio's length. If your source is a 3-minute song, set duration to around 180 seconds.

9. Generate — Click "Generate Music." Generate a batch of 2-4 to give yourself choices.

Via the REST API

import requests

response = requests.post("http://localhost:8001/generate", json={
    "task_type": "cover",
    "src_audio_path": "/path/to/source_song.mp3",
    "caption": "jazz piano trio, upright bass, brushed drums, warm intimate vocals",
    "lyrics": "[Verse 1]\nYour new lyrics here...\n\n[Chorus]\n...",
    "audio_cover_strength": 0.6,
    "audio_duration": 180,
    "vocal_language": "en",
    "batch_size": 2
})

Via the Python API

from acestep.inference import GenerationParams, GenerationConfig, generate_music

params = GenerationParams(
    task_type="cover",
    src_audio="original_song.mp3",
    caption="jazz piano trio, upright bass, brushed drums, warm intimate vocals",
    lyrics="[Verse 1]\nYour new lyrics here...",
    audio_cover_strength=0.6,
    duration=180,
    vocal_language="en",
)

config = GenerationConfig(batch_size=2)
result = generate_music(dit_handler, llm_handler, params, config)

---

The Audio Cover Strength Slider

This is the single most important control in cover mode. It determines what proportion of the diffusion process uses the source audio's structure as guidance.

    Audio Cover Strength

    0.0                    0.5                    1.0
    ├──────────────────────┼──────────────────────┤
    │                      │                      │
    Source audio           Balanced blend.         Source audio
    is ignored.            Recognizable            is followed
    Equivalent to          but clearly             very closely.
    text-to-music.         transformed.            Subtle changes.

Practical Settings

What You Want	Suggested Range	Example
Dramatic genre change (country to metal, classical to EDM)	0.3 - 0.5	Give the model room to completely reinvent the sound
Moderate restyle (pop to jazz, rock to R&B)	0.5 - 0.7	Source structure is recognizable but style is clearly new
Subtle transformation (rock to indie rock, pop to dream pop)	0.7 - 0.9	Light touch — same feel, different texture
"Same song, different take"	0.9 - 1.0	Minimal change — variations in performance details only
Style transfer (apply general vibe, loose structure)	0.1 - 0.3	Very loose — more of an inspiration than a cover

What's Actually Happening

Technically, audio_cover_strength controls what fraction of the denoising steps use the source audio's semantic codes versus the text-only (non-cover) encoding. At strength 1.0, every step uses the cover codes. At strength 0.5, half the steps use cover codes and half use text-only conditioning. At 0.0, no steps use cover codes — it becomes equivalent to generating from text alone.

When the value is less than 1.0, the system computes two separate text encodings: one with the cover instruction ("Generate audio semantic tokens based on the given conditions") and one with the standard text-to-music instruction. The DiT blends between these based on the strength, giving you smooth control over how much the source audio constrains the result.

---

Writing Captions for Cover Mode

Your caption in cover mode describes the target style — the style you want the output to sound like, not the style of the source audio.

Good Practices

Be specific about what should change:

Source: A country ballad

Good caption:    "heavy metal rock with heavily distorted electric guitars,
                  aggressive double bass drumming, powerful screaming vocals,
                  fast tempo, high energy, intense dark atmosphere"

This tells the model exactly how the new version should sound.

Describe multiple dimensions:

"smooth jazz, warm Rhodes piano, walking upright bass, brushed snare,
 female vocal with breathy intimate tone, late-night club atmosphere,
 polished studio production"

The more dimensions you specify (genre, instruments, vocal style, mood, production), the less the model has to guess.

Use era and reference points:

"80s synthwave, analog synth pads, pulsing bassline, gated reverb drums,
 reminiscent of Kavinsky or The Midnight, neon-lit night drive energy"

Common Caption Mistakes in Cover Mode

Mistake	Why It's a Problem	Fix
"cover version of this song"	The model doesn't know what "this" is — it only sees text, not the audio's genre	Describe the target style explicitly
"make it sound better"	Too vague — better how?	Name specific qualities: warmer, crisper, more energetic
"same style but different"	Contradictory — same style isn't different	Pick what to change: instruments? mood? tempo feel?
"pop rock jazz fusion classical"	Too many conflicting styles	Pick one primary style, add 1-2 modifiers
Describing the source style	The model tries to generate what you describe, not what the source sounds like	Describe only the new target style

---

Lyrics in Cover Mode

You can provide new lyrics or leave them empty. Here's how each approach behaves:

Providing New Lyrics

If you write new lyrics, the cover will attempt to sing your new words while following the source audio's structural timing. This is great for:

• Translating a song to a different language
• Writing a parody with different words
• Changing the theme while keeping the melody

Keep the structure similar to the source: If your source has two verses, a chorus, a bridge, and a final chorus, your lyrics should follow a similar layout. The source audio's structural timing won't change, so drastically different section lengths may cause timing mismatches.

[Verse 1]
New words that roughly match
The syllable count of the original
So timing feels natural

[Chorus]
Keep chorus lines punchy
Similar length to source

Leaving Lyrics Empty

If you provide no lyrics, the model generates audio that follows the source structure but may produce instrumental sections or use its own judgment for vocal content. This works well for:

• Instrumental style transfers
• Cases where you care about mood/genre but not lyrics

Keeping Original Lyrics

If you want to keep the original lyrics, you'll need to type them in (or provide them). The model doesn't extract lyrics from source audio — it only reads the semantic structure (melody, rhythm, etc.).

---

Two Paths: Source Audio vs. Audio Codes

There are two ways to provide the source material for a cover:

Path 1: Upload Source Audio (Recommended for Most Users)

Upload an audio file directly. ACE-Step handles the conversion to semantic codes internally.

• Upload goes in the Source Audio slot under Audio Uploads
• The system encodes it through the VAE and quantizer automatically
• This is the simplest approach

Path 2: Use Pre-Extracted Audio Codes (Advanced)

You can convert audio to semantic codes first, then paste those codes into the LM Codes Hints box. This is useful for:

• Reusing codes across multiple covers — Convert once, try many styles without re-encoding
• Editing codes — Trim, extend, or combine codes from different sources (DJ mixing style)
• Sharing codes — Send text-based codes to someone else without sharing the audio file

To extract codes from audio:

1. Upload audio to the Source Audio slot
2. Click "Convert to Codes" button (below the audio upload)
3. The codes appear as a string of tokens like <|audio_code_123|><|audio_code_456|>...
4. Copy this string into the LM Codes Hints text box
5. You can now remove the source audio — the codes are self-contained

Auto-detection: If you're in text-to-music mode but paste audio codes into the LM Codes Hints box, ACE-Step automatically switches to cover mode. You don't need to manually change the task type.

Code rate: Audio codes are generated at 5Hz (5 codes per second of audio). A 60-second song produces about 300 codes. A 3-minute song produces about 900 codes.

---

Common Scenarios and Recipes

Scenario 1: Genre Transformation

"I have a folk song and want to hear it as electronic music."

Setting	Value
Task type	cover
Source audio	Your folk song
Caption	electronic dance music, heavy synth bass, four-on-the-floor kick drum, layered synth pads, vocoder-processed vocals, high energy club atmosphere
Audio cover strength	0.4 (give room for electronic elements to dominate)
Lyrics	Leave empty or provide new lyrics
Duration	Match source length

Scenario 2: Acoustic Reimagining

"I have a produced pop track and want a stripped-down acoustic version."

Setting	Value
Task type	cover
Source audio	Your pop track
Caption	stripped acoustic version, solo acoustic guitar fingerpicking, soft intimate male vocal, minimal production, raw live recording feel, warm and vulnerable
Audio cover strength	0.7 (maintain melody and structure closely)
Lyrics	Same lyrics as original
Duration	Match source length

Scenario 3: Vocal Swap / Language Change

"I have an English pop song and want a Japanese version."

Setting	Value
Task type	cover
Source audio	Your English song
Caption	J-pop, bright female vocal, polished pop production, catchy and uplifting
Audio cover strength	0.6
Vocal language	ja
Lyrics	Japanese lyrics following the same song structure
Duration	Match source length

Scenario 4: The "Retake Lottery"

"I like the structure of this song but want to hear different interpretations."

Setting	Value
Task type	cover
Source audio	Your song
Caption	Same style description as the original
Audio cover strength	0.8 - 0.9
Batch size	4
Use random seed	Yes

Generate a batch. Each variation follows the same structure but with different performance details — different vocal nuances, slightly different instrument phrasing, different mix balance. Pick your favorite.

Scenario 5: Remix With New Lyrics

"I want to keep the musical structure but write completely new lyrics."

Setting	Value
Task type	cover
Source audio	Your song
Caption	Describe the style (can be same or different from original)
Audio cover strength	0.6
Lyrics	Your completely new lyrics, with structure tags matching the source song's sections

---

Troubleshooting Common Problems

"My cover sounds nothing like the source"

Cause: Audio cover strength is too low.
Fix: Increase audio_cover_strength toward 0.7-0.9. At very low values (0.1-0.3), the source audio is barely used and the model mostly generates from your text prompt alone.

"My cover sounds almost identical to the source"

Cause: Audio cover strength is too high.
Fix: Decrease audio_cover_strength toward 0.4-0.6. At very high values (0.9-1.0), the model sticks so closely to the source that your caption and style changes have minimal effect.

"The timing/duration is wrong"

Cause: Duration mismatch between source audio and the duration parameter.
Fix: Set your duration parameter to match your source audio's actual length. If your source is 2:30, set duration to 150 seconds. If you don't set a duration, the system uses the source audio's length automatically, but it's good practice to set it explicitly.

"The vocals sound garbled or don't match my lyrics"

Cause: Lyrics structure doesn't match the source audio's section timing.
Fix: Make sure your lyrics follow the same verse/chorus/bridge structure as the source. If the source has 4 lines per verse, write roughly 4 lines per verse. Keep syllable counts similar to the original so the vocal timing works naturally.

"I get an error about model not initialized"

Cause: The service hasn't been started yet.
Fix: Click "Initialize Service" in the Gradio UI. Wait for models to load (you'll see progress messages). Cover mode needs the DiT model and VAE, but does not require the LM to be initialized.

"The style didn't change much despite a descriptive caption"

Cause: Audio cover strength is probably too high.
Fix: Lower the strength. Also make sure your caption describes the target style vividly with specific instrument names, production qualities, and vocal characteristics. Vague captions like "rock version" give the model too little to work with.

"I want to use cover mode but I'm on a low-VRAM GPU (4-6 GB)"

Tip: Cover mode is actually lighter on VRAM than text-to-music with thinking enabled, because the LM is skipped entirely. You only need the DiT and VAE loaded. Enable CPU offloading if needed (--offload_to_cpu true).

"Convert to Codes button gives an error"

Cause: The DiT model and VAE must be initialized before you can convert audio to codes.
Fix: Initialize the service first, then try the conversion again.

---

Reference Audio vs. Source Audio: What's the Difference?

This is a common point of confusion. Both are audio uploads, but they serve very different purposes:

    REFERENCE AUDIO                         SOURCE AUDIO
    ┌──────────────────────┐                ┌──────────────────────┐
    │ Global style guide   │                │ Structural blueprint │
    │                      │                │                      │
    │ Affects timbre,      │                │ Affects melody,      │
    │ production quality,  │                │ rhythm, chord        │
    │ and overall sonic    │                │ progression, song    │
    │ texture of output    │                │ form, and energy arc │
    │                      │                │                      │
    │ Like showing a       │                │ Like giving a band   │
    │ mixer a reference    │                │ sheet music for a    │
    │ track and saying     │                │ song and saying      │
    │ "make it sound       │                │ "play this but in    │
    │ like THIS quality"   │                │ a different style"   │
    └──────────────────────┘                └──────────────────────┘

	Reference Audio	Source Audio
Purpose	Style / timbre guide	Structural blueprint
What it controls	Overall sonic character, production texture	Melody, rhythm, chords, song form
Used in cover mode?	Optional (can combine with source audio)	Required (this IS the cover source)
Where to upload	"Reference Audio" slot (left)	"Source Audio" slot (right)
Strength control	N/A (global influence)	audio_cover_strength slider

Pro tip: You can use both together. Upload a song as Source Audio for structure and a different recording as Reference Audio for timbre/texture guidance. For example: use a rock song's structure but aim for the warm analog sound of a specific jazz recording.

---

Advanced: The Audio Codes Workflow

For power users, working with audio codes directly opens up creative possibilities beyond basic cover mode.

What Are Audio Codes?

Audio codes are a compressed text representation of a song's musical structure. They look like this:

<|audio_code_14523|><|audio_code_8891|><|audio_code_45102|>...

Each code is a token from a 64,000-entry codebook. Codes are generated at 5Hz (5 per second of audio), so a 60-second clip produces ~300 codes.

Creative Uses of Audio Codes

1. Mix-and-match structure: Take codes from the verse of Song A and the chorus of Song B. Paste them together to create a hybrid structure, then cover it in a new style.

2. Loop and extend: Copy a section of codes and paste it multiple times to create a looped or extended version of a phrase.

3. Share without sharing audio: Codes are just text. You can share a song's structural DNA without sharing the actual audio file.

4. Rapid iteration: Convert your source once to codes, then try dozens of different captions/styles without re-encoding the audio each time.

Working With Codes

1. Upload source audio and click "Convert to Codes"
2. Copy the resulting code string
3. Paste it into the "LM Codes Hints" text box
4. Clear the source audio upload (optional — codes take priority over uploaded audio when both are present)
5. Set your caption and cover strength
6. Generate

---

Tips From Experience

1. Start with strength 0.5-0.6 for your first try. It's the sweet spot where you can hear both the source structure and the new style clearly. Adjust from there.

2. Generate batches of 2-4. Each generation with the same settings will sound different due to randomness. Pick your favorite from a batch rather than trying to get one perfect result.

3. Your caption matters more than you think. At lower cover strengths, the caption increasingly drives the output. Invest time in writing a detailed, specific caption.

4. Match your lyrics to the source structure. If the source has short punchy lines, write short punchy lines. If it has long flowing phrases, match that. Mismatched syllable density creates awkward vocal timing.

5. The LM is skipped in cover mode. Don't worry about LM settings (temperature, top-k, CFG scale, thinking mode) — they have no effect. Cover mode goes straight to the DiT.

6. Duration should match your source. If you set a very different duration than your source audio's actual length, results may be unpredictable.

7. Use repaint to fix sections after covering. Generate a cover, then if one section isn't right, switch to repaint mode to regenerate just that part while keeping the rest intact. Cover then Repaint is a powerful combo.

8. Turbo model works great for covers. The turbo model (default) supports cover mode and produces results in seconds. The base/SFT models also support cover but take longer. Use turbo for exploration and base for final quality if needed.

9. You don't need the LM initialized. Cover mode only needs the DiT and VAE. If you're short on VRAM, you can skip loading the LM entirely and still use cover mode.

10. Try using both Reference Audio and Source Audio. Upload a song as source (for structure) and a different track as reference (for sonic texture). This gives you two independent axes of control.

---

Quick Reference

    ┌─────────────────────────────────────────────────────────────┐
    │                COVER MODE QUICK REFERENCE                   │
    │                                                             │
    │  TASK TYPE:      cover                                      │
    │  REQUIRED:       Source Audio (upload or paste codes)       │
    │  IMPORTANT:      Caption (describes target style)           │
    │  KEY CONTROL:    Audio Cover Strength (0.0 - 1.0)           │
    │                                                             │
    │  STRENGTH GUIDE:                                            │
    │    0.3 - 0.5     Dramatic genre change                      │
    │    0.5 - 0.7     Moderate restyle                           │
    │    0.7 - 0.9     Subtle transformation                      │
    │                                                             │
    │  LM REQUIRED?    No — cover skips the Language Model        │
    │  MODELS NEEDED:  DiT + VAE only                             │
    │  TURBO SUPPORT:  Yes                                        │
    │                                                             │
    │  WORKFLOW:                                                  │
    │    1. Select "cover" task type                              │
    │    2. Upload source audio (or paste codes)                  │
    │    3. Write caption describing NEW style                    │
    │    4. Set cover strength (start at 0.5-0.6)                 │
    │    5. Set duration to match source audio                    │
    │    6. Generate batch of 2-4                                 │
    │    7. Pick favorite, optionally Repaint weak sections       │
    └─────────────────────────────────────────────────────────────┘

[ChuxiJ]

@sigalarm Nice writing. Could you please update this in the docs?

[ChuxiJ]

see: https://www.youtube.com/watch?v=sv4pNrjRh7s

[tester4488]

in my case if use cover strenght > 0.07 i have a noisy distorted output or only noise, Why?

[RayEbb]

I created this guide from #sigilarm for myself as an .md file. I'll upload it here so others may find it useful. If this isn't allowed, please accept my apologies and delete this post.
# ACE-Step Cover Mode — A Complete Guide.md

[giantamoeba]

A great guide indeed, can we have one please for the repaint functionality?

[RayEbb]

A great guide indeed, can we have one please for the repaint functionality?

Thank you, but I didn't make the manual itself. All the credits go to sigilarm . I just copied all the text from his manual, and made it into a Markdown file.