Skip to content

Live2D.md

Latest commit

 

History

History
161 lines (124 loc) · 14.6 KB

Live2D.md

File metadata and controls

161 lines (124 loc) · 14.6 KB
Raw

🎭 Live2D Model Integration & Rigging Guide 🎭

Mascot with Wand

Let's get your Live2D avatar ready for Persona Engine!

1. Overview: What's Inside? 🌸

This document outlines the functionality of the Live2D animation system used in Persona Engine and provides essential guidelines for riggers and designers creating compatible Live2D models. Think of it as the blueprint for bringing your character's digital soul to life! ✨

The system integrates several C# services working together like a well-oiled machine:

  1. EmotionAnimationService: Handles expressions and full-body motions triggered by emotional cues, often originating from an AI personality like Aria (based on detected emojis 😊).
  2. IdleBlinkingAnimationService: Manages default idle animations and automatic, natural-looking eye blinking 👀.
  3. VBridgerLipSyncService: Synchronizes the model's mouth movements 👄 with spoken audio using timed phoneme data and adhering strictly to the VBridger parameter standard.
  4. LLM Integration: An AI model (like the one defined for "Aria") generates text responses containing specific emotion tags (e.g., [EMOTION:🤩]) which are parsed and used by the EmotionAnimationService.

Proper rigging according to the specifications outlined below is crucial for ensuring the model behaves as expected and takes full advantage of the system's capabilities. Let's dive in!

2. Emotion System (EmotionAnimationService) ❤️‍🔥

This service brings your character's feelings to the forefront!

Functionality

  • Triggering: Emotions are primarily triggered by specific emoji tags (e.g., 😊, 😤, 🤩) embedded in text generated by the controlling AI (like Aria). These tags are detected and mapped to corresponding Live2D actions.
  • Mapping: The EmotionMap dictionary within the service defines which Live2D expression and/or motion group corresponds to each recognized emoji.
    • Example: { "😊", new EmotionMapping("happy", "Happy") } maps the 😊 emoji to the expression named "happy" and a random motion from the group named "Happy".
  • Expressions: When an emotion with a mapped expression is triggered, that expression is applied to the model.
    • It remains active for a set duration (EXPRESSION_HOLD_DURATION_SECONDS, default: 3 seconds).
    • After the duration, or if playback stops and no new emotion overrides it, the model attempts to revert to a defined neutral expression (NEUTRAL_EXPRESSION_ID, default: "neutral").
  • Motions: If an emotion maps to a motion group, a random motion from that group is played with high priority (EMOTION_MOTION_PRIORITY, default: PriorityForce). This allows emotion motions to override idle or neutral talking animations.
  • Neutral Talking: During audio playback without a specific emotion motion active, the system can play a random motion from a designated neutral talking group (NEUTRAL_TALKING_MOTION_GROUP, default: "Talking") with normal priority (NEUTRAL_TALKING_MOTION_PRIORITY, default: PriorityNormal).

Emoji -> Expression/Motion Mapping Table

Here's how the default emojis map to your Live2D assets. Your Expression and Motion Group names MUST match these exactly!

Emoji ExpressionId (Required Name) MotionGroup (Required Name) Description / Intended Look
😊 happy Happy General happiness, cheerful smile, bright eyes. Motion: Positive, gentle gestures.
🤩 excited_star Excited Starry eyes, wide smile, high energy. Motion: Bouncy, expressive, maybe clapping.
😎 cool Confident Slight smirk, maybe sunglasses effect (if rigged). Motion: Smooth, confident posture/gestures.
😏 smug Confident Smirk, half-lidded eyes, knowing look. Motion: Similar to cool/confident, maybe a head tilt.
💪 determined Confident Focused eyes, set jaw (subtly!), confident stance. Motion: Strong pose, maybe fist pump.
😳 embarrassed Nervous Blushing heavily, averted eyes, maybe slight frown. Motion: Fidgeting, shrinking posture.
😲 shocked Surprised Wide eyes, open mouth (slightly, avoid lip-sync conflict!), raised eyebrows. Motion: Recoil, jump back.
🤔 thinking Thinking Furrowed brow, looking aside/up, maybe hand on chin. Motion: Head tilt, stroking chin.
👀 suspicious Thinking Narrowed eyes, slight side-eye, neutral/slight frown. Motion: Leaning forward slightly.
😤 frustrated Angry Puffy cheeks (subtle!), angry eyebrows, visible steam/stress effect (if rigged). Motion: Stomping, sharp gestures.
😢 sad Sad Downturned mouth, teary/sad eyes, lowered eyebrows. Motion: Slumped posture, head down.
😅 awkward Nervous Sweat drop effect (if rigged), nervous smile, averted eyes. Motion: Fidgeting, scratching head.
🙄 dismissive Annoyed Eye roll (animate this!), slight frown or unimpressed look. Motion: Head turn away, sighing gesture.
💕 adoring Happy Heart eyes effect (if rigged), soft smile, blushing. Motion: Leaning forward, gentle sway.
😂 laughing Happy Eyes squeezed shut, wide smile (careful with lip-sync!), maybe tears. Motion: Shaking, holding stomach.
🔥 passionate Excited Fiery eye effect (if rigged), intense/determined look. Motion: Energetic, strong gestures.
✨ sparkle Happy Sparkling eye effect (if rigged), bright happy look. Motion: Gentle, positive gestures, maybe a twirl.
N/A neutral N/A The default resting face of the character.
N/A N/A Talking N/A - Used for subtle background movement while speaking neutrally.

Rigging Guide

  • Expressions:
    • Create distinct expressions matching the ExpressionId names in the table exactly.
    • ❗ Crucial: Create a neutral expression named exactly matching NEUTRAL_EXPRESSION_ID (default: neutral). Without this, the model might not revert correctly!
    • ❗ Lip Sync Priority: Expressions should primarily affect eyes, eyebrows, blush, and other non-mouth facial features. Avoid heavily manipulating core lip-sync parameters (like ParamMouthOpenY, ParamJawOpen, ParamMouthForm, ParamMouthPuckerWiden, ParamMouthPressLipOpen) as these will be actively controlled by the VBridgerLipSyncService during speech. Design expressions to complement speech, not override the mouth shape.
  • Motion Groups:
    • Create motion groups matching the MotionGroup names in the table exactly (case-sensitive).
    • Populate these groups with .motion3.json files representing appropriate animations.
    • ❗ Focus on Body/Head: Motions should generally focus on body movements, head tilts/nods/shakes, arm gestures, and overall posture shifts. While they can include facial parameter changes, be aware that mouth parameters will likely be overridden by lip-sync, and eye/brow parameters might be overridden by active expressions. Design motions to work alongside talking and emoting.
    • ❗ Neutral Talking: Create a dedicated motion group for neutral talking animations (subtle head nods, slight weight shifts). Name this group exactly matching NEUTRAL_TALKING_MOTION_GROUP (default: Talking). Essential for keeping the model alive during neutral speech!
  • Priorities: Be mindful of the motion priorities used (PriorityForce for emotions, PriorityNormal for neutral talking). Ensure your idle animations (see next section) use a lower priority (PriorityIdle).
  • Transitions: For smooth transitions between expressions or motions, define FadeInTime and FadeOutTime within the .exp3.json or .motion3.json files in the Live2D editor. See Section 6.

3. Idle & Blinking System (IdleBlinkingAnimationService) 😴

Keeps your character looking alive even when quiet.

Functionality

  • Idle Animation: When no higher-priority motions are active, this service attempts to play random motions from a designated 'Idle' group (IDLE_MOTION_GROUP, default: "Idle") with low priority (IDLE_MOTION_PRIORITY, default: PriorityIdle).
  • Automatic Blinking: This service directly controls the eye-opening parameters (ParamEyeLOpen, ParamEyeROpen) to simulate natural blinking at random intervals (BLINK_INTERVAL_MIN_SECONDS to BLINK_INTERVAL_MAX_SECONDS). The blink follows a defined timing pattern (closing, closed, opening duration). This happens automatically unless an expression forces the eyes shut/open.

Rigging Guide

  • Idle Motion Group:
    • Create a motion group named exactly matching IDLE_MOTION_GROUP (default: Idle).
    • Populate this group with subtle, looping animations (breathing, slight swaying, minor posture shifts).
    • Ensure these motions use PriorityIdle.
  • Eye Parameters:
    • ✅ Crucial: Ensure your model uses the standard Live2D eye parameters: ParamEyeLOpen and ParamEyeROpen.
    • These parameters must control vertical eye opening from 0.0 (closed) to 1.0 (open).
    • Verify smooth closing/opening animation when rigging these parameters.
    • The constants BLINK_CLOSING_DURATION, etc., define the blink speed – rig the parameters smoothly to accommodate this.

4. Lip Sync System (VBridgerLipSyncService) 👄

The magic behind realistic speech animation!

Functionality

  • ❗ Standard: This service specifically targets models rigged according to the VBridger parameter standard. It will not work correctly with models using only basic ParamMouthOpenY or other lip-sync conventions.
  • Input: Relies on TimedPhoneme data (from TTS analysis) providing phonemes (/a/, /i/, /s/) and their timings.
  • Mapping: A detailed _phonemeMap translates phonemes into target PhonemePose structs, defining the desired values for multiple VBridger parameters for that mouth shape.
  • Interpolation & Smoothing: Smoothly transitions between phoneme poses based on audio time using Lerp and smoothing factors, creating natural movement.
  • Parameters Controlled: Directly manipulates a specific set of VBridger parameters.

Rigging Guide

  • !!! CRITICAL: Use VBridger Standard !!!
    • Your model MUST be rigged using the standard VBridger parameter set. Consult VBridger documentation and tutorials if needed.
    • The service explicitly controls these parameters. Ensure they exist and are rigged correctly:
      • ParamMouthOpenY: Vertical mouth opening.
      • ParamJawOpen: Jaw rotation/drop.
      • ParamMouthForm: Lip corners (-1 Frown to +1 Smile).
      • ParamMouthShrug: Upward lip tension.
      • ParamMouthFunnel: Lip pursing/funneling ('oo').
      • ParamMouthPuckerWiden: Horizontal shape (-1 Wide to +1 Pucker).
      • ParamMouthPressLipOpen: Lip separation (-1 Pressed Thin, 0 Touching, +1 Separated/Teeth).
      • ParamMouthX: Horizontal mouth shift (-1 Left to +1 Right).
      • ParamCheekPuffC: Cheek puff inflation (0 to 1).
  • Reference Poses: The InitializeMisakiPhonemeMap_Revised function in the code is your cheat sheet for the target shapes the system expects. For "i" ('ee'): 
    // map.Add("i", new PhonemePose(openY: 0.1f, jawOpen: 0.1f, form: 0.7f, shrug: 0.4f, puckerWiden: -0.9f, pressLip: 0.9f));
// Rig so these values create a visual 'ee' shape: Slightly open, strong smile, slight shrug, very wide, teeth showing.
    Test combinations thoroughly!
  • Parameter Ranges: Adhere to expected ranges (usually 0 to 1 or -1 to +1).
  • Interdependencies: Rig interacting parameters (like MouthOpenY/JawOpen) harmoniously.
  • ❗ Keep Mouth Clear: Remember that expressions and motions should ideally avoid conflicting with these core mouth parameters to allow for clear, uninterrupted lip-sync.

5. LLM Integration (Aria Example) 🧠

Connecting the brain to the face!

Functionality

  • The AI personality (e.g., Aria) generates responses with embedded [EMOTION:emoji] tags based on its prompt.
  • These tags are parsed by the system and trigger the EmotionAnimationService (Section 2).

Rigging Guide

  • No direct rigging actions here, but ensure your rigged expressions/motions accurately reflect the character's personality for the corresponding emojis (e.g., Aria's [EMOTION:🙄] should look genuinely dismissive).

6. General Rigging Best Practices 🛠️

Tips for a smooth experience!

  • Consistency: Use consistent naming for parameters, parts, deformers.
  • Testing: Thoroughly test expressions, motions, and parameters in the Live2D editor and, if possible, with engine testing tools.
  • Organization: Keep motion groups logically organized.
  • Parameter Ranges: Double-check parameter min/max values.
  • Smoothness: Ensure smooth deformations across parameter ranges.
  • VBridger Focus: Prioritize accurate and comprehensive VBridger rigging for the best lip-sync results.
  • ✨ Smooth Transitions (Fade In/Out): For expressions and motions to blend smoothly rather than snapping instantly, you must define FadeInTime and FadeOutTime (in seconds) within their respective .exp3.json or .motion3.json files using the Live2D Cubism Editor. The engine respects these values if they are greater than zero. If set to zero, transitions will be immediate.

By following these guidelines, you can create Live2D models that integrate seamlessly with the PersonaEngine animation system, resulting in expressive, lively, and well-synchronized character performances. Happy rigging! 😊