Towards Customized Multimodal Role-Play

Authors: Chao Tang, Jianzong Wu, Qingyu Shi et al.

arXiv · PDF

Summary

The authors tackle "Customized Multimodal Role-Play" (CMRP)—making a unified model generate both text and images that consistently match a specific character's persona, dialogue style, and visual appearance. They built RoleScape-20, a dataset of 20 characters with persona descriptions, style cues, and text-image interactions. Their method, UniCharacter, uses two-stage training: supervised finetuning on unified multimodal data, then character-specific reinforcement learning to optimize cross-modal consistency. With just 10 images plus interaction examples, the model learns the target character in about 100 GPU hours.

Main takeaways:

• New task: jointly customizing persona, dialogue style, and visual identity across text and image generation
• RoleScape-20 dataset with 20 characters including training/eval data for persona, style, and visual cues
• UniCharacter uses Unified-SFT followed by Character-GRPO (group relative policy optimization)
• Requires only 10 images plus interaction examples, takes ~100 GPU hours
• Substantially outperforms prior approaches on cross-modal consistency

Relevance

Directly relevant to my "What is the assistant persona exactly?" and midtraining behavior installation work—this is essentially persona installation but for multimodal models. Their two-stage training (SFT then RL for consistency) parallels my questions about installation-path equivalence, and their consistency metrics could inform how I measure persona localization and leakage.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses evaluation.

Abstract

arXiv:2605.08129v1 Announce Type: new Abstract: Unified multimodal understanding and generation models enable richer human-AI interaction. Yet jointly customizing a character's persona, dialogue style, and visual identity while maintaining output consistency across modalities remains largely unexplored. To mitigate this gap, we introduce a new task, Customized Multimodal Role-Play (CMRP). We construct the RoleScape-20 dataset comprising 20 characters, including training and evaluation data that cover persona, stylistic descriptions, visual/expressive cues, and text-image interactions. Building on a unified model, we devise UniCharacter, a two-stage training framework containing Unified Supervised Finetuning (Unified-SFT) and character-specific group relative policy optimization (Character-GRPO). Given only 10 images plus corresponding interaction examples, the model acquires the target character and exhibits coherent persona, style, and visual identity in both generated text and images. This process takes about 100 GPU hours. Experiments on the RoleScape-20 dataset show that the proposed method substantially outperforms prior approaches. Ablation studies further validate the effectiveness of our cross-modal consistency design and few-shot customization strategy. We argue that CMRP, coupled with unified modeling, provides a basis for next-generation characterful and immersive interactive agents.