VITA-QinYu: Expressive Spoken Language Model for Role-Playing and Singing

Authors: Jiacheng Xu, Heting Gao, Liufei Xie et al.

arXiv · PDF

Summary

The authors built VITA-QinYu, a spoken language model that can do more than just chat — it can role-play different personalities and even sing. The model uses separate tokens for text and audio (with multiple codebooks for richer sound representation) and was trained on nearly 16,000 hours of conversation, role-playing, and singing data. It beats other spoken models both on objective role-playing tasks and on singing quality, while also improving conversational accuracy.

Main takeaways:

• First end-to-end spoken model that handles natural conversation, role-playing (e.g., speaking in a comforting tone), and singing generation in one system
• Uses a hybrid design that keeps text and audio modalities separate to avoid them interfering with each other, while using multi-codebook tokens to capture expressive vocal features
• Outperforms peer models by 7 percentage points on role-playing benchmarks and by 0.13 MOS points (on a 5-point scale) for singing
• Also achieves state-of-the-art on conversational benchmarks, beating prior models by 1.4–5 percentage points
• Code, models, and a streaming full-duplex demo are open-sourced

Relevance

Directly relevant to my persona installation work — this is literally about installing expressive personas (role-playing behaviors) into a language model, though via speech rather than text. The distinction between "natural conversation" and "role-playing" modes maps onto my assistant-vs-character framing, and their training pipeline is essentially behavioral installation at the midtraining/fine-tuning stage.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses benchmark.

Abstract

arXiv:2605.06765v1 Announce Type: new Abstract: Human speech conveys expressiveness beyond linguistic content, including personality, mood, or performance elements, such as a comforting tone or humming a song, which we formalize as role-playing and singing. We present VITA-QinYu, the first expressive end-to-end (E2E) spoken language model (SLM) that goes beyond natural conversation to support both role-playing and singing generation. VITA-QinYu adopts a hybrid speech-text paradigm that extends interleaved text-audio modeling with multi-codebook audio tokens, a design enabling richer paralinguistic representation while preserving a clear separation between modalities to avoid interference. We further develop a comprehensive data generation pipeline to synthesize a total of 15.8K hours of natural conversation, role-playing, and singing data for training. VITA-QinYu demonstrates superior expressiveness, outperforming peer SLMs by 7 percentage points on objective role-playing benchmarks, and surpassing peer models by 0.13 points on a 5-point MOS scale for singing. Simultaneously, it achieves state-of-the-art conversational accuracy and fluency, exceeding prior SLMs by 1.38 and 4.98 percentage points on the C3 and URO benchmarks, respectively. We open-source our code and models and provide an easy-to-use demo with full-stack support for streaming and full-duplex interaction.