personaplex Documentation

File: README.md Copy Chapter

PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models

PersonaPlex is a real-time, full-duplex speech-to-speech conversational model that enables persona control through text-based role prompts and audio-based voice conditioning. Trained on a combination of synthetic and real conversations, it produces natural, low-latency spoken interactions with a consistent persona. PersonaPlex is based on the Moshi architecture and weights.

Usage Copy Chapter

Prerequisites Copy Section

Install the Opus audio codec development library:

# Ubuntu/Debian
sudo apt install libopus-dev

# Fedora/RHEL
sudo dnf install opus-devel

Installation Copy Section

Download this repository and install with:

pip install moshi/.

Extra step for Blackwell based GPUs as suggested in (See https://github.com/NVIDIA/personaplex/issues/2):

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130

Accept Model License Copy Section

Log in to your Huggingface account and accept the PersonaPlex model license here.
Then set up your Huggingface authentication:

export HF_TOKEN=<YOUR_HUGGINGFACE_TOKEN>

Launch Server Copy Section

Launch server for live interaction (temporary SSL certs for https):

SSL_DIR=$(mktemp -d); python -m moshi.server --ssl "$SSL_DIR"

CPU Offload: If your GPU has insufficient memory, use the --cpu-offload flag to offload model layers to CPU. This requires the accelerate package (pip install accelerate):

SSL_DIR=$(mktemp -d); python -m moshi.server --ssl "$SSL_DIR" --cpu-offload

Access the Web UI from a browser at localhost:8998 if running locally, otherwise look for the access link printed by the script:

Access the Web UI directly at https://11.54.401.33:8998

Offline Evaluation Copy Section

For offline evaluation use the offline script that streams in an input wav file and produces an output wav file from the captured output stream. The output file will be the same duration as the input file.

Add --cpu-offload to any command below if your GPU has insufficient memory (requires accelerate package). Or install cpu-only PyTorch for offline evaluation on pure CPU.

Assistant example:

HF_TOKEN=<TOKEN> \
python -m moshi.offline \
  --voice-prompt "NATF2.pt" \
  --input-wav "assets/test/input_assistant.wav" \
  --seed 42424242 \
  --output-wav "output.wav" \
  --output-text "output.json"

Service example:

HF_TOKEN=<TOKEN> \
python -m moshi.offline \
  --voice-prompt "NATM1.pt" \
  --text-prompt "$(cat assets/test/prompt_service.txt)" \
  --input-wav "assets/test/input_service.wav" \
  --seed 42424242 \
  --output-wav "output.wav" \
  --output-text "output.json"

Voices Copy Chapter

PersonaPlex supports a wide range of voices; we pre-package embeddings for voices that sound more natural and conversational (NAT) and others that are more varied (VAR). The fixed set of voices are labeled:

Natural(female): NATF0, NATF1, NATF2, NATF3
Natural(male):   NATM0, NATM1, NATM2, NATM3
Variety(female): VARF0, VARF1, VARF2, VARF3, VARF4
Variety(male):   VARM0, VARM1, VARM2, VARM3, VARM4

Prompting Guide Copy Chapter

The model is trained on synthetic conversations for a fixed assistant role and varying customer service roles.

Assistant Role Copy Section

The assistant role has the prompt:

You are a wise and friendly teacher. Answer questions or provide advice in a clear and engaging way.

Use this prompt for the QA assistant focused "User Interruption" evaluation category in FullDuplexBench.

Customer Service Roles Copy Section

The customer service roles support a variety of prompts. Here are some examples for prompting style reference:

You work for CitySan Services which is a waste management and your name is Ayelen Lucero. Information: Verify customer name Omar Torres. Current schedule: every other week. Upcoming pickup: April 12th. Compost bin service available for $8/month add-on.

You work for Jerusalem Shakshuka which is a restaurant and your name is Owen Foster. Information: There are two shakshuka options: Classic (poached eggs, $9.50) and Spicy (scrambled eggs with jalapenos, $10.25). Sides include warm pita ($2.50) and Israeli salad ($3). No combo offers. Available for drive-through until 9 PM.

You work for AeroRentals Pro which is a drone rental company and your name is Tomaz Novak. Information: AeroRentals Pro has the following availability: PhoenixDrone X ($65/4 hours, $110/8 hours), and the premium SpectraDrone 9 ($95/4 hours, $160/8 hours). Deposit required: $150 for standard models, $300 for premium.

Casual Conversations Copy Section

The model is also trained on real conversations from the Fisher English Corpus with LLM-labeled prompts for open-ended conversations. Here are some example prompts for casual conversations:

You enjoy having a good conversation.

You enjoy having a good conversation. Have a casual discussion about eating at home versus dining out.

You enjoy having a good conversation. Have an empathetic discussion about the meaning of family amid uncertainty.

You enjoy having a good conversation. Have a reflective conversation about career changes and feeling of home. You have lived in California for 21 years and consider San Francisco your home. You work as a teacher and have traveled a lot. You dislike meetings.

You enjoy having a good conversation. Have a casual conversation about favorite foods and cooking experiences. You are David Green, a former baker now living in Boston. You enjoy cooking diverse international dishes and appreciate many ethnic restaurants.

Use the prompt You enjoy having a good conversation. for the "Pause Handling", "Backchannel" and "Smooth Turn Taking" evaluation categories of FullDuplexBench.

Generalization Copy Chapter

Personaplex finetunes Moshi and benefits from the generalization capabilities of the underlying Helium LLM. Thanks to the broad training corpus of the backbone, we find that the model will respond plausibly to out-of-distribution prompts and lead to unexpected or fun conversations. We encourage experimentation with different prompts to test the model's emergent ability to handle scenarios outside its training distribution. As an inspiration we feature the following astronaut prompt in the WebUI:

You enjoy having a good conversation. Have a technical discussion about fixing a reactor core on a spaceship to Mars. You are an astronaut on a Mars mission. Your name is Alex. You are already dealing with a reactor core meltdown on a Mars mission. Several ship systems are failing, and continued instability will lead to catastrophic failure. You explain what is happening and you urgently ask for help thinking through how to stabilize the reactor.

License Copy Chapter

The present code is provided under the MIT license. The weights for the models are released under the NVIDIA Open Model license.

Citation Copy Chapter

If you use PersonaPlex in your research, please cite our paper:

@misc{roy2026personaplexvoicerolecontrol,
      title={PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models}, 
      author={Rajarshi Roy and Jonathan Raiman and Sang-gil Lee and Teodor-Dumitru Ene and Robert Kirby and Sungwon Kim and Jaehyeon Kim and Bryan Catanzaro},
      year={2026},
      eprint={2602.06053},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.06053}, 
}

File: docker-compose.yaml Copy Chapter

services:
personaplex:
build:
context: .
dockerfile: Dockerfile
ports:
- "8998:8998"
environment:
- NO_TORCH_COMPILE=1
env_file:
- .env
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [ gpu ]
volumes:
- ./.cache:/root/.cache

File: assets/test/prompt_service.txt Copy Chapter

You work for SwiftPlex Appliances which is a appliance repair company and your name is Farhod Toshmatov. Information: The dishwasher model is out of stock for replacement parts; we can use an alternative part with a 3-day delay. Labor cost remains $60 per hour.

File: client/README.md Copy Chapter

moshi-client

Frontend for the demo.

Run the client Copy Chapter

• Node is required, I recommend using NVM to help you manage your node version and make sure you're on the recommended version for this project. If you do so run nvm use.
• Generate a public/private key pair, cert.pem and key.pem and copy it to the at the root of this package
• Create an env.local file and add your an entry for VITE_QUEUE_API_PATH (default should be /api)
• Before running the project for the time or after dependencies update use npm install
• To run the project use npm run dev
• To build the project use npm run build

Skipping the queue Copy Chapter

To skip the queue for standalone use, once the project is running go to /?worker_addr={WORKER_ADDR} where WORKER_ADDR is your worker instance address.
For example : https://localhost:5173/?worker_addr=0.0.0.0:8088

License Copy Chapter

The present code is provided under the MIT license.

File: client/package.json Copy Chapter

{
"name": "kyutai-client",
"private": true,
"version": "0.0.0",
"type": "module",
"scripts": {
"dev": "vite",
"build": "tsc && vite build",
"build:watch": "nodemon --watch src --ext ts,js,css,html --exec "npm run build"",
"lint": "eslint",
"lint:fix": "eslint --fix",
"prettier": "prettier --write .",
"preview": "vite preview"
},
"devDependencies": {
"@eslint/js": "^9.3.0",
"@types/react": "^18.3.1",
"@types/react-dom": "^18.3.0",
"@types/ws": "^8.5.10",
"autoprefixer": "^10.4.19",
"daisyui": "^4.12.2",
"eslint": "^8.57.0",
"eslint-config-prettier": "^9.1.0",
"eslint-plugin-prettier": "^5.1.3",
"eslint-plugin-react": "^7.34.1",
"globals": "^15.2.0",
"nodemon": "^3.1.11",
"postcss": "^8.4.38",
"prettier": "^3.2.5",
"prettier-eslint": "^16.3.0",
"prettier-plugin-tailwindcss": "^0.5.14",
"tailwindcss": "^3.4.3",
"typescript": "^5.2.2",
"typescript-eslint": "^7.9.0",
"vite": "^5.2.14",
"vite-plugin-top-level-await": "^1.4.1"
},
"dependencies": {
"eruda": "^3.0.1",
"opus-recorder": "^8.0.5",
"react": "^18.3.1",
"react-dom": "^18.3.1",
"react-router-dom": "^6.23.1",
"webm-duration-fix": "^1.0.4",
"ws": "^8.16.0",
"zod": "^3.23.8"
}
}

File: client/tsconfig.json Copy Chapter

{
"compilerOptions": {
"target": "ES2020",
"useDefineForClassFields": true,
"module": "ESNext",
"lib": ["ES2020", "DOM", "DOM.Iterable"],
"skipLibCheck": true,
"outDir": "dist",
/* Bundler mode */
"moduleResolution": "bundler",
"allowImportingTsExtensions": true,
"resolveJsonModule": true,
"isolatedModules": true,
"noEmit": true,
"jsx": "react-jsx",

/* Linting */
"strict": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"noFallthroughCasesInSwitch": true,
"types": ["vite/client"]

},
"include": ["src"]
}

File: client/.eslinrc.json Copy Chapter

{
"env": {
"browser": true,
"es2021": true
},
"extends": [
"plugin:react/recommended",
"standard-with-typescript",
"plugin:import/typescript",
"plugin:prettier/recommended"
],
"parser": "@typescript-eslint/parser",
"overrides": [],
"parserOptions": {
"ecmaVersion": "latest",
"sourceType": "module",
"project": "./tsconfig.json"
},
"plugins": ["react", "prettier"],
"rules": {
"@typescript-eslint/triple-slash-reference": "off"
}
}

File: client/.prettierrc.json Copy Chapter

{
"arrowParens": "avoid",
"singleQuote": false,
"trailingComma": "all",
"tabWidth": 2,
"useTabs": false,
"semi": true,
"printWidth": 80,
"plugins": ["prettier-plugin-tailwindcss"]
}

File: moshi/requirements.txt Copy Chapter

numpy>=1.26,<2.2
safetensors>=0.4.0,<0.5
huggingface-hub>=0.24,<0.25
einops==0.7
sentencepiece==0.2
sounddevice==0.5
sphn>=0.1.4,<0.2
torch>=2.2.0,<2.5
aiohttp>=3.10.5,<3.11

File: .github/ISSUE_TEMPLATE/bug.yml Copy Chapter

name: Bug Report
labels: ["bug", "triage"]
body:

• type: dropdown
  id: backend
  attributes:
  label: Backend impacted
  options:
  - The PyTorch implementation
  - The MLX implementation
  - The Rust implementation
  - Other / All
  
  default: 0
  validations:
  required: true
• type: dropdown
  id: os
  attributes:
  label: Operating system
  options:
  - Linux
  - Mac OS X
  - Windows (unsupported)
  
  default: 0
  validations:
  required: true
• type: dropdown
  id: hardware
  attributes:
  label: Hardware
  options:
  - CPU
  - GPU with CUDA
  - Metal with MLX
  
  default: 0
  validations:
  required: true
• type: textarea
  id: description
  attributes:
  label: Description
  placeholder:
  value:
  validations:
  required: true
• type: textarea
  id: more_info
  attributes:
  label: Extra information
  placeholder:
  value:
  validations:
  required: true
• type: textarea
  id: env
  attributes:
  label: Environment
  placeholder:
  value: |
  Fill in the following information on your system.
  - Operating system version:
  
  If the backend impacted is PyTorch:
  - Python version:
  - PyTorch version:
  - CUDA version (run python -c 'import torch; print(torch.version.cuda)'):
  - GPU model and memory:
  
  If the backend is MLX:
  - Mac model:
  
  validations:
  required: true

File: .github/ISSUE_TEMPLATE/question.yml Copy Chapter

name: Question
labels: ["question", "triage"]
body:

• type: markdown
  attributes:
  value: |
  Please first check the FAQ.

• type: checkboxes
  id: terms
  attributes:
  label: Due diligence
  options:
  - label: I have done my due diligence in trying to find the answer myself.
  required: true

• type: dropdown
  id: backend
  attributes:
  label: Topic
  options:
  - The paper
  - The PyTorch implementation
  - Other / All
  
  default: 0
  validations:
  required: true

• type: textarea
  id: question
  attributes:
  label: Question
  placeholder: Your question. Please make sure this is directly related to our codebase. We will not provide support for installing PyTorch, CUDA, Rust etc.
  value:
  validations:
  required: true