code-and-canvas/pocket-tts-mimi-coreai

Kyutai Pocket-TTS for CoreAI

This repository contains the converted CoreAI (.aimodel) asset for Kyutai Pocket-TTS, a lightweight, high-quality, streaming text-to-speech model capable of real-time voice cloning.

The primary autoregressive Flow Language Model has been compiled and optimized using coreai_torch for high-performance execution on Apple Silicon (NPU/GPU).

---

📦 Model Files

File	Description	Size
pocket-tts-flow.aimodel	CoreAI compiled FlowLM model asset	~380 MB
tokenizer.model	SentencePiece English Tokenizer	~1.2 MB
export_pockettts.py	Source conversion script (PyTorch $\rightarrow$ CoreAI)	~5 KB

---

⚡ Quickstart (Python CoreAI Runtime)

import torch
import pathlib
import sentencepiece as spm
import coreai
from pocket_tts import TTSModel

# 1. Load SentencePiece Tokenizer
sp_tokenizer = spm.SentencePieceProcessor()
sp_tokenizer.load("tokenizer.model")

# 2. Load CoreAI Compiled FlowLM Asset
coreai_model = coreai.load_model("pocket-tts-flow.aimodel")

# 3. Load Base Model for Voice State & Mimi Audio Decoder
tts_model = TTSModel.load_model()
voice_state = tts_model.get_state_for_audio_prompt("alba")  # Or custom voice WAV

# 4. Prepare Text Embeddings
text = "Hello! This is Pocket-TTS running on CoreAI."
tokens = torch.tensor(sp_tokenizer.encode(text), dtype=torch.long).unsqueeze(0)
text_embeddings = tts_model.flow_lm.conditioner(tokens)

# 5. Generate Audio Latents Autoregressively on CoreAI
sequence = torch.full((1, 1, 32), -999.0) # Initial BOS token
generated_latents = []

for step in range(100):
    noise = torch.randn(1, 32)
    # Execute inference on CoreAI
    next_latent, is_eos = coreai_model.run(sequence, text_embeddings, noise)
    generated_latents.append(next_latent)
    sequence = next_latent
    if is_eos.item():
        break

# 6. Decode Latents to 24kHz Audio Waveform
all_latents = torch.cat(generated_latents, dim=1)
audio_waveform = tts_model.mimi.decode_from_latent(all_latents)

---

🎙️ Custom Voice Cloning

Pocket-TTS on CoreAI supports custom voice cloning from any audio sample or .safetensors voice file:

# Clone voice from a local audio file
voice_state = tts_model.get_state_for_audio_prompt("./reference_speaker.wav")

# Or load from HuggingFace predefined voices
voice_state = tts_model.get_state_for_audio_prompt("hf://kyutai/tts-voices/alba-mackenna/casual.wav")

---

🛠️ Export & Re-compilation

To export the model from PyTorch to CoreAI yourself, run:

python export_pockettts.py

This script monkeypatches dynamic slicing and data-dependent guards into 100% static KV cache operations, registers model states as PyTorch buffers, and optimizes the asset via coreai_torch.

---

📜 Citation & Credits

• Original Model by Kyutai Labs
• CoreAI Conversion & Optimization powered by coreai_torch.