Voice cloning that holds up in a studio context is an entirely different problem from text-to-speech. Here is the architecture behind the system we built for a music-tech client.
The Model Stack
HuBERT for self-supervised speech embeddings, RMVPE for robust pitch extraction, and a custom RVC-style conversion model trained on the target voice.
FAISS for Timbre Retrieval
Pure model output drifts on long phrases. We index reference timbre vectors in FAISS and retrieve nearest-neighbour timbre at inference time, lifting naturalness scores by 32 percent.
Real-Time GPU Inference
Sub-second turnaround required CUDA streams, persistent worker processes, pre-allocated tensor pools, and FP16 inference.
The Trade-Offs Worth Knowing
Higher pitch-extraction accuracy costs latency. Larger reference sets improve fidelity but balloon memory. We exposed both as per-request parameters.
