OpenAI's new audio models can finally take direction

OpenAI just dropped a fresh batch of audio models into its API, and the headliner isn’t just better transcription—it’s steerability. For the first time, developers can tell a text-to-speech model how to speak, not just what to say. The new gpt-4o-mini-tts model accepts natural language instructions like ‘talk like a sympathetic customer service agent,’ which opens the door to voice agents that actually sound appropriate for the context rather than just generically pleasant.

On the speech-to-text side, the company is rolling out gpt-4o-transcribe and gpt-4o-mini-transcribe, both of which outperform the older Whisper models on standard benchmarks. OpenAI says the improvements are especially noticeable in messy, real-world conditions—thick accents, background noise, people talking too fast or too slow. That’s where most transcription systems fall apart, and it’s exactly the kind of reliability you’d need for call center analytics or automated meeting notes that don’t make everyone sound drunk.

The secret sauce is a mix of reinforcement learning and what the company calls ‘extensive midtraining’ on specialized audio datasets. They’ve also gotten better at distillation, squeezing the capabilities of their largest models into the smaller, more affordable gpt-4o-mini variants without sacrificing much conversational quality. The self-play datasets they used for distillation mimic genuine back-and-forth interactions, which should make voice agents feel less stilted.

These aren’t models that generate new, unconstrained voices from scratch—OpenAI is careful to note the TTS system is limited to preset, monitored synthetic voices to avoid deepfake headaches. But the instruction-following capability is a genuine shift. Until now, tuning a voice agent’s tone meant fiddling with SSML tags or swapping between a handful of predefined styles. Natural language instructions make that whole process feel almost quaint. The models are available now in the API for any developer who wants to wire them up.