OpenAI: Why 'evals' are the missing link for business AI

More than a million businesses are deploying AI, but a gulf remains between adoption and actually getting expected results. OpenAI, drawing from its own internal playbook, is now publicly detailing a fix: contextual evals. These aren’t the frontier evaluations the company uses to stress-test models like GPT-4. Instead, they are painstakingly specific frameworks designed to translate a fuzzy business goal—like ‘convert qualified inbound emails into scheduled demos while staying on brand’—into a measurable, reliable system.

The process begins not with code, but with a small, cross-functional team. OpenAI’s guidance insists that domain experts, such as sales professionals for a lead-qualification tool, must share ownership with technical leads. Together, they map out an entire workflow, defining what success and failure look like at every single decision point. The output is a ‘golden set’ of examples, a living document of expert judgment that serves as the authoritative benchmark. This is a messy, iterative slog, not a one-off prompt engineering session. OpenAI recommends starting by reviewing 50 to 100 outputs from an early prototype to build a taxonomy of errors and their frequencies.

Measurement then moves to a dedicated test environment that mirrors real-world pressure, far beyond a simple prompt playground. The goal is to reliably surface where the system breaks down. OpenAI warns against over-emphasizing superficial rubric items at the expense of core objectives, acknowledging that some crucial qualities are very hard to measure. Sometimes you’ll lean on traditional business metrics; other times, you’ll need to invent entirely new ones. This cross-functional rigor is how the company itself operates, using dozens of contextual evals internally to ensure models perform not just well in a lab, but on a specific workflow.

There’s a refreshing honesty that definitive processes for this have yet to emerge. OpenAI positions this more as a collection of observed best practices than a rigid manual, acknowledging that an eval for a flashy consumer product will look different from one built for internal automation. The call to action is straightforward: business leaders can’t toss a model at a problem and hope for the best. Without a structured eval framework built by the people who understand the work, scaling an AI system is just a gamble on consistency you’ll probably lose.