Arize Phoenix gives your AI agents an X-ray for debugging and evals

Building an AI agent is just the starting line. The real work begins when you need to understand why it made a specific decision or hallucinated a fact. That’s where Arize Phoenix comes in. It’s a platform designed to give you an X-ray view into your agent’s step-by-step reasoning, combining tracing and evaluation in one place.

Here’s how it works. You instrument your agent—in this case, a smolagents setup using Hugging Face models and a DuckDuckGo search tool—with OpenTelemetry and the OpenInference standard. A single SmolagentsInstrumentor().instrument() call captures every tool invocation and decision point and sends them to a Phoenix instance, which you can run locally with a single command. Suddenly, a black-box query like “What time is it in Tokyo?” becomes a fully transparent sequence of spans you can inspect and debug.

But seeing the steps isn’t enough. The platform also tackles the evaluation piece, which is often the messiest part of agent development. Phoenix lets you run LLM-as-a-judge evaluations, using a model like OpenAI’s GPT-4o to score the relevance of your agent’s tool outputs. The article walks through setting this up for a DuckDuckGo search tool, showing how to query for specific execution spans and then programmatically assess whether the retrieved information was actually useful. It’s a practical loop: trace the run, spot the weak link, then quantify the damage.

This integration matters because it bridges a critical gap in the agent-building workflow. Tracing without evals is just voyeurism; evals without tracing are guesswork. Arize Phoenix combines them for a framework like smolagents, letting a single developer run a sophisticated observability stack without stitching together five different open-source projects. The catch, as always with LLM-as-a-judge, is that you’re paying for API calls to evaluate your other API calls, and GPT-4o’s judgment isn’t infallible. But it’s a far cry better than manually reading through hundreds of search results.