Benchmarking search APIs has a built-in flaw: give the model a fetch tool and a public dataset with gold labels, and it’ll simply read the answers instead of searching. Keenable’s response is NEEDLE, an open-source benchmark that rebuilds its query set from live sources rather than freezing one. News queries regenerate hourly from RSS feeds and Google Trends; finance, scholar, legal, and deep-tail queries refresh daily from SEC XBRL, arXiv, Europe PMC, CourtListener, and public agent logs. Fifteen search APIs run under one protocol, and every score is measured against a pooled oracle called ultimate.
What makes this interesting is the ceiling. For each query, NEEDLE combines results from every engine into a synthetic oracle, then ranks that combined set by relevance. The gap between a given engine and ultimate tells you whether the problem is ranking or retrieval. If the gap is large, better results existed but the engine failed to surface them. If ultimate itself is weak, the whole market came up short.
Published 7-day means ending 2026-08-28 tell a nuanced story. Finance is nearly solved: Exa hits 0.910, Keenable 0.872, Perplexity 0.871, Google 0.847, against an ultimate of 0.965. Scholar spreads far wider, from Keenable’s 0.774 down to Tavily’s 0.310 against a 0.869 ceiling, because title queries are answerable from metadata while body queries demand actual retrieval. Deep-tail, which draws rare-word queries from real agent-trajectory releases like DeepResearchGym and OpenResearcher, is hardest: Exa leads at 0.557 of ultimate, Keenable follows at 0.470, Bing sits at 0.199.
Latency matters too since agents call search dozens of times per task. Keenable-realtime posts 193 ms p50 latency versus Exa’s 1,876 ms and Bing’s 2,767 ms. That’s the kind of spread that compounds quickly in agentic workflows. The harness runs on a laptop or in CI, needs an OpenRouter key for judging plus one API key per engine tested, and ships per-run artifacts to a Hugging Face dataset. It’s reproducible, MIT-licensed, and the live dashboard is public. The gap between delivered and achievable quality widens as queries approach how agents actually search — which suggests the industry still has real work to do on the messy, ambiguous end of retrieval.
