Supabase open-sources an AI agent benchmark — and 78% of coding agents flub auth without a skill file

Supabase just gave every AI-coding-pessimist a receipt. The company open-sourced supabase/evals, an Apache-2.0 benchmark that throws coding agents at real Supabase projects — building schemas, debugging failed Edge Functions, fixing broken Row Level Security policies — and scores the wreckage. The results are public on a leaderboard that doubles as an internal regression suite, so you can watch models improve (or not) daily.

The harness doesn’t mess with mocks. It boots actual containerized Supabase stacks and a local CLI, forcing agents to call the real MCP server. Scenarios are drawn from support tickets and GitHub issues, split into a published benchmark suite and a private regression set that refreshes daily to catch known failure modes. Scoring mixes deterministic checks with an LLM-as-a-judge, and agents get exactly one retry before the grade sticks.

The raw pass rates are revealing. Top-tier models like Opus 5 and Kimi K3 aced the Build stage without any skill file loaded. Everyone else needed help. Sonnet 5 jumped from 78% to 100% with skills; GPT-5.6 Sol climbed from 89% to 100%. But the gaps between models without skills are where the real story lives. Agents consistently hand-write migrations instead of using declarative schemas, verify auth manually rather than reaching for the @supabase/server package, and show wildly different appetites for reading documentation. Codex with GPT-5.6 reads roughly eight docs pages per scenario. Claude Code clocks under two and checks docs in fewer than 40% of cases, even when skills are loaded.

Three concrete weaknesses surfaced immediately, and Supabase updated its skill guidance in response. That feedback loop — benchmark reveals a pattern, docs and tooling improve — is exactly what makes this more useful than a synthetic coding contest. For teams in fintech or healthcare where a wrong RLS policy is a security incident, knowing which agent reads the manual and which one wings it isn’t academic. It’s a deployment gate. The framework runs locally with Docker and pnpm, so any team can wire it into CI and stop guessing.