Microsoft's open-source testing agent hits 92% on its own benchmark, trouncing Copilot

Microsoft just open-sourced something that fixes one of the most annoying gaps in AI coding assistants: unit tests that don’t actually work. The tool, called code-testing-generator, doesn’t just spit out test files when you ask nicely. It researches your repo first, figures out which framework you’re using, writes the tests, runs them, and then tries to break them to make sure they’re real.

The numbers tell a clear story. On Microsoft’s internal benchmark of 152 real-world tasks, the agent completed 140 — that’s 92.1%. Stock GitHub Copilot, using the exact same model and prompts, managed 120, or 78.9%. That’s 63% fewer failures. Where it really shines is the messier stuff: vague prompts like “add unit tests” and tasks targeting a specific code diff. On vague prompts, the agent hit 88.8% versus Copilot’s 66.3%. On diff-targeted tasks, it went 15 for 15. Copilot went 0 for 15.

What’s clever here isn’t raw volume. The agent actually generated 2.3% fewer tests than Copilot (6,963 vs 7,129) while hitting nearly identical line coverage. It was slightly faster too — 359 seconds per task on average compared to 380. The real win is reliability. The agent runs a five-step verification gate that includes a lightweight form of mutation testing: it reasons about small code changes that should make the tests fail, then confirms they do. It also checks for weak assertions and verifies the repo’s own test command can discover what it produced.

This isn’t a hosted service, which matters for teams handling sensitive code. It ships as an agent definition inside the MIT-licensed dotnet/skills repository and runs locally within your existing coding agent. For startups and mid-market teams that don’t have time to encode repository conventions, this fills a real gap. The agent reads existing tests to learn your patterns before writing anything — the kind of grunt work a senior dev would do manually. Regulated industries with legacy test debt should pay attention. So should anyone who’s ever merged a PR with tests that pass locally but fail in CI because nobody wired them up.