Google's EnvHarness bends frozen benchmarks to each AI's flaws, boosting scores 9 points

Static benchmarks have a shelf life. An AI agent trains against them, learns their quirks, and eventually they stop teaching anything. Google Cloud AI Research, Washington University, and UNC Chapel Hill are proposing a different approach: stop generating new environments and start reshaping the ones you already have.

EnvHarness does this through three plug-in components that operate strictly on the standard reset() and step() interface. Stage replays a fixed action list to start episodes in different places — hiding a mug in a closed drawer forces search instead of reach. Contract installs hooks that block actions, rewrite responses, or truncate observations. Chain composes two environments under one step budget, so the agent has to satisfy both verifiers at once. Crucially, the reward term is left untouched. The human-built verifier stays intact, and nothing reaches into the simulator backend.

An LLM designer called EnvRigger handles the actual adaptation. It watches five baseline rollouts, diagnoses systemic flaws in the policy, writes wrapper code as real Python, and validates against five fresh rollouts. Bad candidates — trivially solvable or unsolvable — get rejected, with up to five revision rounds per task. Generated code compiles in an isolated subprocess, so failures become logged traces instead of dead runs.

Across ALFWorld, WebArena, SWE-bench Verified, OfficeQA, and SpreadsheetBench, skills mined from reshaped environments beat both controls on held-out tasks. ALFWorld jumps from 62.4 to 68.3, with a 9.0-point gain on out-of-distribution splits. SWE-bench Verified rises from 49.88 to 52.58 while steps drop 9.8%. Perhaps most telling: on WebArena and SpreadsheetBench, skills from unmodified environments actually performed below the no-skill baseline. Reshaping is what makes mining worthwhile at all. Under GRPO on Qwen3-8B-base, RL in reshaped environments hit 87.9 on ALFWorld in-distribution versus 81.4 for originals.

The catch is deployability. EnvHarness ships as Apache-2.0 Python with reproduction drivers for six environments, but it requires a resettable environment. Live user accounts and physical robots are out. That’s a meaningful boundary — this works for simulation-heavy training loops, not production systems where you can’t rewind state.