The Jack of All Trades project, or JAT, is an open-source attempt to build what its creators call a generalist agent — a single Transformer model that can handle everything from Atari games to robotic control to text and image tasks. The work began as an open reproduction of DeepMind’s Gato, the 2022 model that first showed a Transformer could juggle vision-and-language and decision-making tasks under one roof. The JAT team built an open version of Gato’s dataset, then trained multi-modal Transformer models on it with what they describe as several improvements for handling sequential data and continuous values.
The project ships three things: a collection of expert RL agents trained to state-of-the-art performance across Atari, BabyAI, Meta-World, and MuJoCo environments; the JAT dataset, which contains hundreds of thousands of expert trajectories collected from those agents and is available on the Hugging Face Hub; and the JAT model itself, built on EleutherAI’s GPT-Neo architecture. The dataset is genuinely useful — you can load it with a standard load_dataset call and immediately access continuous observations, actions, and rewards.
The architecture’s defining trick is how it embeds different data types. Observations and actions get interleaved in the sequence, with each embedding type handled by a different encoder: CNNs for images, linear layers for continuous vectors, linear projections for discrete values. Text goes through GPT-2 tokenization, images through a ViT-style encoder. The loss is computed per modality — MSE for images and continuous values, cross-entropy for discrete — then averaged. That averaging detail matters more than the authors let on in passing; unequal weighting between action and observation prediction could quietly shape what the model actually learns to prioritize.
Performance lands at 65.8% of expert-level across all four domains, which sounds respectable but hides wide variance. On Atari 57, JAT hits just 14.1% of its expert’s score, or 37.6% of human performance, beating humans on 21 games. That’s a real gap between domains, and it’s the kind of number that should temper excitement about generalist agents. The model mimics expert behavior across many tasks, but the distance between ‘mimics’ and ‘matches’ varies enormously depending on where you look.
