Tabular Q-Learning hits a wall when the state space explodes. FrozenLake’s 14 states and Taxi-v3’s 500 states are manageable. Space Invaders is not. Atari frames arrive at 210x160 pixels with three color channels, each pixel ranging from 0 to 255. That works out to 256^100,800 possible states—a number that dwarfs the roughly 10^80 atoms in the observable universe. No Q-table can handle that.
Deep Q-Learning swaps the table for a neural network. The network takes a state as input and outputs a vector of Q-values, one for each possible action. From there, the familiar epsilon-greedy policy picks the move. The architecture is straightforward: a stack of four preprocessed frames goes in, Q-values come out.
That preprocessing matters more than it might seem. The frames get downsized to 84x84 and converted to grayscale, cutting three RGB channels down to one. Color doesn’t add much signal in Atari games, so dropping it saves real computation time. Some games also get cropped to remove screen regions that carry no useful information.
The four-frame stack solves a problem a single frame cannot. Show one frame from Pong and you can’t tell which direction the ball is moving. Stack four consecutive frames and the motion becomes obvious. Temporal information lives in the sequence, not the snapshot.
Hugging Face’s Deep Reinforcement Learning Class introduces this through RL-Zoo, a training framework built on Stable-Baselines that handles the heavy lifting: training scripts, evaluation, hyperparameter tuning, plotting, and video recording. It’s the same practical approach that made the earlier Q-Learning unit work—learn the concept, then run it against a real environment without building every piece of infrastructure yourself. The jump from 500 states to 256^100,800 is the whole story. Tabular methods were never going to scale; function approximation was always the exit ramp.
