How transfer learning taught AI to read proteins like English

The protein language models now folding amino acid sequences into three-dimensional structures owe their existence to a problem that plagued deep learning for years: models trained from scratch needed absurd amounts of data to do anything useful. The article walks through exactly why that was, using a clever thought experiment. Show a model training examples in an unfamiliar language and it struggles to learn even basic patterns without hundreds or thousands of samples. Show the same model four English movie reviews, and it immediately grasps the task because it already knows the vocabulary, grammar, and cultural context.

The breakthrough came in 2018 with ULMFiT and BERT, two papers that cracked transfer learning for natural language. The core trick: train a neural network on a text task where data is essentially unlimited, then copy that entire network to a new task and only swap out the output layer. The ULMFiT paper reported performance gains equivalent to having more than 100X as much training data, a staggering result that reshaped the field. Modern large language models have pushed those gains even further.

Why does this work? Because in solving any non-trivial task, neural networks internalize deep structure about their inputs. Visual networks learn edges and curves. Text networks learn grammar. That structural knowledge is not task-specific, which means it transfers. The same principle now drives protein language models: instead of learning from raw amino acid sequences alone, they inherit representations trained on massive protein databases, then fine-tune for specific tasks like structure prediction.

What makes this relevant for biologists is that the ESMFold implementation currently runs on PyTorch only, thanks to OpenFold dependencies, while fine-tuning workflows support both PyTorch and TensorFlow. That’s a practical constraint worth planning around. The conceptual leap from BERT to protein folding is smaller than it looks: both rely on the idea that a model which has seen enough examples of a language, whether English or amino acids, already knows most of what it needs before it ever sees your specific task.