AI Plays 20 Questions to Understand Words Better Than Ever

Entity disambiguation—the task of figuring out whether ‘jaguar’ means the car, the animal, or the sports team—has a new state-of-the-art approach, and it works by playing a high-stakes game of 20 questions. A team at OpenAI has built a system that doesn’t try to directly reason about word meanings. Instead, it uses a neural network to ping a pre-selected set of about 100 automatically discovered ‘types’ or categories, asking in essence: is it a vehicle? A feline? A brand? By chaining these probabilistic answers together using Bayes’ theorem, the system rapidly narrows down the possibilities to the correct entity.

The numbers are a significant leap forward. On the CoNLL (YAGO) dataset, the system hits 94.88% accuracy, up from a previous best of 91.70%. On the TAC KBP 2010 challenge, it scores 90.85%, compared to the old guard’s 87.70%. The researchers note that if the type prediction were perfect, accuracy would top out at 98.6-99% on these tasks, suggesting that the bottleneck isn’t the ‘20 questions’ concept itself, but the current fuzziness in the answers. That’s a stark departure from older methods that relied heavily on distributed word representations.

Training this system is a clever feat of data recycling. The process begins by scraping every internal link on Wikipedia to build a massive map of words to all their possible meanings. It then walks the Wikipedia category tree using Wikidata’s knowledge graph to assign a set of categories to each of those entities. The magic is in picking an optimized set of roughly 100 categories that can compactly represent any entity. After converting entities into 100-dimensional binary vectors of category membership, the team used the original Wikipedia links and their surrounding context as training data for a neural network to predict those vectors from text alone. The paper also details a crucial cleanup step using Wikidata to de-noise the training data, fixing common problems like ‘king’ linking to Charles I of England 974 times instead of the concept of a monarch.

What’s particularly elegant is how the system handles the real world’s messy language. A raw Wikipedia scrape would tell you ‘queen’ overwhelmingly refers to the band Queen, thanks to 4,920 links, while linking to the concept of a monarch a measly 32 times. By using Wikidata’s property graph to heuristically transform links into their more generic meaning, the system re-routes those links, boosting the count for ‘monarch’ from 32 to a far more representative 3,553. The search for the perfect type system itself is an intractable optimization problem, so the team used a heuristic search and an evolutionary algorithm to find the best set, paired with gradient descent to train the classifier. The result is a system that learns to ask the right questions, even when the source material is shouting the wrong answers.