Searching for research papers is nothing like searching for regular text. A useful engine needs to nail exact titles and arXiv IDs while also understanding fuzzy queries like “small language models for code generation” — even when those words never appear together in any paper. Papers with Code rebuilt their search around that tension, and the architecture is surprisingly pragmatic.
The system now maintains embeddings for more than 110,000 current papers sourced from arXiv and Daily Papers. It splits cleanly into two paths: an offline batch job that embeds the entire corpus using Hugging Face Jobs, and an online service that only runs the tiny query-embedding step against a low-latency Inference Endpoint. Keyword search comes from PostgreSQL’s full-text capabilities, dense embeddings live in pgvector, and reciprocal rank fusion combines the two. If the endpoint goes cold or unhealthy, search falls back to lexical retrieval instantly.
The team treats embedding format as a versioned API — model revision, output dimension, input-format version, normalization method, and a content hash are all recorded per vector. They settled on Qwen/Qwen3-Embedding-0.6B pinned to an exact revision with 256-dimensional L2-normalized vectors. That 256-dimension choice came from Matryoshka Representation Learning, letting them trade embedding quality against speed and storage costs. Document and query prompts are handled separately, which prevents the subtle failure mode where papers and queries get embedded with the wrong instructions.
This mirrors the hybrid retrieval approach the team honed at ML6 building RAG systems for clients, where combining lexical and vector search consistently beat either alone. Rerankers could push relevance higher, but they add latency and overhead. For a search engine that agents hit through a CLI skill — and that humans hit expecting instant navigational answers — the deliberate split between expensive batch work and a minimal request path is what keeps it fast.
