PixelRAG ditches HTML parsing, retrieves documents as screenshots with 7-query Recall@k eval

Most retrieval pipelines treat every web page as a bag of parsed text, but that assumption breaks the moment you encounter a PDF with complex layouts, an infographic, or a table that defies clean extraction. The tutorial released today builds a complete pixel-native RAG system from scratch — and it doesn’t touch a single HTML tag. Instead, it renders entire documents as images, slices them into overlapping 1024×1024 tiles, and embeds those tiles with SigLIP or CLIP. The result is a retrieval engine that sees documents the way a human would.

The pipeline uses Playwright to capture full-page screenshots at a configurable device scale, then tiles them with a 128-pixel overlap to avoid cutting through important visual elements. Each tile gets a multimodal embedding via SigLIP (or optionally Qwen3-VL for a heavier but potentially sharper representation), stored in a FAISS index for fast approximate search. What makes this genuinely useful is the hybrid scoring layer: OCR-extracted text feeds a BM25 sparse retriever, and the system fuses dense and sparse results using reciprocal rank fusion with a configurable k of 60.

Seven evaluation queries — covering photosynthesis, vector databases, transformers, and RAG itself — test the pipeline against five Wikipedia pages plus an optional synthetic PDF. The system reports Recall@k and mean reciprocal rank, which gives you an honest read on whether pixel-level retrieval actually works. A lightweight residual adapter trained with contrastive learning can further tune the embeddings, though you’ll want to watch for overfitting on such a small corpus.

What’s striking is the optional VLM answer-generation step: once the top tiles are retrieved, you can pass them directly to Qwen2.5-VL-3B-Instruct for grounded answers that reference specific visual regions. That’s a fundamentally different paradigm from chunk-and-embed workflows. The tradeoff, of course, is storage — image tiles are heavier than text vectors — and the latency cost of rendering every page before indexing. But for document collections where layout carries meaning, this approach fills a gap that conventional parsers have ignored for years.