NVIDIA's NeMo Retriever now pulls answers from charts and tables, not just text

NVIDIA just dropped a detailed walkthrough for building a multimodal RAG pipeline using NeMo Retriever, and it’s a genuine step forward for anyone tired of AI that chokes on a bar chart. The tutorial shows how to go from a PDF to a searchable, vectorized knowledge base that understands tables, infographics, and standard text — all without a local GPU for the initial extraction phase.

The process starts lean. You can run offline text extraction using PDFium, which is CPU-only and requires no API keys. That gets you basic text rows in a pandas DataFrame. The real magic, however, kicks in when you connect to NVIDIA’s hosted NIM endpoints. The pipeline then calls a series of specialized vision models: one detects page elements (nemotron-page-elements-v3), another handles OCR, and separate models tackle table structure and graphic elements.

What gets built on the back end is a dense retrieval system stored in LanceDB. The retrieval isn’t just a keyword match; it’s paired with a vision-language reranker (llama-nemotron-rerank-vl-1b-v2) that scores how relevant a chart or table is to your query. Response generation is handled by a 49-billion-parameter Llama model, which also produces inline citations pointing back to the source material. The tutorial even includes a lightweight recall-at-k evaluation to sanity-check whether the right multimodal chunks are surfacing.

For developers, this collapses a bunch of painful plumbing. You’re no longer stringing together separate OCR, embedding, and reranking services. The create_ingestor API orchestrates everything — extraction, chunking, embedding, and storage — in a few method calls. The most practical detail might be the error policy set to “collect,” which keeps the pipeline running even if one page’s graphic element detection times out. That’s the kind of production realism that tutorial code often skips.