Running Local AI in a Chrome Extension: The MV3 Architecture That Actually Works

Building a Chrome extension that runs local AI models isn’t just about picking the right library—it’s an architectural puzzle defined by Manifest V3’s strict runtime constraints. Nico Martin’s open-source Gemma 4 Browser Assistant lays out a battle-tested blueprint, and after digging through the codebase, the design choices here are refreshingly pragmatic.

The core insight is deceptively simple: treat your background service worker as the single control plane. All inference happens there, using Transformers.js to run both an LLM (onnx-community/gemma-4-E2B-it-ONNX at q4f16 quantization) and an embedding model (all-MiniLM-L6-v2-ONNX). This isn’t a theoretical preference—it’s a practical necessity. Loading models in the background worker means they’re cached under the extension’s origin and shared across all tabs, dodging the memory bloat and sluggishness you’d get from loading models per-website. The side panel and content scripts are deliberately kept thin: the panel handles chat UI and streaming updates, while the content script is just a DOM bridge for page extraction and highlighting.

Messaging is what makes this separation actually functional, and the project uses a typed contract that’s worth stealing. The side panel fires events like AGENT_GENERATE_TEXT, the background worker runs inference, appends to the conversation history, and then emits MESSAGES_UPDATE back. There’s no direct communication between the side panel and content script—everything routes through background. This isn’t just clean code; it’s what keeps the extension responsive when a model is chewing on a 500-token response.

One detail that tripped up the developer is worth flagging: Manifest V3 service workers can be terminated by Chrome at any time. The extension’s model artifacts are cached, but you’ll still need to handle re-initialization gracefully. The codebase tackles this with consistent KV caching via a custom DynamicCache class, which speeds up text generation on subsequent runs. The overall takeaway? You can absolutely run capable local AI in a browser extension today, but the real engineering isn’t in the model—it’s in the message routing and state management that keeps everything from falling apart when Chrome decides to nap your service worker.