ServiceNow drops SyGra, a free framework to stop your AI dataset nightmares

Let’s be honest: wrangling training data is the miserable, unglamorous part of AI work that nobody wants to talk about. You’ve got a PDF dump that needs to become a clean Q&A set, a shallow instruction dataset that needs more complex reasoning, or a fine-tuned model that now demands DPO preference pairs. Most teams stitch together fragile, bespoke scripts that fall apart the moment the use case shifts. ServiceNow’s new open-source release, SyGra, is a direct hit on this exact pain point.

SyGra is a low-code Python framework designed to be the Swiss Army knife for LLM and SLM data pipelines. It’s not just another tool for generating synthetic Q&A. The architecture specifically targets the chained, multi-step transformations that make data engineering a grind. It handles over a dozen distinct scenarios out of the box: think converting a raw knowledge base into a question-answering format, automatically deepening shallow questions into multi-turn reasoning problems, or translating a German dataset into an English RAG-optimized corpus. It also tackles the tricky jump from supervised fine-tuning (SFT) datasets to Direct Preference Optimization (DPO) by helping generate those crucial preference pairs.

The practical appeal here is the backend flexibility. SyGra plugs into whatever inference engine you’re already running, whether that’s vLLM, Hugging Face’s TGI, Ollama, or Triton. You’re not locked into a specific model provider just to build your data. This lets teams prototype with a local Ollama model and scale to a production vLLM cluster without rewriting their data generation logic. The framework is built around the idea that prompt engineering, not pipeline engineering, should be where you spend your brainpower.

ServiceNow has published a paper laying out the methodology, but the real value is in the repo itself. It includes concrete task examples and a tutorial walking through an image-to-Q&A pipeline, which addresses the painful process of extracting structured data from visuals. The framework doesn’t promise magic; it promises to stop you from reinventing the wheel every single time a new data format challenge lands on your desk. For teams stuck in a cycle of manual curation and brittle custom code, that’s a genuinely useful upgrade.