ServiceNow Details AutoSynthData After a 2,000-Task Agent-Training Run Improved Pass@1 by 7.2 Points

ServiceNow published the method and two training results on October 2

ServiceNow CoreAI described AutoSynthData on October 2, 2026 as a pipeline for turning an enterprise agent’s measured failures into new training tasks. In the headline Hybrid run, the target was google/gemma-4-26B-A4B-it and the teacher was Qwen/Qwen3.8-27B. ServiceNow says the system generated 2,000 synthetic samples in about 18 hours, then used them for supervised fine-tuning.

The reported best checkpoint, at epoch 5, improved mean Pass@1 by 7.2 percentage points, which the authors describe as a 35% relative gain. They also report verifier success moving from 63.01% to 68.55% and say the checkpoint closed 59% of the original Pass@1 gap to the reference model. These are ServiceNow-run results in EnterpriseOps-Gym Hybrid; no independent reproduction accompanies the post.

A second domain improved from 18.77% to 27.18%

ServiceNow also reports an ITSM experiment with the same Gemma target and DeepSeek-V4.1-Flash as teacher. AutoSynthData generated 1,994 training samples in 66 hours, and mean Pass@1 reportedly increased from 18.77% to 27.18%. The authors attribute the longer generation time partly to a larger teacher and to running ITSM before later pipeline optimizations.

Two positive domain results establish an internal replication across Hybrid and ITSM, not general performance across every enterprise workflow, model, teacher or training method. The announcement does not provide confidence intervals, per-task paired outcomes, failed-generation counts, compute specifications, token use, model charges or total fine-tuning cost.

The curriculum starts with failures rather than generic prompts

AutoSynthData first runs a target model and stronger teacher on diagnostic tasks. It records the capability being tested, tools and workflow structure, where the target fails, how the teacher succeeds, required final-state properties and dimensions that can vary. Those observations become sanitized capability specification cards.

ServiceNow says the generator does not receive the original evaluation prompts, entities, trajectories or verifier details. It instead creates new prompts, initial states, entity combinations, solution paths and verifiers from the cards. That separation is intended to reduce direct task copying, but AccessAllGPT could not inspect the private cards, generation prompts or resulting corpus to test semantic leakage.

Target, multiply and verifier gates form the production loop

The target phase creates independent core samples around identified gaps. The multiply phase makes variants only from accepted target samples; a multiplied example cannot seed another multiplied example, which limits generation drift. An environment adapter handles state setup, execution, reference replay, deterministic verification, solver runs and profiling while a shared controller coordinates coverage and dataset construction.

For the configuration described, ServiceNow favors candidates solved by the target on no more than one of three trials and by the stronger solver on at least two of three. Positive verification executes the reference trajectory and checks the resulting state. Negative verification mutates expected outcomes and requires those incorrect states to fail. A critic may diagnose and repair a failed candidate, but retries are bounded and repaired tasks must pass the gates again.

Batch review addresses repetition, not just sample validity

Individually executable tasks can still make a narrow training set. AutoSynthData therefore reviews accepted and rejected samples for overrepresented task families, missing capability dimensions, repeated examples and low-yield targets. Its controller reduces generation in crowded regions and redirects effort toward gaps.

That is a useful operational distinction: a passing verifier tests one candidate’s final state, while batch review asks whether the collection offers diverse learning signal. Neither check automatically establishes realism, unbiased coverage or transferable gains. Those properties require review against actual deployment traffic and a held-out evaluation that generation never used.

The public benchmark is not the new training release

EnterpriseOps-Gym is publicly available under Apache-2.0, with task prompts, selected tools, MCP server configuration and SQL-verifier fields across Calendar, CSM, Drive, Email, HR, Hybrid, ITSM and Teams. Its paper describes 1,150 expert-curated tasks, 512 tools and 164 database tables. At the pinned revision, the dataset card’s prose table totals 1,115 tasks while its machine-generated split metadata enumerates 649 examples in the oracle configuration and 637 in each distractor-tool configuration. Teams should therefore bind any reproduction to a revision and reconcile the paper, card and hosted files before comparing scores.

The AutoSynthData announcement uses that environment to generate new training examples, but does not link public AutoSynthData source code, the 2,000- or 1,994-sample corpora, trained checkpoints, generation manifests or a managed service. There is no announced API identifier, product tier, price or availability date. Readers can inspect the benchmark today; they cannot reproduce the full new pipeline from the announced artifacts alone.

The benchmark’s earlier findings explain the focus on planning

The EnterpriseOps-Gym authors previously reported a 37.4% best overall task-success rate among 14 evaluated models and a 53.9% best clean-refusal rate on infeasible tasks. They report that human-authored plans improved execution by 14–35 percentage points, while added distractor tools had little effect. Those results motivated treating strategic planning as the principal training gap.

They remain author-reported benchmark measurements from March, not proof that an AutoSynthData-trained model is safe for deployment. Higher Pass@1 can coexist with policy violations, unintended side effects or poor refusal behavior. Buyers need separate acceptance thresholds for task completion, state integrity, access control, refusal and recovery.

What agent teams should do next

Use the release as a design pattern, not a downloadable product. Start with one stateful environment where final outcomes can be checked independently. Freeze target and teacher versions; isolate training from held-out evaluation; preserve diagnostic tasks; log every generated candidate, repair and rejection; and measure tokens, compute, expert review and elapsed time per accepted sample.

Adopt the pattern when you own a resettable environment and can audit verifiers and final state. Constrain generated data to supervised experiments when policy or side-effect checks are incomplete. Wait for code, corpora and checkpoints if reproducibility is required. Reject a production rollout when improvement is demonstrated only on generation-adjacent evaluation or when the agent can mutate consequential state without deterministic authorization and rollback.

Disclosures

AccessAllGPT did not receive ServiceNow products, model access, generated data, checkpoints, code, compute, a briefing, review or compensation for this article. ServiceNow, Hugging Face, Google, Qwen and DeepSeek did not sponsor, review or endorse it. AccessAllGPT Research is operated by NeuralArc, is independent, and is not affiliated with ServiceNow or organizations cited. Publication-wide relationships are listed on the disclosures page.