LLM Evaluation Platforms: Keep the Decision Contract Portable

The decision: build, buy, extend or reject

Buy a managed evaluation platform when it meets every mandatory data, security and portability gate and its workflow, review, scheduling or governance functions save measured operating effort. Extend the existing test and observability stack when it already preserves the required artifacts and only needs model-aware adapters or graders. Build a narrow internal control plane when the evaluation contract, sensitive data boundary or deployment environment cannot be represented safely elsewhere. Reject a candidate when evidence cannot be exported, reproduced or governed to the consequence level of the decision.

This four-way rule is AccessAllGPT guidance. The decision unit is one configured evaluation path: product and version, runners, models, prompts, tools, dataset, graders, reviewers, storage, integrations, access controls and export format. A polished dashboard or large integration list is not a transferable evaluation result.

Why portability is a current procurement gate

OpenAI’s deprecations page states that deprecation of its Evals platform was announced on June 3, 2026, existing evals become read-only on October 31, 2026, and the dashboard and API are scheduled to shut down on November 30, 2026. OpenAI points users to a migration path. These are current vendor statements for that product; they do not show that every hosted evaluation product will close or that the stated migration preserves every artifact.

The engineering lesson is narrower and durable: evaluation infrastructure changes. A team that owns only dashboard state may lose the exact cases, grader behavior, run provenance or decision history needed to compare a replacement. Treat a scheduled shutdown as a migration deadline, not as evidence that evaluation itself is optional.

Own the evaluation contract before choosing the runner

Store the decision objective, system boundary, cases, input schema, reference evidence, expected and prohibited outcomes, rubric, evaluator rules, repetitions, missing-run policy, thresholds and stop conditions in a versioned format controlled by the team. Bind each run to immutable identifiers for the dataset, candidate configuration and evaluator configuration.

OpenAI and Anthropic both publish vendor guidance that starts with explicit objectives or measurable success criteria and representative cases. That agreement is useful practice guidance, not independent proof of either vendor’s tooling. AccessAllGPT recommends making the contract runner-neutral so the same declared decision can be executed in a local harness, CI job or managed platform.

Require case-level evidence, not a score export

For every attempt, preserve the input or a governed reference, system configuration, output, tool trajectory where applicable, deterministic check results, grader prompt and model, rubric version, human judgment, error state, latency, usage and cost fields available from the system. Keep aggregate calculations reproducible from those case records.

A CSV containing only case ID and final score is insufficient when a disputed verdict cannot be reconstructed. Export must preserve nulls, timeouts, refusals, parser failures, safety denials and retries rather than silently dropping them. Sensitive content should be minimized, access-controlled and retained for a declared period; portability does not justify collecting unrestricted transcripts.

Choose graders by failure mode

Use deterministic evaluators for properties that can be computed reliably: schema validity, exact fields, policy-engine outcomes, tool authorization, executable tests and bounded reference checks. Use qualified human review for consequential, ambiguous or domain-dependent judgments. Use model-based graders for scalable signals only after calibrating them against an appropriate human-reviewed set and preserving grader configuration.

OpenAI and Anthropic describe multiple evaluator or grading methods in their guidance. Those descriptions do not establish a universal hierarchy. Model graders can share blind spots with candidates, be sensitive to prompts and change with model versions; human review can be inconsistent and expensive; deterministic checks can miss semantic quality. Record disagreement instead of forcing every case into false precision.

Separate gates from trade-offs

Declare non-compensable gates for outcomes such as cross-tenant disclosure, unauthorized action, prohibited content, invalid citations, required schema failure or an unreviewable high-consequence result. A candidate that fails a mandatory gate is rejected or constrained even if its average quality is high. Evaluate quality, latency, cost and reviewer effort as trade-offs only after mandatory gates pass.

HELM’s published design separates scenarios and uses multiple metrics to expose trade-offs beyond accuracy. Its historical results do not define a production threshold for another workload. The transferable idea is multi-dimensional evidence; the specific metrics, cases and acceptance boundaries must be justified for the local decision.

Evaluate the configured system, not just the model call

Include retrieval, prompts, memory, tools, policy checks, retries, fallbacks, parsers and human intervention in the candidate identity. Sample failures in dependencies and environment state. A platform that accepts only prompt–response pairs may be useful for a narrow model or prompt comparison but cannot substantiate a decision about a tool-using production system.

Require nested traces or stable links to external traces, plus a documented rule for what the platform observes and omits. The evidence should distinguish model behavior from application control: a safe final answer after an unauthorized tool attempt is not equivalent to preventing the attempt.

Test repeatability and evaluator change

Replay a frozen sample multiple times through the same declared configuration and verify that all attempts are retained. Then change one component at a time—candidate model, prompt, grader model, grader prompt, rubric or parser—and confirm the system creates a new version rather than overwriting history. Recompute aggregates from exported case records and compare them with the platform view.

Expect model outputs and model-based judgments to vary. Repeatability means the experiment and provenance can be repeated and the variation can be inspected; it does not mean identical text. A vendor’s continuous-evaluation feature is workflow capability, not evidence that alerts, samples or graders are valid for the buyer’s risk.

Put data and access boundaries ahead of convenience

Map prompts, responses, retrieved documents, reference answers, traces, human comments and grader inputs as separate data classes. Record storage and processing regions, retention, training-use terms, subprocessors, encryption, access roles, deletion, support access and incident evidence for the exact evaluation features in use. Confirm whether model-based grading sends data to another model provider or project.

Minimize production data and use synthetic or appropriately authorized cases where possible. Separate test-case authors, run operators, reviewers and decision approvers when consequence warrants it. A platform cannot inherit approval merely because the production model provider is already approved; evaluation creates its own copies, users and data paths.

Measure total operating effort with a shadow run

Run the incumbent and candidate path on the same frozen decision contract. Measure setup, case authoring, integration maintenance, run time, queue failures, reviewer effort, verdict correction, export work, incident reconstruction and migration effort. Attribute model and grader usage separately from software fees and internal labor. Compare cost per accepted decision or release gate, not only price per evaluation call.

Do not invent a universal build–buy break-even point. Internal systems carry ongoing schema, runner, security and reviewer-workflow costs; managed products carry subscription, usage, integration, data and exit costs. A short shadow run can bound the local difference but cannot establish long-term total cost without explicit workload and change assumptions.

Run an exit drill before signing

Export a representative project into a documented, non-proprietary or fully specified format. Recreate the cases, candidate configuration, rubric, grader settings and decision calculation in a minimal alternate runner. Verify record counts, case identifiers, missing outcomes, attachments, timestamps, judgments and aggregate calculations. Record what cannot be reproduced and why.

Time the drill and test it without privileged vendor assistance. Check API limits, bulk export, deletion after export, license restrictions and the behavior of proprietary graders. Portability does not require every visual feature to transfer; it requires enough evidence and specification to preserve the decision, audit past results and continue mandatory gates safely.

Procure operations, not unverifiable intelligence

Score concrete capabilities: versioned datasets and rubrics; runner and model adapters; CI and release gates; reviewer assignment and disagreement; case-level provenance; access and retention controls; alerting; documented APIs; bulk export; and support response. Require a live proof against the buyer’s frozen contract and include failed and missing runs.

Treat proprietary automatic insight, “objective” scoring and one-click optimization claims as vendor claims until validated on the workload. Ask what model, prompt, rubric and data produce each judgment; how changes are disclosed; whether old results can be recomputed; and how the product behaves when a grader or provider is unavailable. A feature that cannot be inspected may still aid exploration, but it should not become a mandatory approval gate.

Authorize a bounded adoption

Reject a platform when a mandatory security, data, evidence or exit gate fails. Extend the current stack when it meets the contract and the remaining workflow gap is narrow. Run a bounded managed-platform trial when mandatory gates pass and the shadow run has a named question, workload ceiling, expiry and export plan. Build only the smallest missing control plane when the requirement cannot be bought or extended safely.

Adopt only when the selected path reproduces the decision contract, preserves case-level evidence, passes the exit drill and shows a defensible operating advantage over the incumbent. Re-open the decision when storage, graders, export formats, pricing, ownership, integrations or shutdown terms change. This authorization rule is AccessAllGPT guidance, not a requirement from NIST, HELM, OpenAI or Anthropic.

Disclosures

AccessAllGPT did not use, test, score, rank or receive access to an evaluation platform for this article. OpenAI and Anthropic documentation is included as labeled vendor evidence; neither company reviewed or sponsored this work. HELM and NIST provide research and risk-management context, not product endorsements. No vendor supplied data, paid for placement or received an endorsement. AccessAllGPT Research is operated by NeuralArc, is independent, and is not affiliated with OpenAI. Publication-wide relationships are listed on the disclosures page.