LLM Observability: Build the Evidence Layer Before You Buy the Dashboard

The decision: instrument first, then build, buy or combine

Make the decision for one configured AI workflow, not for “all LLMs.” Record one outcome: extend the existing observability stack; buy a specialist AI observability service; use a hybrid with portable telemetry and specialist analysis; run a bounded trial; or make no new purchase. The selected path must answer predeclared operating questions under the workload’s data, latency, reliability, cost and incident-response constraints.

This choice is not a contest between open source and SaaS. Instrumentation, transport, storage, analysis, evaluation and operational response are separate layers. A team can own one and buy another. AccessAllGPT recommends preserving those boundaries so a useful visualization does not silently become the only place where prompts, scores, identities and incident evidence can be reconstructed.

Start with questions a decision or incident must answer

List the decisions telemetry must support before selecting fields: which configured version produced an outcome; where latency accumulated; which provider usage was billed; whether retrieval returned permitted evidence; which tool was requested and executed; which policy denied or approved it; why a fallback ran; whether a critical evaluation failed; and which users, tenants or cohorts were affected. Name the responder and maximum answer time for each question.

Separate four evidence classes: service reliability, economic usage, AI outcome quality and security or governance events. They overlap but should not collapse into one score. A fast, cheap response can still be wrong; a high evaluator score can still hide a prohibited tool call; a trace can be complete while retaining data the organization was not allowed to collect.

What the primary sources establish—and do not

OpenTelemetry’s Generative AI project defines developing conventions for events, exceptions, metrics, model spans and agent spans. Its span guidance covers logical operations across retries and includes fields for provider, requested and response model, error type, prompt version, token usage and time to first chunk. Input messages, output messages and system instructions are opt-in fields with explicit sensitive-data warnings. This is a useful vocabulary under active development, not proof that two libraries emit identical data or that any backend answers production questions.

The W3C Trace Context Recommendation defines HTTP headers and formats that propagate context between services. That can join application, retrieval, model and tool activity into a distributed trace. It does not define AI quality, evaluator correctness or content policy. OWASP logging guidance addresses event selection, sensitive-data exclusion, sanitization, protection and monitoring. NIST AI 600-1 is a voluntary risk-management resource. None of these sources selects a commercial product or validates the scorecard below.

Define a minimum viable evidence contract

For every production decision unit, require a correlation identifier; event time; service and deployment version; workflow, prompt and policy versions; requested and returned model identifiers when available; operation type; attempt and retry count; latency milestones; error class; input and output usage units; cache use when reported; retrieval and tool-call identifiers; authorization and approval result; fallback path; and final application outcome. Record tenant or user identity only through an approved pseudonymous key when the question requires it.

Specify semantics, type, cardinality, allowed values, owner and retention class for every field. Record unknown rather than inventing a value. OpenTelemetry says token usage should not be reported when instrumentation cannot obtain it efficiently unless optional counting is enabled, and its current convention prefers billable counts when both billable and consumed units exist. Preserve the provider receipt or raw usage reference needed to reconcile invoices instead of assuming one token field is universal cost evidence.

Trace the configured system, not only the model call

Begin the trace at the user or workload request and carry context through orchestration, retrieval, model calls, validation, policy checks, tools, approvals, queues, fallback and the resulting application state. A model span without the source permissions, tool side effect or accepted outcome cannot explain whether the system worked. Keep domain outcomes—ticket resolved, field accepted, deployment blocked—distinct from transport success.

Use trace links or documented correlation where asynchronous jobs, batch work or external systems break a direct parent-child chain. Verify propagation with a real route that crosses every intended boundary. W3C Trace Context standardizes propagation, but the deploying team must decide which boundaries may accept incoming context and prevent untrusted baggage or high-cardinality values from becoming authority, sensitive metadata or an uncontrolled cost surface.

Treat content capture as a separate data product

Do not turn on prompt, response, retrieved-document or tool-payload capture merely because a library supports it. For each content class, record the purpose, permitted data, lawful and contractual basis, collection point, redaction or tokenization rule, storage region, encryption, access roles, audit trail, retention and deletion path. Test the policy with secrets, personal data, customer content, malicious log injection and oversized payloads before production.

Prefer metadata, hashes of versioned templates, typed outcome labels and secure references when they answer the question without raw content. Sampling does not make prohibited data permissible, and redaction after export may be too late. OWASP says sensitive data should usually be removed, masked, sanitized, hashed or encrypted and that logs need protection against unauthorized access, modification and deletion. The exact legal and privacy requirements require qualified review.

Connect observability to evaluation without confusing them

Attach evaluator name, version, rubric or test-set version, score, pass rule and provenance to the trace or a linked evaluation record. Preserve deterministic failures and human-review disagreement. Never store only the aggregate score: responders need the configured system, case, output reference and evaluator evidence that produced it. Keep online monitoring thresholds separate from offline release gates even when they share evaluators.

An LLM judge is another configured system with its own model, prompt, data access, errors and cost. Calibrate it against labeled cases and do not let it override non-compensable authorization, privacy or safety checks. Observability can reveal drift in measured distributions or failure counts; it cannot prove an uncaptured failure is absent or convert a subjective evaluator into ground truth.

Price telemetry from emission to investigation

Measure instrumentation labor, SDK maintenance, collectors or gateways, network egress, event and span volume, attribute cardinality, storage, retention tiers, indexing, queries, evaluator calls, alert execution, seats, support, redaction and incident-review time. Apply representative normal, burst, retry, long-context and agent-loop traffic. Report cost per accepted outcome and per investigated incident alongside monthly spend.

Test how sampling changes the questions that can be answered. Head sampling can omit a later failure; tail sampling requires enough buffering and classification to retain it; content sampling creates a different privacy and evidentiary boundary from trace sampling. These trade-offs are AccessAllGPT guidance, not results from the cited sources. Set volume and spend ceilings before a trial so exhaustive traces cannot become an accidental production bill.

Compare existing stack, specialist platform and hybrid

Favor the existing stack when it can ingest the required signals, preserve trace continuity, enforce field-level data policy, join deployment and incident context, run the necessary queries and alerts, and remain operable by the on-call team. Favor a specialist platform trial when AI-specific trace views, prompt or dataset versioning, evaluator workflows, annotation, experiment comparison or review operations would otherwise require substantial bespoke work.

A hybrid is justified when open instrumentation and existing service telemetry provide the system record while a specialist layer supplies a measured analysis or review advantage. Avoid duplicate uncontrolled content stores and conflicting definitions. “OpenTelemetry support” is not a binary procurement answer: verify which signals and convention version are accepted, which fields are transformed or dropped, whether raw export is possible, and whether proprietary features survive an exit.

Run failure and evidence drills before purchase

Replay a production-shaped set containing success, provider timeout, retry, streaming interruption, retrieval miss, permission denial, malformed output, tool rejection, tool failure, evaluator disagreement, fallback and user cancellation. Ask an operator who did not build the instrumentation to answer the predeclared questions within the response target. Record missing fields, ambiguous semantics, query steps and false alerts.

Then run data-control drills: submit seeded secrets and personal data, exercise redaction, revoke access, delete a subject or tenant record where required, export evidence, rotate credentials, lose the telemetry backend and restore a degraded path. A screenshot is not completion evidence. Preserve queries, exported records, deletion results and operator time. Do not use synthetic drills to claim the platform is secure or compliant.

Make portability and exit testable

Pin the instrumentation libraries and convention version, retain your internal field dictionary, and export a representative trace set into a second destination before approval. Verify identifiers, timestamps, parentage, events, links, resource fields and required AI attributes after export. Because OpenTelemetry labels the Generative AI conventions Development, define how schema changes are reviewed, dual-written or migrated rather than treating today’s field names as permanent.

Inventory proprietary evaluators, prompt stores, annotations, datasets, dashboards, alerts and retention features. Put export format, API limits, deletion, termination assistance and post-termination access into the commercial record. Portability is demonstrated by a dated drill with measured loss and labor—not by a logo, collector endpoint or nominal standards claim.

Apply mandatory gates and make the recorded choice

Reject a path if it cannot answer a mandatory incident question, prevent prohibited content collection, enforce access and retention, detect telemetry failure, preserve required audit evidence, meet the response objective or support a tested exit. Choose a bounded trial when gates pass but analysis value, operator effort, cost or portability remains uncertain. Adopt only when the exact configuration clears every gate and beats the incumbent on predeclared decision value after full operating cost.

Record scope, chosen architecture, rejected alternatives, evidence version, content policy, retention classes, sampled traffic, volume and spend ceilings, drill results, unresolved uncertainty, owners, rollback and re-evaluation triggers. Re-evaluate on material workflow, model, provider, tool, evaluator, convention, privacy policy, traffic or contract change. This decision rule is AccessAllGPT guidance; it is not prescribed or empirically validated by OpenTelemetry, W3C, OWASP or NIST.

Disclosures

AccessAllGPT did not use, score, rank or receive access from an LLM observability vendor for this article. OpenTelemetry, W3C, OWASP and NIST did not review or sponsor this work. No vendor supplied data, paid for placement or received an endorsement. AccessAllGPT Research is operated by NeuralArc, is independent, and is not affiliated with OpenAI. Publication-wide relationships are listed on the disclosures page.