Find the delay before changing the model
A customer waits for the whole service: document lookup, model response, connected tools, and the final application screen. A model benchmark cannot tell you which of those steps caused the delay. For a business AI pipeline, start with one question: can the team explain where time went when a useful task took longer than expected?
Use common measurements, with a version attached
OpenTelemetry’s current GenAI client inference conventions describe duration, time to the first response chunk, and input and output token measurements. These conventions are marked Development, so check what your instrumentation actually supports and record its version before building dashboards. Our suggested implementation is to connect the model call to the application trace, then measure document retrieval and tool execution as separate steps. Record the workflow version and model identifier so a release comparison has context. See the client inference conventions.
Keep usage accounting separate from distribution charts
The current OpenTelemetry token guidance distinguishes usage counters from histograms that describe individual operations. It also treats cache-read, cache-write, and reasoning categories as subsets of broader token totals. Adding every category together can therefore double-count consumption. Use the appropriate usage counters for totals and distribution charts to find unusually large requests. Token counts support cost estimates; reconcile them against provider billing before describing an estimate as actual spend. See the token accounting guidance.
Observe the workflow without copying its private contents
Operational visibility does not require storing every customer prompt or document. OpenTelemetry’s security guidance recommends collecting only what serves the observability purpose and reviewing the data emitted by instrumentation. Begin with timings, error categories, and necessary operational identifiers. Review logs and exported traces for credentials and personal data, restrict access, and define retention. Where detailed examples are needed for evaluation, keep them in a separately governed review process. See the sensitive-data guidance.
Make one measured improvement
Choose a representative task set and agree what counts as a correct result. Establish a baseline for completion time, failures, and reviewer acceptance. Inspect slow traces before changing a model, shortening context, or rewriting a connector. Compare one change against the baseline using the same tasks and acceptance rules. A shorter response time is useful only if the result still meets the business requirement. This is our proposed pilot approach, not a claim about measured client performance.