Tracing LLM and Agent Apps with OpenTelemetry, Plus Audit Logs

GenAI semantic conventions, instrumentation libraries, what to capture and what to keep out of traces, agent span trees, and how audit records and EU AI Act logging differ from traces.

November 1, 2026 · 12 min read

An LLM feature that works in a demo can still fail in production in ways that are hard to see: a prompt change that makes answers longer and costlier, a retrieval step that silently returns the wrong document, an agent that calls the same tool six times. Tracing shows you what happened inside each request. Audit logs prove, later, what the system decided and why. They overlap but are not the same thing, and mixing them up leads to either too little evidence or far too much personal data in your observability vendor. This article covers both. The LLM Tracing Setup Generator writes the OpenTelemetry setup for your stack, and the AI Audit Log Designer builds the audit record.

OpenTelemetry and the GenAI semantic conventions

OpenTelemetry is the vendor-neutral standard for traces, metrics and logs. A trace is a tree of spans; each span has a name, a start and end time, a kind and attributes. The GenAI semantic conventions define what an LLM span should look like so that any backend can recognise it.

The core rules are short:

• The span name is the operation and the model: chat gpt-4.1, embeddings text-embedding-3-small, invoke_agent support-agent, execute_tool lookup_order.
• gen_ai.operation.name and gen_ai.provider.name are required. Well-known operations include chat, text_completion, generate_content, embeddings, invoke_agent, create_agent, execute_tool and retrieval.
• Recommended attributes carry the facts you will query most: gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reasons and gen_ai.response.id.
• Content is opt-in: gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions, and for tools gen_ai.tool.call.arguments and gen_ai.tool.call.result.

Two facts matter when you read older blog posts. First, gen_ai.system was renamed gen_ai.provider.name in semantic conventions 1.37.0; Datadog's LLM Observability, for example, expects spans that follow 1.37 or later. Second, the conventions moved in 2026 to their own repository, semantic-conventions-genai, and every gen_ai.* attribute is still in Development status. Development means the names can change, so pin your instrumentation versions and check release notes before upgrading.

Choosing an instrumentation library

You rarely write these spans by hand for SDK calls. Three families of libraries do it:

• OpenTelemetry GenAI instrumentations. For Python, the OpenTelemetry project publishes opentelemetry-instrumentation-genai-openai, -anthropic, -langchain, -llama-index and others from the opentelemetry-python-genai repository. They replaced the older opentelemetry-instrumentation-openai-v2. For Node, @opentelemetry/instrumentation-openai covers the OpenAI SDK. They are beta releases that track the conventions most closely.
• OpenInference, maintained by Arize, covers OpenAI, Anthropic, LangChain, LlamaIndex and more in Python and TypeScript. It adds its own attributes (such as openinference.span.kind and input.value) that Phoenix and Arize read.
• OpenLLMetry, maintained by Traceloop, offers a one-line SDK and individual instrumentations. Some of its Python packages have names close to official ones, such as opentelemetry-instrumentation-anthropic, so check the publisher.

The Vercel AI SDK has its own integration: in version 7 you register new OpenTelemetry() from @ai-sdk/otel with registerTelemetry, and calls emit invoke_agent, chat and execute_tool spans. Version 6 and earlier used the experimental_telemetry option.

A minimal Python setup with the official OpenAI instrumentation looks like this:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.genai.openai import OpenAIInstrumentor

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
OpenAIInstrumentor().instrument(tracer_provider=provider)

Everything else goes in environment variables, so the same code runs everywhere:

OTEL_SERVICE_NAME=support-assistant
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production
OTEL_EXPORTER_OTLP_ENDPOINT=https://cloud.langfuse.com/api/public/otel
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic%20<base64 of pk:sk>
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.25
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=no_content

Switching backend means changing the endpoint and header, nothing else. That portability is the main reason to prefer OpenTelemetry over a vendor SDK.

Tracing an agent

Agents make traces far more useful, because one user request becomes many model and tool calls. The conventions give a clear shape: one invoke_agent span for the run, a chat span (kind CLIENT) for each model call, and an execute_tool span (kind INTERNAL) for each tool call. The tool spans sit beside the chat spans, in time order, as children of the agent span.

If your framework does not create the agent span, add it yourself around the loop. Set gen_ai.agent.name and gen_ai.conversation.id on it, record error.type when the loop gives up (for example after a maximum number of turns), and mark failed tool calls with an error status. The tracing generator's Agent spans tab has a complete loop in Python and TypeScript.

What to capture, and what to keep out

The default should be metadata only: model, tokens, latency, finish reason, tool names, errors. Those answer most debugging and cost questions. Prompts and responses are different: they often contain names, emails, account numbers and pasted documents, and once they are in a tracing backend they are subject to its retention, access controls and data-processing terms.

Each library has a switch. The OpenTelemetry Python libraries use OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT with no_content (the default), span_only, event_only or span_and_event; the Node OpenAI instrumentation takes true or false. OpenInference uses OPENINFERENCE_HIDE_INPUTS and OPENINFERENCE_HIDE_OUTPUTS. OpenLLMetry records content unless TRACELOOP_TRACE_CONTENT=false, which is the opposite default, so set it explicitly.

If you do capture content, for example in staging or for a sampled share of traffic, mask what you can before it leaves your network. An OpenTelemetry Collector between the app and the backend can rewrite attributes with the transform processor's replace_all_patterns function, so masking works the same for every language. Regex masking catches common formats; it does not make content safe, so combine it with short retention and tight access.

Sampling is the other lever. parentbased_traceidratio keeps a fixed share of traces and makes child spans follow the root's decision, so you never get half an agent run.

Audit logs are a different job

A trace is engineering data: sampled, kept for days or weeks, optimised for latency questions. An audit record answers accountability questions months or years later: who or what acted, on whose behalf, with which model and prompt version, on which input and sources, what policy checks ran, who approved it, and what the system finally did. It must exist for every decision, not a sample, and it must be hard to change.

Good audit records share a few traits:

• Versioned context. Model name and exact version, prompt template id and version, policy versions. Without these you cannot tell whether a behaviour change came from a release.
• Inputs without unnecessary personal data. Store a SHA-256 hash of the input when you only need to prove which input was used, or a redacted copy when people need to read it.
• Tool calls with arguments and results, because that is where an agent changes other systems.
• Human approvals with a pseudonymous reviewer id, role, decision and time.
• A trace id, so an investigation can jump to the detailed trace while it still exists.
• Tamper evidence. An append-only table plus a hash chain, where each record stores the previous record's hash, makes edits and deletions detectable. Copy the latest hash somewhere the database owner cannot change.

Record-keeping obligations

For high-risk AI systems under the EU AI Act (Regulation (EU) 2024/1689), Article 12 requires that the system technically allows the automatic recording of events over its lifetime, with logging that helps identify situations that may present a risk or lead to a substantial modification, supports post-market monitoring and supports the deployer's monitoring. Article 12(3) sets minimum fields for remote biometric identification systems, such as the start and end of each use and the identity of the people who verified results. Article 19 requires providers, and Article 26(6) requires deployers, to keep the logs under their control for a period appropriate to the intended purpose and of at least six months, unless other Union or national law, in particular data protection law, provides otherwise.

The dates matter. The Act originally applied most high-risk rules from 2 August 2026; amendments move this to 2 December 2027 for Annex III systems and 2 August 2028 for Annex I products. Check the current consolidated text on EUR-Lex rather than relying on any summary, including this one. This is a description of the obligations, not legal advice.

Outside the EU, the NIST AI Risk Management Framework is voluntary guidance organised into Govern, Map, Measure and Manage functions; logging and monitoring support all four. Sector rules for finance and health often require longer retention than six months, and data protection law pulls the other way, towards keeping less for shorter. The AI Audit Log Designer shows a checklist of these points against the fields you choose, and the LLM Cost Calculator helps turn the token counts in your traces into money.

Open the LLM Tracing Setup Generator