Most teams start monitoring their LLM applications by adding latency tracking. It's the obvious first step -- you add a timer around your API call, send the number to a dashboard, and feel like you have observability. You don't. Latency is useful but it is the least specific signal in the stack. A high latency might mean a slow model, a large context window, a slow retrieval step, a retry, or a network issue. You can't act on it without something else telling you where the problem actually is.
These five signals give you a clearer operational picture. Together they cover the failure modes that actually surface in production LLM applications.
1. Per-request token cost
Not total monthly spend -- per-request, per-flow token cost. This is the signal that lets you understand cost before the invoice arrives.
The calculation is straightforward: (input_tokens * input_rate) + (output_tokens * output_rate). The difficulty is that you need to capture these numbers at the span level, not just at the API response level. A RAG workflow has at least two LLM calls (the retrieval step and the generation step), and possibly more if you're doing query rewriting or iterative refinement. Knowing that "the workflow cost $0.004" is less useful than knowing that the context-assembly step is consuming 60% of that cost because your retrieval is pulling too much.
Track this per-flow and per-user-segment. It will show you exactly which paths are expensive and whether cost is growing as your user base or feature set changes.
2. Input token count trend per flow
This is the early-warning signal for context bloat. Prompts grow. System prompts accumulate instructions. Conversation history gets appended. Retrieved chunks multiply. None of these changes feel significant in isolation, but the combined effect on input token count is often dramatic over weeks of iteration.
Tracking input token count as a time series -- by flow, by endpoint, by user segment -- lets you see this before it becomes a cost or quality problem. A steady upward trend in input tokens for a particular workflow almost always reflects unintentional change: a prompt that grew, a retrieval limit that got bumped, a new instruction added without removing an old one.
Set a threshold alert at some percentage above your baseline. When a flow's average input token count is 20% above its 7-day average, something changed. You want to know what.
3. Retrieval hit rate
If you are building a RAG application, retrieval quality is the primary quality lever -- and it is one of the least-monitored signals. Retrieval hit rate measures how often the retrieved chunks are actually used in the generation step, or more precisely, how often the retrieval step returns relevant content given the query.
Measuring this precisely requires some judgment about what counts as "used," but a practical proxy is to track the ratio of retrieved chunks to final context tokens. If you retrieve 5 chunks and your context window shows that the model used material from 1 of them, that's a retrieval efficiency problem. At scale, wasted retrieval tokens are a direct cost line.
The other relevant retrieval signal is miss rate -- queries where retrieval returns nothing useful, and the model is generating an answer without grounding. These are often the highest-risk responses in any retrieval-augmented system.
4. Prompt version distribution
At any point in time, what percentage of your requests are using which version of your prompt? This sounds like a question you should be able to answer trivially -- just look at your deployment. In practice, for most teams it's harder than it appears.
Prompt templates often exist in multiple places: the codebase, config files, database-stored templates, environment variables, and sometimes hardcoded strings. If you don't instrument which template version is resolved at request time and record it in your trace, you cannot know the distribution. And if you can't know the distribution, you can't diagnose quality regressions that affect only some fraction of requests.
Track the prompt template identifier (a hash or version tag) per span. When you deploy a new version, watch the distribution shift and correlate it with output quality signals if you have them.
5. Error and fallback rate by span type
LLM applications fail silently more than most systems. An API call might return a malformed response that triggers a catch block and falls back to a default. A retrieval step might time out and return an empty result. A parse step might fail and substitute a placeholder. None of these register as errors in your HTTP layer -- they all return 200.
Track the rate at which each span type in your workflows falls back or triggers error handling. Segment by span type (retrieval, LLM call, parse, function call) so you can see which step is failing. A high fallback rate in the parse step usually means your prompt format instructions are not working reliably. A high error rate in retrieval usually means a vector store or index issue.
These five signals, tracked together at the span level, give you the operational foundation that latency alone cannot. They cover cost, efficiency, quality, change detection, and failure mode visibility -- the five things you actually need to run an LLM application with confidence in production.