The bill arrived first
In early 2025, Petra and I were running an LLM-powered internal tool for a contract client. It had been in production for about four months. We thought it was working fine. Then the OpenAI invoice arrived.
The number was not catastrophic. But it was about 40% higher than our estimate. We had no idea why.
We spent two days trying to figure it out. We looked at request logs. We counted tokens in a spreadsheet. We added console.log statements to time things. None of it gave us a clear picture. The problem was that we could see inputs and outputs, but we couldn't see the shape of what was happening between them -- which calls were expensive, which retrieval steps were duplicating work, which prompts had quietly grown longer over three months of iteration.
What we actually needed
The tool we wanted didn't exist in a form that worked for us. There were general observability platforms -- Datadog, Honeycomb -- that required significant instrumentation work and treated LLM calls as opaque HTTP requests. There were some early LLM-specific monitoring tools, but they focused on tracing the overall request and returning a latency number. That's not the same as understanding what happened inside a chained LLM workflow.
What we needed was span-level visibility. Not "your API call took 2.4 seconds" but rather: the retrieval step took 0.3s and pulled 4 chunks, the context-assembly step added 1,800 tokens to the prompt, the LLM call itself consumed 2,400 input tokens and 380 output tokens at a cost of $0.0031, and the final parse step failed silently and fell back to a default.
We wanted to see that in a timeline view. We wanted to compare it against yesterday's version of the same workflow. We wanted an alert when the average input token count for a particular flow crossed a threshold we cared about.
Building the tool we wanted
We built a first version in about two weeks -- enough to instrument our own production system. It was rough. The trace storage was a SQLite file on the server. The dashboard was a single HTML page with a hand-rolled table. But it worked, and within a day we had found three separate issues we hadn't known about:
- A retrieval step that was pulling twice as many chunks as it needed because we had raised a limit and forgotten to revert it after testing.
- A system prompt that had grown from 320 tokens to 890 tokens over six weeks of incremental edits, adding roughly $0.0004 per call.
- A prompt template variant we thought we had deprecated that was still active in 8% of requests.
None of these were visible from the outside. From the product perspective, everything was working. From the cost and quality perspective, we were running quietly worse than we should have been.
Why we made it a product
After we had the internal tool running, Daniel joined us and we started showing it to other teams building LLM applications. The reaction was consistent: everyone had the same problem and nobody had a good answer to it.
The teams we talked to were not early-stage experimenters. These were engineers who had shipped LLM features into production, had real users, and were making real API calls. They were flying blind in exactly the way we had been -- relying on invoice-level summaries, manual log grep, and intuition to understand what their systems were doing.
That was the case for making Spanloom a product. Not a category bet, not a market thesis -- just the straightforward observation that a specific problem was widespread and unsolved in a form that was actually usable by engineering teams.
We are still early. The product today handles the core instrumentation, the span timeline, cost tracking, and prompt drift detection. We are working on retrieval miss analysis and more sophisticated alerting. We are building it in production, using Spanloom to observe Spanloom, which has the useful side effect of forcing us to eat every bit of dogfood we ship.
If you are running LLM applications in production and you don't know what they're actually doing, that's what we built this for. You can be up and running in under five minutes.