Thoughts and experiences from my daily interactions with AI.

The Least Reliable Dependency in My API

On one of my projects we added our first AI feature, and I walked into it thinking of the model as one more third-party service. We’d call it, get a response, and return it, the same way we called a payments provider or an email provider, so I figured the integration would be the easy part. The integration really was easy, and unwinding that assumption still took longer than building the feature did.

Every dependency I’d integrated before then failed in ways I already had answers for. It was down, or it was slow, or it sent back an error, and timeouts, retries, and fallbacks all exist because those failures are visible. An LLM is the first dependency I’ve used where slow is the normal case, where the same request gets a different answer each time, where every call has a price, and where it can be wrong while returning a 200. Each of those broke a default I didn’t know I was relying on.

It probably doesn’t belong in the request

Our loading states were built for calls that come back in well under a second, so the first thing we noticed was that a call taking several seconds made people think the app had frozen. We also started hitting our service timeout on longer prompts, and my first instinct was to raise it, which just meant people stared at a spinner for longer before getting the same error. We ended up streaming the response, and that did feel a lot better to use.

Looking back, streaming was the answer to a question I’d skipped, which was whether the call belonged in the request path at all. Once it’s in there, my endpoint is as slow and as available as the provider is on its worst day, and I’ve signed up for that on behalf of everyone who calls me. Streaming makes sense when a person is sitting there watching the answer show up. Anything nobody is waiting on is better off as a background job with a status the client can check, since a slow or failed job is a much smaller problem than a slow or failed request.

The reason I’d settle this early is that it decides the shape of the API. A JSON response, a stream, and a job id you poll are three different contracts, and clients get built against whichever one ships first, so changing my mind later means a migration for everyone downstream.

Retries cost money now

Our retry policy came from the rest of the stack, which was exponential backoff on anything that looked transient. That’s a fine default when a retry is free and the failed attempt was quick. Here the failed call might’ve already burned several seconds, and the retry gets billed like any other call, so during a provider incident the usual policy multiplies both the traffic and the bill at the exact moment things are going badly. What we settled on was one retry after a short delay for a few specific errors, a longer wait on rate limits, and no retry at all for anything that looked like a bad request.

Cost was strange in a second way, because the price follows tokens, and so it depends on what people send more than on how many requests they make. We’d estimated spend from an average input length, and then some people sent inputs way longer than anything we’d modeled. I used to think of input limits as an optimization, but with per-token pricing they’re a product decision about what the feature is for, and somebody has to own that decision along with the retry budget. Left alone, both get set by whoever wrote the HTTP client.

I can’t replay a bug

When a normal dependency returns something odd, I log the request, replay it, and get the same odd thing back. That doesn’t work when the same input produces a different output every time, because by the time someone reports that the AI said something weird, the response they saw is gone unless we kept it. So the logs stopped being a debugging aid and turned into the only record of what the system told a user. We logged the full prompt, the model, the settings, and the full response, tied to the action that triggered the call, and I’d treat that as part of the feature in the same way the endpoint is. The catch is that the log now holds whatever people pasted in, so it needs the same access and retention rules as any other user data.

Caching a deterministic call is an optimization that nobody can observe, but when the output changes every time, caching it means picking a single sample and showing it to everyone who asks the same thing. Sometimes that’s exactly what I want, since a person who asks twice probably expects the same answer and I only pay once. It’s still a decision about how the product behaves, and it’s an easy one to make by accident while trying to save money.

A 200 can be wrong

Honestly, this is the one I had no pattern for, because every other dependency tells me when it fails and my error handling is built on that. A model can return a well-formed response with every field present and the right types, and Pydantic will happily validate it while the content is still wrong. Schema validation checks the shape, and nothing in my stack was checking the meaning.

The way I see it, the design question is how far an unchecked answer gets to travel. If it’s shown to a person who can judge it and ignore it, the blast radius is one confused user. If it gets written to the database or kicks off an action, then a wrong answer becomes data that everything downstream treats as fact, and nobody finds it until much later. So I’d keep model output away from anything other systems trust until something has checked it, whether that’s a rule I can verify in code or a person approving it.

It also changes who gets blamed, because when a payments provider goes down, they have a status page and it’s clearly their outage. When the model gives someone a wrong answer through my API, it’s my API that said it, and the fact that the wrong part came from a vendor doesn’t change who the user holds responsible.

What I’d decide before writing the endpoint

The integration itself was never the hard part, since the client libraries are simple and well documented. The hard part was that my defaults for dependencies were decisions someone had made for a different kind of dependency, and I only noticed each one when it broke.

If I were starting again, I’d make four calls on purpose before writing any code: whether the model sits in the request path, what a retry is allowed to cost, what we record about every call, and how far an answer can go before something checks it. I’d probably still get one or two of them wrong, but at least they’d be written down somewhere a teammate could argue with them.