Tier 3 to Tier 1 mid-flight: the outage no 5xx will catch

Most teams draw their LLM failure domain — the set of things that can take their service down — around the inference endpoint: if the provider throws 5xx or times out, fail over; otherwise everything is fine. The forum record this year supplies two counterexample classes, on two different providers, and neither one looks like an outage. Both arrive through the business layer — billing tiers and catalog schedules. One pauses real production traffic while every status page stays green; the other retires models on a calendar that owes your roadmap nothing. The failure domain is bigger than the API. This post walks both incident classes from their primary sources, then the code change they imply, then four things to do while everything is still green.

Incident class one: the disappearing tier

Straight from Google’s AI developer forum, and worth reading in the reporter’s own numbers: a production app — voice AI, paying customers — had its billing account downgraded from Paid Tier 3 to Paid Tier 1 with no notification and no action on the owner’s end. The Tier 1 monthly cap (£187.74, against £1,126.46 already spent that month) sat far below actual usage, so the platform paused the service — with a valid payment method, no failed payments, and no policy violations the owner knew of.

Sit with the shape of that for a second. Same API keys, same code, a fraction of the quota — mid-flight. Nothing was down. The bill-payer relationship had changed state, and the API was faithfully enforcing the new state. The threads kept accumulating reports through this week, and the detail that gives the game away: some users reported service coming back after switching their billing to prepaid. The fix lived in the payment relationship, not in anyone’s code. Your payment state machine is part of your system state, whether you model it or not.

Incident class two: the catalog that moves under you

Over on NVIDIA’s forums, users of the free NIM inference endpoints have spent the year living with a moving model catalog. The timeline, assembled from the threads themselves:

No outage here either. Every step was a catalog decision, executed on schedule from the provider’s point of view, abrupt from everyone else’s — and the post-migration detail matters as much as the retirement itself: the replacement wasn’t a drop-in; it was measurably slower. For anything pinned to a retired model, a catalog decision is indistinguishable from downtime. For anything auto-migrated, it’s a silent latency regression.

Why your error handling won’t catch this

Different vendors, different mechanics, same shape:

Your retry logic was built for transient faults: catch the 5xx, back off, try again, escalate to the fallback provider if it persists. Structural failures wear different clothes — and the wardrobe is small enough to enumerate:

In every row the response is well-formed, the connection is healthy, and the retry loop becomes the mechanism by which you don’t notice for another hour. The fix starts with classification — transient errors deserve retries; structural errors deserve a state change:

type FaultClass = 'transient' | 'structural'

const QUOTA_RECOVERY_WINDOW = 60 * 60 * 1000 // 1 hour, in ms — tune to your traffic

// A 429 whose limit doesn't recover on schedule is not congestion — it's a quota
// state change. Retrying harder is exactly the wrong move for that class.
function classify(status: number, throttledForMs: number, errorCode = ''): FaultClass {
  if (status === 429 && throttledForMs > QUOTA_RECOVERY_WINDOW) return 'structural'
  // Only model-lifecycle 4xx counts: a malformed request is your bug, not their catalog.
  if ((status === 404 || status === 400) && /model_(not_found|deprecated|retired)/.test(errorCode)) {
    return 'structural'
  }
  return 'transient'
}

The constant at the top is doing the real work: “how long can a rate limit persist before I stop believing it’s congestion?” Pick a number — and notice the bookkeeping the signature implies: something on your side has to remember when the throttling started, because no provider reports that for you. An hour of unexplained 429s at steady traffic is not a busy afternoon — it’s your account telling you something your inbox hasn’t yet.

Draw the domain wider

Four things to do while everything is green, in rough order of payoff:

1. Inventory your model pins. Every hard-coded model ID is a bet on someone else’s catalog calendar. Know where they all are; the incident is a grep away from being a config change instead of a rewrite.
2. Alert on quota as an SLO, not an error. Honest caveat: providers rarely hand you an “effective rate limit” metric, so build the cheap proxy — count 429s per hour, and record the rate-limit-remaining headers where they exist. A downward step that doesn’t recover is a page-worthy event even at 3% utilization — that’s your tier moving under you. The one-minute version you can run today: pull yesterday’s hourly 429 counts; a step that never recovered means your next stop is the billing console, not the retry config.
3. Keep a second provider warm, not just configured. A fallback that hasn’t served traffic in a month is a fallback with unknown auth state, unknown quota, and an unknown bill. The minimal version: a scheduled probe request every few minutes, on its own API key so the spend shows up as its own line item. Neither this nor item 2 is free to build — which is why this class of plumbing tends to end up in whatever gateway layer you already run — but the minimal versions above are an afternoon.
4. Put deprecation dates on an actual calendar. Providers publish them; the timeline above shows replacements sometimes lag the retirements and don’t always perform like them. A quarterly review of every pinned model’s published sunset beats learning it from a 404.

None of this is exotic. It’s the same discipline you already apply to certificates and disk space, extended to a dependency whose failure modes happen to be denominated in dollars and catalog entries instead of packets. The provider’s billing system — and its catalog calendar — can quietly take your service down. The forum record says both have, for somebody, this year. Budget for them like any other single point of failure.