Routers aren't dead — per-request routing is
A widely shared post-mortem says prompt caching killed the per-request model router, and its arithmetic holds up: a 10x cache discount beats a 2.5x list-price spread every time. We build a router. Here is what we co-sign, what we concede, and where routing actually earns its keep.
A post made the rounds this month under the URL slug “prompt caching killed the multi-model router,” and the Hacker News thread underneath it — 129 points, genuinely sharp on both sides — ran with that framing: routers, as a category, dead. The post itself is more careful than its viral summary, and that distinction is the whole story. We build a router, so let me state our position plainly: the arithmetic in that post-mortem is correct, its actual bottom line — route at the session boundary, not the request — is one we co-sign without reservation, and the thing that died is narrower than the discourse suggests. Per-request routing died. Routing moved up a level. What follows is the honest version of what that means from the vendor’s seat.
The argument, steelmanned
The source is worth reading in full: “Your AI Router Is Trading a 10x Discount for a 2.5x One” (Rajesh Beri, The Daily Brief, 2026-08-01). Its summary paragraph is the whole argument in three lines:
Manifest killed its four-tier LLM router after four months and 7,000 users, and the arithmetic explains why: cache reads bill at 10% of base input, so routing an agent step to a model 2.5x cheaper makes it 3.5x more expensive. Route at the session boundary, not the request.
For anyone who hasn’t internalized the caching mechanics, here is the primer the argument rests on. Prompt caching bills repeated input prefixes at a steep discount — the prefix being the mostly stable front of a request (system prompt, tool definitions, accumulated history) that a provider stores and, on an exact re-send, charges at a fraction of the normal input rate. The post pins the fraction: roughly 10% of base input on the APIs it measured. The catch: the cache lives with one provider and one model. Your inference bill has two discounts available, and you cannot claim both:
| List-price spread | Cache discount | |
|---|---|---|
| Typical size | ~2–3× between adjacent tiers | ~10× on cached input |
| Binds to | nothing — any provider | one provider + one model + one prefix |
| Survives a mid-session model switch? | yes | no — rebuilt from zero |
| Who advertises it | every multi-model routing pitch | the provider’s pricing page |
An agentic session re-sends its enormous prefix dozens of times. Keep every call on one model and most of those tokens bill at the cached rate. Let a router move call seventeen to a “cheaper” model and the new provider sees the prefix cold: you pay full price to rebuild a cache you already had. That is the 10x-for-2.5x trade in the title, and in agentic workloads it loses money on arithmetic you can check with a calculator.
The experience behind the post is specific, not vibes: a four-tier router — classifier sorting each request into simple/standard/complex/reasoning tiers across four vendors — run in production for four months, serving 7,000 users, launched in March and deprecated by June. The cited reasons: classification failures, coherence breaks from mid-session model switches, and cache-invalidation costs. With production cache-hit rates north of 95%, the post’s arithmetic puts the break-even for a mid-session switch at about a 10× price gap — a bar almost no model pair clears.
Where the critique lands on us
Concession first, because it is owed. Per-request cost arbitrage — the thing many routers, ours included at times, were marketed on — has a shrinking domain: batch and stateless traffic, mostly, where there is no prefix worth preserving. Anyone selling per-request cost routing without a cache term in the equation is selling you a spreadsheet error. That deserves to be said by a router vendor, so there it is.
The judgment even fits on one line, worth writing down: switching models mid-session pays off only when the expected savings beat prefix tokens × uncached input rate × expected remaining calls — the cache rebuild you would be buying. If you cannot fill in those three numbers for a workload, you are not routing it; you are gambling on it.
The granularity that was actually load-bearing
Here is what the thread cannot tell you, because it argued about the wrong unit. The decisions that made routing worth deploying were never “which model should answer this specific call.” Put the two granularities side by side and the shapes barely resemble each other:
| Per-request routing | Session-boundary routing | |
|---|---|---|
| Decides | model for this call | model for this session / workload / tenant |
| Decision rate | every request | once per session, revisited on events |
| Cache effect | forfeits it on every switch | preserves it by design |
| Failure modes | misclassification, mid-session coherence breaks | none of those exist at this boundary |
| What it optimizes | list price per call | failover, residency, cost including cache |
| Verdict after this month | shrinking to stateless/batch | strengthened |
Concretely: pick the model once when an agent session opens, keep the session there while its cache is warm, revisit when the session ends or the provider degrades. That is a session boundary — and notice that two of the three reasons in the Manifest post-mortem, misclassification per call and coherence breaking mid-session, simply do not exist at it. The third, cache invalidation, becomes the thing the router is protecting instead of the thing it is causing.
Caching, in other words, adds a constraint the last generation of routers didn’t model: stickiness. A cache-aware router treats the warm prefix as an asset with a value, switches only when the expected gain clears that value plus the rebuild, and otherwise holds. That is not less routing. It is routing with a better objective function — and it is not something application code juggling two SDKs does well at 2 AM. Meanwhile the decisions that were always session-shaped kept their force: which provider is healthy right now, which regions this tenant’s data may touch, what happens when the primary starts timing out. Caching weakened none of them. It raised the price of doing them badly, because a failover that abandons a warm cache now has a number attached, and something has to know it.
Where this leaves us
The honest version of our position, stated as commitments. One: we treat the session, not the request, as the unit of routing — a boundary the caller declares rather than one we guess at. Two: cache stickiness is a first-class term in the objective function, not a footnote. Three: the uncertainty worth stating out loud — cache pricing itself is moving fast. This month’s DeepSeek repricing moved cache-hit input rates by up to an order of magnitude, which means any stickiness math you write down today needs an owner and a review date. The 10x discount that anchors the whole post-mortem is a parameter, not a constant.
The post-mortems are doing the field a service. A tool category that cannot say what its unit of decision is deserves the scrutiny — and on the boundary where ours lives, the case got stronger this year, not weaker.