Sits outside the engine
A proxy in front of stock OpenAI-compatible servers (llama-server). Backend-agnostic, no fork.
The problem, plainly
You run a big model on your own computer. You ask it something that needs a long answer. It starts writing — and halfway through, it runs out of GPU memory. The whole answer dies.
You waited minutes. You get nothing at all — no partial answer, no saved work, just a dead generation. That dead generation is the problem this system solves.
Final document · 26 October 2026
A small program that sits in front of the model you already run. When memory fills up mid-answer, it quietly moves the live answer to a smaller model — and the answer finishes, through the same connection, instead of dying.
Tokens delivered across a handoff in the battery repeats — with zero disconnects.
Real VRAM pressure when the watcher fired in the Oct 21 proof run.
Cost of the handoff itself — warm vs. cold re-prefill. The only pause the client feels.
Engine patches or forks. It works in front of stock servers, unmodified.
Every other system found in the research does part of this. None does all five parts together. That combination — not any single trick — is the claim.
A proxy in front of stock OpenAI-compatible servers (llama-server). Backend-agnostic, no fork.
A watcher monitors measured VRAM per card while a generation is running — not guesses, not error codes.
When pressure crosses the line during a streaming answer, it intervenes then — not on the next request.
The live generation moves from the large primary to a smaller fallback on another server.
Prompt + tokens already emitted are replayed into the fallback, and the client’s original stream stays open as one continuous generation.
Verdict from the uniqueness research: no exact match was found. This is a genuinely novel combination of well-established parts — not a novel primitive. Confidence is moderate-high for “no shipped exact match in public sources,” not absolute. The open checks are listed under Honest limits.
Step 1 of 4
The client asks through the proxy on :8090. The primary model on :8081 streams tokens back. Nothing unusual — and the watcher is already measuring VRAM on every card, the whole time.
Live deployment: three systemd user units — watcher, proxy, fail-open guard. Deliberate split: the proxy handles chat routes only; the decision path (/v1/systemone) stays on :8080 direct, because the proxy speaks chat routes only and routing decisions through it would 404.
The handoff has been proven three times over: the Oct 21 proof run, the Oct 24 battery repeats, and a live test through :8090 on Oct 26. Every number below is from those runs.
| Run | What happened | Key numbers | Result |
|---|---|---|---|
| Proof run 21 Oct | Watcher fired at real pressure; handoff mid-stream; client connection held to completion. | fired 0.9747 · handoff at 49 tokens · 249-token completion · 254 ms | EXIT 0 · FINISH LENGTH |
| Repeat 1 24 Oct battery | Cold-cache handoff. Fallback generated the bulk of the answer after the switch. | handoff 67 · fallback 553 · total 620 · re-prefill 2.724 s cold · fallback 32.9 tok/s · overall 24.0 tok/s in 25.8 s | STREAM HELD |
| Repeat 2 24 Oct battery | Same run, warm cache — the re-prefill cost collapses. | handoff 68 · total 621 · re-prefill 0.444 s warm | STREAM HELD |
| Control direct :8081 | No proxy in the path — the baseline the handoff runs are measured against. | 657 chunks · finish stop · 34.0 s | BASELINE |
| Static run safe settings | 100 tokens direct on :8081; VRAM flat before and after. | 100 tokens · 5.297 s · finish length · 31,336 MiB flat | PASS |
| Gate verdicts | Spillover rows checked against the memory gate. | 30.60 GiB ≥ 30.4 GiB · control N/A | PASS |
Measured KV growth: 0.000000 GB per 1k tokens — 195 tokens, 186 watcher samples, dead flat at 31,336 MiB. The KV cache is preallocated at load, so pressure doesn’t arrive token by token. It arrives as a step jump when another instance loads — the sidecar loading added +5,041 MiB at peak. A watcher built for a slow creep would miss the thing that actually kills generations.
This wasn’t a straight line. The first “success” was fake, and the document keeps it in — because catching it is what made the later proof mean anything.
Long generations on the 35B model pair died on VRAM exhaustion — RemoteDisconnected-class crashes. A long run would simply end. The goal from the start was narrow: save the generation, not make the model fit.
The proxy logged a handoff — but zero post-switch characters reached the client. Root cause: a Python iterator binding bug — for line in resp kept draining the original stream even after resp was rebound to the fallback. Fix: an explicit iterator rebuilt after handoff, plus stricter pass criteria — no error, a handoff, resumed streaming, tokens from more than one model, post-switch characters > 0.
After the fix, 739 characters arrived from the fallback after the switch — tokens genuinely reaching the client post-handoff. The full proof was then redirected to the correct dual-GPU configuration instead of a single-card setup.
Watcher fired at 0.9747, handoff at 49 tokens, 249-token completion, exit 0, connection held. Honest footnote: an evidence record built earlier that same day was marked “proof PENDING” at build time — the proof landing later that night superseded that marking. The label was stale, not the result.
The SGLang / EAGLE-4 spike was killed (SGLang XPU unusable on the A770s). A Nemotron download was scrapped. The d1-3B decision lane went live — separately, on its own path.
The repeats (67→620, 68→621), the control run, the gate verdicts, per-card gate semantics, and the boundary finding: the proxy replays the entire stream with no cap, so the fallback’s 8,192-token context truncates replays of very long primary contexts.
The publish pack was rescoped off a “70B-class” headline to the proven claim only. A wrap-up one-pager shipped. No inflated headline survived contact with the evidence.
Live deployment proven (3 units, fail-open guard, a live chat completion through :8090 returning the 35B model), all 9 measurement cells filled, and the public repo published — initial commit 0e7fbaf plus corrections commit 35cdc4d fixing the proof table, the primary-line wording, and LICENSE detection (verified against the remote: ls-remote = 35cdc4d…). The uniqueness deep research completed the same night — next section.
Four reasons, in order of how much they’d matter to anyone else running models at home.
Before this, a long generation under VRAM pressure died and delivered nothing — minutes of compute, zero output. Now it finishes, on a smaller model, through the same connection. It kills a whole crash class, not a percentage of one.
No engine fork, no patch to llama.cpp, vLLM, or anything else. It sits in front of stock servers — so it’s portable, upgradable, and doesn’t die the next time the engine releases.
The trigger is measured VRAM, not a guess or a schedule. The proxy intervenes only when the alternative is death — it never downgrades a generation that would have survived.
Quality drops to the smaller model and says so. The pause costs 0.44–2.72 s. Very long contexts hit the fallback’s context cap. No silent corruption, no fake continuity, no pretending the model never changed.
Deep research across proxies, inference engines, migration systems, routers, and serving stacks (sources opened 26 Oct 2026). Ranked strongest “someone already did this” first. Every candidate has a piece of it. Each card names the one feature that still separates it. Search or filter to interrogate the field.
OpenAI-compatible proxy in front of llama.cpp / Ollama. Watches RAM, VRAM (via nvidia-smi) and disk, and moves the loaded model up/down a quality ladder (e.g. 7B → 3B) under pressure, with ~2 s warm swaps.
Source: pypi.org/project/ramp-llm/0.2.1
Live migration of running inference. ServerlessLLM migrates only tokens and recomputes KV at the destination; SpotServe preserves request state across spot preemption; Pallas prefills a stable prefix at the target while streaming suffix KV blocks, then resumes decoding. The closest technical analogue to prefix replay.
Sources: ServerlessLLM (arXiv) · SpotServe (arXiv) · Pallas (arXiv) · MorphCloud-LLM
Proxy-level fallback chains: after retries and cooldowns, a failed model group fails over to another model group — including a smaller one — with separate context-window and content-policy fallbacks. The client need not implement retry.
Sources: docs.litellm.ai/docs/proxy/reliability · error reference
When KV-cache space runs out mid-generation, vLLM preempts requests, frees their KV blocks, and recomputes them (V1 default RECOMPUTE) when space returns. SGLang evicts radix-tree leaves; TensorRT-LLM admits-only-if-completable or evicts to maximise batching; llama.cpp’s analogue is load-time fit and context shift.
Source: docs.vllm.ai — optimisation
OpenAI-compatible routing daemon: serves local runtimes first, then “spills to cloud providers only when VRAM pressure or policy requires it.” Alongside RAMP, the only other candidate with real VRAM pressure as a trigger.
RouteLLM picks one model per query before generation, on predicted quality/cost. FrugalGPT cascades post-generation, escalating cheap → strong — the opposite direction. llama-swap swaps on requested model name or idleness, never resource state. Mooncake / LMCache and disaggregated prefill/decode transfer KV at a phase boundary, same model. llm-d / KServe pick a replica pre-generation; their KV “spills” down tiers (HBM → DRAM → SSD), same model.
Sources: RouteLLM (arXiv) · Mooncake (arXiv) · LMCache docs · llm-d / KServe · llama-swap overview
| System | Proxy-level | VRAM trigger | Mid-stream | Smaller model | Prefix replay + held stream |
|---|---|---|---|---|---|
| VRAM Spillover (this) | Yes | Yes | Yes | Yes | Yes — all five |
| RAMP | Yes | Yes | No — between requests | Yes | No |
| ServerlessLLM family | No — system | No — locality / preemption | Yes | No — same model | Token / prefix recompute |
| LiteLLM | Yes | No — provider errors | No — pre-first-token safe | Possible | No — SSE error |
| vLLM preemption | No — engine | No — KV accounting | Yes | No — same model | No — recompute & wait |
| NeuralBroker | Yes | Yes | No — pre-generation | No — cloud | No |
On the name: nothing branded “vram-spillover” as a product or paper was found — no GitHub repo, PyPI or npm package owns it for this concept. But “VRAM spillover” is a common generic phrase for a model overflowing into system RAM at a 10–50× slowdown (Spheron, Servnet), and driver-level projects like emuV use “spill” the same way. That’s a naming-confusion risk — readers will first think “layers spilling to RAM” — not a prior-art bar.
Not counted as a match: a single-author “self-healing routing” blog (dev.to) claims a router “re-sends the request, and resumes streaming” after a provider drops a stream. Trigger is provider failure, not VRAM; “re-sends” reads as restart-from-scratch, not prefix replay; and it’s a blog claim, not product documentation. Closest claim of mid-stream switching found — still not the concept.
What this doesn’t do, and what the research couldn’t settle. Both lists ship with the claim — the claim is stronger for carrying them.
Post-handoff output is the smaller model’s output. That’s the trade, stated plainly — the sceptical reader’s first prediction, and it’s true.
The proxy replays the entire stream with no cap; the fallback’s 8,192-token context truncates replays of very long primary contexts.
Re-prefill costs 0.444 s warm vs. 2.724 s cold. Acceptable latency depends on the fallback already being loaded.
2× Intel Arc A770 (SYCL) only. The design is backend-agnostic; the proof, so far, is not multi-hardware.
Whether any current LiteLLM version stitches a new stream after a mid-stream break, in any configuration, couldn’t be settled from docs alone. No evidence was found that it does so on VRAM pressure with prefix replay. A code-level check of a pinned version would settle it.
RAMP’s page describes a “Voltron” that scales precision mid-stream. The arXiv Voltron found (2607.07046) is about elastic multi-device execution instead. Do not cite Voltron as mid-stream prior art on RAMP’s say-so.
Web-index search is not an exhaustive GitHub code/Discussion search (llama.cpp Discussions are poorly indexed — searches there surfaced only load-time OOM items like issue 28964 and downstream spill reports like hermes-agent 106895, neither a mid-generation fallback proposal). An exact-name PyPI/npm/GitHub registry check for “vram-spillover” is cheap and still open, as is an exhaustive sweep of hosted gateways (OpenRouter, Portkey, etc.). A private or unindexed repo could still exist.
“A backend-agnostic proxy that, on measured per-card VRAM pressure during a live stream, replays prompt + emitted tokens into a smaller model on another stock server and stitches the two streams into one client-visible generation.” None of the searched systems claims that sentence.
For a reader evaluating the work, not just the system:
A production failure was solved at the proxy layer, in front of unmodified servers — portable by construction, with a fail-open guard so the fix itself can never become the outage.
The first “success” was a false positive and was caught by the evidence, not by optimism: pass criteria were tightened, the bug was root-caused to an iterator binding, and every headline number in this document traces to a named run.
An inflated “70B-class” headline was cut to the proven claim; limits and unverified questions ship inside the document; and the novelty claim was stress-tested against the closest prior art before it was made.
Live under systemd with a battery-proven configuration, proven end-to-end three times, and published open-source (Apache-2.0) with bench results and corrections applied in public.