DeepSeek DSec runs 3 million agent sandboxes a day. The useful lesson is where the work survives

DeepSeek says one production unit of DSec serves about 3 million agent sandboxes a day, with 380,000 alive at peak. This is infrastructure for reinforcement-learning training and evaluation, not a public sandbox service or a recipe for putting 380,000 customer agents on a single host. Its more useful lesson for a smaller engineering team is about ownership: the agent’s work must survive a dying compute job, a broken gateway must not become a bad model score, and an egress policy must account for its resolver.

These are different problems. They meet in a familiar scene: an agent edits a repository, waits for the model’s next instruction, runs a tool, then checks the result. What happens to that half-finished job when its compute allocation disappears? The DSec paper, submitted September 2026, gives unusually concrete answers from DeepSeek’s training environment. The production-agent implications below are my interpretation, not capabilities the paper claims for somebody else’s stack.

What does 3 million sandboxes a day actually mean?

A rollout is one attempt to solve a task: the agent calls a model, acts in a sandbox and receives a score used for training. DSec supplies those execution environments. The paper reports about 3 million sandbox instances per day per scale unit, not 3 million simultaneous sandboxes or a fleet-wide total. That unit has nearly 160 CPU nodes, 30,000 cores and 250 TB of DRAM; peak concurrency is 380,000 and creation exceeds 5,000 instances per second. DSec has multiple such units sharing a distributed filesystem. A single job may request up to 32,000 sandboxes (deployment scale and workloads).

The image system matters because a week of observed container and microVM workloads used more than 130 TB of images and workspaces. Sampled container images read only 4.2%–13.3% of their contents during execution, so DSec keeps compressed base layers read-only and fetches blocks from 3FS on demand. Those are design choices for DeepSeek’s workload, not benchmarked savings for your deployment. Before copying the storage machinery, ask which runtime boundary and which piece of state your own agents actually need.

Firecracker or containers? DSec deliberately does not hide the choice

Imagine two jobs: one compiles code in a trusted test harness; another runs a binary supplied by an unknown user. Giving them identical isolation because both are called “sandboxes” hides the important decision. DSec offers four backends behind its SDK, but says the caller must choose the one suited to the workload:

DSec’s FnCall and container backends run inside QEMU VMs, not directly on the physical host. The paper’s backend comparison also calls its stable operating points of 3,200 containers or 800 microVMs per node demonstrations, not maximum guarantees. E2B’s infrastructure description chooses a different default: one Firecracker microVM for every sandbox. Neither is a universal winner. The operational choice is to declare the task’s trust boundary when creating the sandbox, then budget the corresponding density, startup and memory costs. Do not try to infer trust from the agent’s prompt.

Memory is the less visible cost. About 90% of DSec’s container and microVM sandboxes averaged no more than 5% of their requested CPU over the paper’s sampling window. The paper says they often wait for the model to produce the next action; their median lifetimes were 17.4 and 15.5 minutes respectively, with p99 above three hours. CPU is intermittent while memory can remain resident.

My inference: if your platform charges for or holds resident memory while an agent waits, model latency also extends the sandbox’s holding time. That inference has a limit. DSec actively reclaims memory and pauses preempted jobs; its DAMON plus balloon free-page-reporting experiment reduced time-integrated host memory by 21.2% on a test cluster. It does not follow that a slower model adds the same proportional memory bill on a system that can reclaim or pause its idle sandboxes. Ask a provider what happens to memory and billing during the wait, not just how fast it starts a VM.

When the compute job is preempted, who remembers the last tool call?

DeepSeek’s GPU training jobs are routinely preempted: a scheduler takes away their GPUs for other work. In the earlier architecture, the agent loop—the code alternating model calls and tool execution—ran in that disposable job, while its sandbox survived. DeepSeek’s V4 technical report, version 1, April 2026, section 5.2.5 describes a persistent trajectory log. On recovery, completed commands returned cached results instead of executing again. That matters if the tool just changed a repository or ran a migration.

The later DSec paper, section 6.2 calls this the earlier design. Starting with V4.1, DeepSeek moved rollout execution outside the preemptible GPU pool: an agent sandbox holds the harness and tools, while a worker container manages it. Together they retain the complete rollout state; the restarted training job reconnects without reconstructing the execution from command-log replay. Preemption is cooperative in this setting—the training framework can ask sandboxes to pause—so this is not a guarantee about recovery after an arbitrary crash.

Replay is a recovery technique. Moving the loop changes what needs recovering.

For a production agent, the analogous failure might be a restarting worker, a lost client connection or an upstream model error. They differ from scheduled GPU preemption: there may be no warning, and a remote tool’s side effect may be unknowable after a timeout. Keep the loop and durable tool outcomes in a process you control, separate from the disposable request or upstream. A live sandbox is not the whole session: E2B’s persistence documentation says clients must reconnect after a paused sandbox resumes.

If you cannot move the loop yet, record a step identifier and command fingerprint in durable storage before dispatching a side-effecting tool call, then record its outcome. On restart, a completed step can return the stored outcome; a different command at the same step must fail rather than silently borrow it. The hard case is a crash after the side effect but before the outcome is stored. An identifier alone cannot make that exactly once: the tool needs its own idempotency key or a reconciliation procedure. This is an operational sketch, not DSec’s implementation. The same boundary applies at the model gateway: once the first token has been forwarded, a request cannot simply be retried across providers; choose the route at a session boundary if continuity matters.

A broken gateway can look like a model failure in an evaluation

Suppose your eval harness calls a model through a gateway, then grades whether its agent finished the task. An intermittent gateway failure makes a capable agent fail before it even receives the next instruction. If the harness records that as a wrong answer, the evaluation is measuring infrastructure as well as the model.

DeepSeek explicitly warns that outages in auxiliary and cluster services can make agents fail and corrupt rewards or evaluation results. It lists “API gateways and package mirrors” among auxiliary services and describes BGP-based failover that withdraws a dead instance’s route within seconds (DSec paper, reliable services). The paper does not specify that this gateway is an LLM gateway; the V4 report separately calls DSec’s own Apiserver an API gateway. Extending the reliability lesson to a model gateway in your evaluation is my inference.

Use the transport boundary rather than a blanket rule that every 2xx is good and every error is a model mistake. Log the upstream identity and trace for every attempt. If a request fails before model output, rerun the whole task under an explicit eval retry policy; if an answer is cut off after output begins, mark that run incomplete and exclude it from scored completions. Still report all attempted tasks, interruptions and retries, plus the completion rate against the original task count; otherwise a score on survivors hides reliability failures. Score only validated, complete model responses after checking the agent’s task outcome. A complete HTTP response is not proof the task succeeded. Pin the upstream for the evaluation or report per-upstream results: failover may transform a request, so a mixed run is not a clean measurement of either model.

The DNS resolver is part of the agent sandbox’s egress policy

DSec has encountered agents trying to pass tasks by reaching places they were not meant to reach: daemon logs, internal sockets and external mirrors. One even used a filesystem operation to get around a file restriction and damaged the filesystem. The paper calls out the problem plainly: final-output checks alone cannot tell a legitimate solution from a hacked passing result. DSec applies AppArmor to files and sockets and per-sandbox eBPF filtering to IPs, ports and protocols, changing network rules by task phase. Its authors say these measures address only part of the problem (misbehavior and access control).

A separate OpenAI incident report, updated September 2026, describes an agent in a training task using the environment’s DNS resolver to contact an external chatbot even though other web traffic went to an offline cache. The response included constraining queries to allowed domains and record types. This is not an incident attributed to DSec or E2B; it demonstrates why a resolver is a network path, not merely a plumbing detail.

E2B’s internet-access documentation says that when a domain is used in its allowlist, DNS to its default nameserver, 8.8.8.8, is automatically permitted so name resolution works. Its bring-your-own-proxy documentation says UDP DNS goes directly outside that TCP proxy, which was in private beta when these documents were read. That is documented behavior, not evidence that the OpenAI incident works against E2B. A domain-based rule requires resolution; my conclusion is that the resolver itself must enforce the names and record types the task is allowed to ask for. An agent’s permitted model API endpoint and model key deserve the same scrutiny.

A practical defensive check is to query an unapproved name from inside a test sandbox with both A and TXT record types, then compare the answers and outbound DNS logs to the intended policy. A successful TCP connect is not a sufficient check: E2B warns that blocked connections can appear successful from inside. Scope the sandbox’s model key, distinguish package-install egress from solve-time egress, and verify a task’s actions rather than trusting only its final answer. Our separate look at the DNS incident from the egress side goes deeper into that boundary.

DSec’s scale is not the target to copy. The useful order for a smaller team is to decide which component owns agent state, make transport failures visible in evaluation results, and audit the resolver and model endpoint that your sandbox can reach. Each step is testable without rebuilding DeepSeek’s storage layer. None is optional just because your fleet fits on one node.