We can't find the internet
Attempting to reconnect
Something went wrong!
Hang in there while we get back on track
Long runs: give a blunt goal, then a structured brief
Simon Willison built Battle Drones, a browser RTS, mostly with Claude Code. Two of his prompts are worth reusing.
- A vague goal plus orchestration. He sent Opus 5.5 the prompt "This game isn't fun enough yet. Make it a LOT more fun." in Ultracode mode, where Claude Code runs "dynamic workflows": JavaScript scripts that orchestrate many subagents at once . The run took from evening until late the next morning . Claude ran more than a thousand simulated playtests and used the results to rebalance units and add features. In 1,200 AI-vs-AI matches, games still undecided after 25 minutes fell from 14% to 0.9% .
- Research and critique before building. When the game's AI ignored infantry and troop carriers, he wrote a detailed design brief: the incentives, the map constraints and the win condition. It ended with "First research and then critique this design. Only build after we have agreed on any modifications" . The result was the Archipelago map, which he calls the most fun map in the game .
Other lessons from the project:
- Static HTML/JS with Three.js is a strong medium for agent-built games because the models know those platforms well .
- Giving Claude an OpenAI API key lets it generate images, look at them and refine its prompts .
-
Replays and multiplayer need a deterministic simulation. Claude had to replace
Math.sin()/Math.cos()with custom versions because Chrome and Safari returned slightly different numbers .
Peter Steinberger started a loop to move OpenClaw's Lit frontend to Solid 2.0. Removing Lit took about 6 days, and making the code idiomatic Solid took another week. He checked in every other day, and the work took about 600 PRs . The agent missed a few small regressions, mostly button shapes and minor UX details . His setup:
- Astra plans and Sol 6.1 codes. Prompting Astra to run Codex CLI in worktrees worked "FAR better than relying on subagents" .
-
A second agent challenges the first. Instead of one agent checking its own progress against a text file, as with
/goal, a separate agent pushes back on Astra. He built this as an OpenClaw plugin he calls "smart goal" .
He also moved OpenClaw to tsrs (npx tsrs). Its announcement describes a Rust port of the TypeScript compiler, built in collaboration with the TypeScript team. It claims up to 8x the speed of TS 7.0 and about 70% less memory, runs in WASM, and is due to join Oxc . Steinberger calls it a drop-in replacement and saw about a 5x speedup , or 9x against stable releases, since they were already on 7.1-dev .
Give agents a paved path to verification
Kent C. Dodds says the fix for low confidence in legacy codebases is verification. Agents should do what humans did: run with mock data, use the UI, seed varied database data and deploy to a production-like environment. They will do this reliably if you give them a paved path . His agents maintain a control-kody CLI for exactly this (SKILL.md) . Separately, a team that was "all in" on agents turned out not to be running tests in CI. Dodds calls that a prompting problem: an agent asked to audit the codebase for obvious problems would have caught it .
Theo reported something similar after a rough night. Most breakages came from legacy vendors, GitHub Actions and flaky Chromium-driver tests. The "slop" parts, such as ts-rust, were stable . In a new video, he describes building his machine images for the agents rather than for himself. The image is a customized, headless Ubuntu with Bash instead of Zsh. His audit found over 30% of Claude's errors came from Zsh's newline handling breaking ripgrep commands, and agents also did worse on Arch and CachyOS dev boxes .
AGENTS.md as a scheduler for follow-up work
After the eMudhra CA key leak, Geoffrey Huntley asked an agent whether his NixOS fleet was affected. Because Nix has a single source of truth, one patch covers transitive dependencies: patch, commit, autodeploy . His new trick is to have the agent write a dated reminder into AGENTS.md so the temporary workaround gets backed out later . The reminder names the nixpkgs commit to check for, the command to run, a recheck every two days, which files to delete, and which unrelated changes to keep .
Overalls also got "hive mode." Agents in the same profile zone (oss, standard or customer) share a message board for coordinating work. The customer zone has no outbound connectivity, and a machine's entries are wiped when it suspends . Huntley points to Claude Code's crossSessionInbound=accept setting as the apparent primitive behind hive swarms , and says it has him "back to loving claude" . He is moving 95% of his usage to Anthropic because OpenAI "falls over" after every usage reset .
OpenAI: longer cache retention and a usage-limit fix
The API now offers 12-hour prompt caching, so persistent agents pay the cached-input rate even when requests are hours apart . Tibo says logged-in Codex and Work users get cache kept "forever," with guaranteed hits when they resume old conversations . OpenAI also says about 40,000 accounts had usage limits that didn't match their plan. It has corrected the issue and will add credits for affected users .
T3 Code Stable
The new T3 Code Stable release has more than 1,400 changes . Highlights:
- Add a project to another machine from the "Run on" menu.
- PR diffs that separate code, tests and docs.
- Idle Claude threads compact early so the cache stays warm (beta).
-
t3 connectreachability checks. - Webhook-task filters and dedupe.
- More reliable skill discovery .
Theo suggests moving back to Stable because Nightly is getting more experimental .
Other notes
- Tenex's "meta harness" is structured repo docs plus a CLI that agents use to drive plans and readouts. Agents log their decisions, and everything is checked into Git, so the reasons behind changes live in the repo . Tenex has loosened the plan template as models improved . Its engineers mostly drive with Opus 5.5 and use Codex to review, which they say gives the best results . They report output roughly doubled this year .
- Claude Code Projects tip: tell the coordinator to save a project-memory rule to post a one-line update linking the thread whenever a thread is blocked or finishes a milestone. Before this, it wouldn't even report blocked threads .
- Kody lets agents file feedback about their experience, which often becomes a fix within an hour or two .
- Simon Willison built Battle Drones through 12 major iterations and 321 agent commits, mainly by Claude Code; although he knew little TypeScript and reviewed almost none of the code, he playtested it extensively . For a major gameplay revision, he prompted Claude Opus 5.5 with “Make it a LOT more fun” in Ultracode, a dynamic workflow that orchestrates many subagents; the overnight run included more than 1,000 simulated playtests . In 1,200 Normal AI-vs-AI matches, games still undecided after 25 minutes fell from 14% to 0.9%, while wins by sides that had been 800+ scrap behind rose from 24% to 46% .
- When the game’s AI ignored infantry and troop-carrying Condors, Willison turned that observation into a design brief: specify the strategic incentives and map constraints, then have the agent research and critique the design before building, with changes agreed first . The resulting Archipelago map had seven uplinks capturable only by flying infantry in a Condor, and Willison described it as the game’s most fun map .
- To add audio multiplayer, he pointed Claude Code at his working Chirp Pong proof of concept and asked it to adapt the approach; Claude built a Radio Probe to measure bandwidth before implementing an audio modem . He had not extensively tested the feature, though it seemed to work across laptops, iPhones, and iPads; after a player got a headache from the high-pitched sounds, he added a warning and moved toward online multiplayer, using a stateless WebSocket relay after WebRTC failed between his devices .
- The Claude Code team at Anthropic describes a review workflow in which Claude reviews and fixes most changes before a human reviewer is asked to focus on higher-level questions; it frames review around what could go wrong and which automated or human check can catch it, rather than whether a human read every line . As a complementary enforcement technique, Geoffrey Huntley described a pre-commit hook that checks diffs for direct calls bypassing a diagnostics library, fails the commit, and injects a prompt telling the agent to use the library—avoiding reliance on instructions that can drift .
-
DeepSeek open-sourced DeepInfer, the production inference framework used for V4.1 Flash, with distributed inference, speculative decoding, SWA Bounded Replay and structured output; it runs on NVIDIA and Huawei Ascend and includes the
deepseek-recipeAPI/token conversion component. KVCache management and load balancing are not fully open-sourced. - Agent-serving cache pattern: retain reusable Block KVCache prefixes in shared 3FS storage. DeepSeek reports a 3-day TTL for agent traffic; shared storage lets any prefill machine hit any prefix without prefix-affinity routing, reducing repeated prefill work. SWA KV remains in machine memory (about 180 seconds’ TTL and over 98% hit rate); on a miss, SWA Bounded Replay recalculates only the latest 128 tokens to rebuild it.
-
deepseek-recipemaps prompt-builder-inserted special tokens to token IDs and treats user/tool text that merely resembles them as ordinary text, preventing token injection; it also parallel-encodes ordinary-text segments between known special-token boundaries to reduce long-context tokenization latency. DSpark allocates speculative-token validation according to per-token confidence; DeepSeek reports 6% higher online throughput than fixed five-token validation.
-
For an agent-visible personal workspace, he used a Markdown
to-do's.mdfile as the shared data layer and synced it to a web app every 10 minutes; he and his agent could edit it from different devices, and access was restricted to approved email addresses. - To keep build requests from interrupting his personal-agent conversations, he set up a separate builder, Bill, as a new thread using workspace instructions; the personal agent can pass build requests to Bill, and he finds agents better at prompting one another than he is. For review, he asked subagents to test desktop and mobile, check for bugs, and perform a thorough review, then had the agent fix the reported issues.
- In BB, he uses Codex, Claude, Factory, and Pi; for UI exploration, a prototyping scale generated three alternatives that he mixed and refined with a simple request for a Linear-like feel, producing features such as a Kanban board and alternate detail views. He also uses Executor to connect services and tools once, then gives its setup prompt to an agent so the connected tools can be used across workflows.
- Omarchy is designed as an agent-ready Linux desktop: agents are preconfigured so users can start making a theme, app, or plugin from an invitation to “make something cool.” DHH’s stated setup target is a computer ready to use in 30 seconds, rather than spending half an hour or half a day configuring a developer machine.
- He says he used a model he names “Kim E K25” as his primary driver for over a month, praising its 200 tokens per second versus 30–40 tokens per second for frontier models at the time.
- Simon Willison built Battle Drones through 12 major iterations and 321 coding-agent commits, mainly with Claude Code; he reviewed almost none of the TypeScript code but playtested extensively. He found static HTML/JavaScript with WebGL, Phaser, and Three.js effective platforms for agent-built games.
- For a stalled gameplay problem, he gave Opus 5.5 the blunt prompt “Make it a LOT more fun” using Claude Code’s Ultracode dynamic workflows, which orchestrate multiple subagents. The overnight run included more than 1,000 simulated playtests; in 1,200 AI-vs-AI matches on Normal, games still undecided after 25 minutes fell from 14% to 0.9%, while wins by a side that had been 800+ scrap behind rose from 24% to 46%.
- When infantry and troop-carrying Condors lacked a strategic role, he prompted Claude to design a map around that problem and explicitly requested research and critique before implementation, with agreement on modifications before building. The resulting Archipelago map made seven island uplinks capturable only by flying infantry there in a Condor.
- To add audio multiplayer, he pointed Claude Code at his working Chirp Pong proof of concept, then had it build a Battle Drones version. Claude created a Radio Probe to measure device bandwidth before implementing a full audio modem; Willison reported successful trials between laptops, iPhones, and iPads, while cautioning that testing was still limited.
-
For replay and multiplayer, use a deterministic simulation: tiny differences in browser implementations of
Math.sin()andMath.cos()caused desynchronization, so Claude replaced them with custom versions.
- For this add-on, the host treated model choice as secondary: a capable Codex setup was likely enough, while Opus was described as more correct but much more expensive. Cursor was recommended as an approachable interface for building without relying on the terminal.
- Keeping the project folder pointed at the WoW add-on folder made changes testable with a game reload; the host also recommended initializing Git. Local-only version control was considered workable, but a remote copy would help if switching computers or losing the machine.
- Check platform capabilities before settling on a feature: in this WoW add-on exploration, the investigation found no general HTTP/socket or arbitrary file access, and detailed combat-log events were restricted.
- Use a review gate before implementation: the host avoids formal plan mode but has agents discuss their approach and answer questions before he authorizes coding, because their eventual implementation can differ from what he expected.
- Build and verify in small increments: the first generated legacy-menu dropdown reused an existing screen, so the pair tested it in-game and refined the request; the host recommends starting small and working one step at a time.
- The host said dictating ideas on his phone while pacing worked better for him than typing, which could not keep up with his thoughts. He also said agent-assisted coding had become better than his hand coding for completing tasks, while manual coding still produced better quality and thoughtfulness; with implementation less of a barrier, idea and design quality remain limiting factors.
- Tenex’s “meta harness” pairs structured repository documentation with a CLI that agents use for plans, visualizations, client readouts, and other parts of the engineering process; its canonical plan has become less prescriptive as models improved. Agents log decisions and activity, check the artifacts into Git, and keep the context for why decisions were made alongside the code as a durable record.
- The team’s current model preference is Opus 5.5, with Codex still useful for computer-use tasks and for reviewing Opus’s work; engineers report the back-and-forth produces the best results. One engineer uses AMP’s cloud “orbs” to run work off his computer and have another orb review Opus’s output; he says a newly added subscription hookup cuts costs by about 80%.
- Tenex reports roughly doubled engineer output over the period discussed, tracks average story points per engineer per month, and re-evaluates story-point criteria and project estimates at least quarterly as models improve.
- DeepSeek open-sourced DeepInfer, its large-scale inference framework for V4.1 Flash. It supports distributed inference, speculative decoding and structured output, and runs on NVIDIA and Huawei Ascend hardware.
- DeepSeek Recipe handles encoding and decoding between mainstream LLM API protocols and model tokens, with processing aligned to DeepSeek’s official API for third-party deployments. For agent-serving systems, DeepSeek identifies prefix KVCache reuse as a way to cut repeated prefill as contexts grow; its stated agent-traffic cache TTL is 3 days. Its persistent Block KVCache is stored on 3FS, while SWA KVCache stays in machine memory and has a reported ~180-second TTL and over 98% hit rate.
- Caveat: DeepInfer’s KVCache-management and load-balancing components are coupled to DeepSeek’s internal infrastructure and are not yet fully open source.
Sanfilippo describes DeepSeek Infer as a cloud inference engine for DeepSeek V4.1 Flash, built from a vLLM fork and ideas from other engines; its key design choice is specializing kernels for a particular model rather than maintaining a general-purpose engine. The release targets SM100/B200-class Nvidia and Huawei accelerators, but is incomplete for some serving needs: KV-cache management is withheld as a proprietary advantage, and prefill/generation separation for microbatching is not included. For local deployment, he cautions that kernels tuned for SM100 do not transfer directly to DGX Spark’s SM121 architecture and are optimized for heavy batching; he may adapt selected parts only if they offer a benefit.
- After a CA-key leak, Huntley prompted an agent to check whether his NixOS fleet was affected. He used Nix’s single source of truth to patch transitive dependencies fleet-wide, then commit and autodeploy the fix.
-
For a temporary workaround, he had the agent add an
AGENTS.mdreminder to reassess and back out the patch later. The reminder specified checking on or after Oct. 22 whethernixos-26.05included nixpkgs commit7816d39, checking again two days later if not, and removing the workaround once it was included—while retaining separate TLS andrustls-webpkichanges.
Simon Willison built browser-based RTS Battle Drones mainly with Claude Opus 5.5; Claude also generated all of its Three.js 3D models, and a model explorer lists their animation states.
The game can pit two LLMs against each other by pausing every five seconds for them to assess the game state and queue commands; a replay pairs Claude Haiku 5.5 with GPT-6 Luna. The device-to-device audio multiplayer produced a persistent high-pitched noise, prompting a warning that it might cause a headache; Willison added online multiplayer using a small Fly.io-hosted WebSockets relay, so players can share a link to join.
- Tenex’s MetaHarness combines shared skills, structured project documentation and a CLI. Agents work from a canonical plan, log decisions and activity, and check the records into Git alongside the code—preserving the rationale and context needed to resume long-running work. Tenex cautions that this level of structure can be overkill for a small team on one project.
- As models can get to roughly 90% of a solution when given the right context, Tenex’s CJ says his work has shifted from coding toward drawing out customer context and providing it to the model.
- Tenex describes Opus 5.5 as the team’s usual model, with Codex used for computer-use tasks and review; the discussion also describes using AMP’s remote Orbs to switch models and review another agent’s work.
- Tenex reports engineering output roughly doubled over the period as models improved at long-running tasks, and says it reevaluates story-point criteria at least quarterly.
A TODO-board build was shared with a video and a post containing copy/paste instructions for agents, giving developers a starting point for trying an agent-assisted project.
- Macroscope describes a PR-review workflow in which language-specific code “walkers” provide change context and team rules run as native GitHub checks that can block failing PRs. The workflow auto-approves a PR when it passes those checks and a code review and has a small blast radius; Macroscope says it now auto-approves 40% of customer PRs.
- For motion graphics, Claude can generate the code that renders a sequence instead of producing video with diffusion; having the code makes the result easier to customize and analyze.
- The video’s self-described “Trust Me Bro” benchmark roundup gives Gemini 4 Argon a mixed coding profile: it reports a lead on deep software engineering, but Claude Opus 5.5 scores 66 to 57 on Terminal Bench 4.0, GPT-6 Astra leads by more than 10 points on Frontier S, and Argon ranks ninth in webdev Code Arena. These are the video’s benchmark-specific claims, not a settled overall ranking.
For AINews reporting, swyx made Opus 5.5 the default after a side-by-side evaluation, praising its concise, tasteful writing and reduced “slopese” versus Opus 5 . He also cautioned that Opus’s writing quality can be undermined by unspecified “no-no words” in prompts, making it unsuitable for the critical path of his AI media organization .
- Theo built a custom Ubuntu-based environment for agents rather than his personal preferences: it is headless but can spin up XFCE-fork GUIs for agent computer use, and uses Bash instead of Zsh. He says over 30% of Claude’s errors across his machines came from Zsh newline formatting that led Claude to misformat ripgrep commands; he also found agents less capable on Arch and CachyOS dev boxes.
- Choose the stack around agent throughput, not developer preference: Theo kept the React Native mobile app instead of pursuing his preferred SwiftUI rewrite because agents could work faster in the existing app; he says team performance work made it comparable to, or in some ways faster than, the alternative.
- Theo reports that an AI-assisted Rust rewrite of the TypeScript compiler was working in under 20 hours, ran 2–3 times faster than TSGO, supported WASM, and achieved 99.99% accuracy against other TypeScript versions on the real projects he cared about.
Peter Steinberger says his “claw” automatically mutes most replies he characterizes as “slop.”
Ben Tossell describes a custom workspace combining a Superhuman-like email experience with a Linear-style todo board; the email is built on Rhys Sullivan’s Executor. His bot uses the workspace and can be tagged in comments; it also tracks domains, expirations, hosts, and linked local repositories, with plans to add links to mini-apps. He shared a video and copy/paste instructions for agents explaining how he built it.
ThePrimeagen says he dislikes Claude’s apparent preference for legacy versions “just in case.”
The quoted post claims DeepSeek V4.1 Flash 1017 beat GPT 6 Sol on a multi-step offensive-cybersecurity task and says it can be run locally . Primeagen disputed the local-running claim, saying that even his “incredible setup” is far from sufficient to run it locally .
DeepSeek V4.1 Flash Large-Scale Inference Deployment Practice
DeepSeek V4.1 Flash Large-Scale Inference Deployment Practice

this is a translation of https://zhuanlan.zhihu.com/p/2095560295930508953 (opens in new tab) for our own use/ twitter audiences to read
The capability boundaries of a large model are defined by the model itself, and the core problem that inference frameworks (opens in new tab) need to solve is how to enable these capabilities to serve a massive number of users at a reasonable cost .
Faced with massive AI invocation demands, inference frameworks have evolved from standalone service programs into complex systems that collaborate across clusters. Operators, communication, scheduling, and cache management interact and jointly determine end-to-end performance (opens in new tab) . For large-scale online services, performance optimization (opens in new tab) needs to be tailored to specific deployment conditions: what chips to use, how to organize the hardware, how to allocate computation, and how to reuse existing computation results.
From its inception, the DeepSeek V4.1 Flash (opens in new tab) inference system established three goals:
Compatible with computing chips (opens in new tab) from multiple manufacturers : Supports deployment on different hardware platforms and fully leverages the performance of each platform.
Based on common hardware organization, Prefill and Decode (opens in new tab) : Use common 8-card machines for Prefill and common 64-128-card supernodes (opens in new tab) for Decode, and design computation, communication and scheduling around the load characteristics of the two stages.
Supports retaining KVCache (opens in new tab) for more than 72 hours : provides a longer time window for context reuse, reduces redundant prefill calculations when the cache is hit, and incorporates the capacity and management overhead required to retain the cache for a long time into the system design (opens in new tab) .
To achieve these goals, we continuously optimize computing, communication, scheduling, and storage to improve the system’s effective throughput while meeting latency requirements, supporting more requests with less computing power and energy, and enabling intelligent services to reach more users at a lower price.
Previously, we open-sourced TileLang for developing high-performance operators and a series of inference operator libraries (opens in new tab) . Today, we officially open-source DeepInfer , a large-scale cloud service inference framework (opens in new tab). Based on this framework, we have implemented a high-throughput, low-latency large-scale inference service for DeepSeek V4.1 Flash. Currently, the service throughput of a single cluster is 155 trillion tokens/day, and the overall revenue is $38 per card per hour (based on off-peak prices).
DeepInfer integrates high-performance operators and a deeply optimized inference engine, supporting capabilities such as distributed inference, speculative decoding, SWA Bounded Replay (opens in new tab) , and structured output. It can run on NVIDIA (opens in new tab) and Huawei Ascend platforms. Simultaneously, we have open-sourced the production-grade token encoding/decoding (opens in new tab) component deepseek-recipe (opens in new tab) , supporting efficient and accurate encoding/decoding between mainstream large model API protocols and model tokens. Its processing logic is fully aligned with the official DeepSeek API, ensuring that third-party deployments receive the same API experience as official deployments. In the future, the inference code for new DeepSeek models and subsequent performance optimizations will also be continuously open-sourced within DeepInfer.
The KVCache management and load balancing (opens in new tab) components in the inference system are deeply coupled with the internal infrastructure, and their code is not yet fully open source. This article will start from the above design goals, introduce the overall architecture and key implementation of the DeepSeek V4.1 Flash inference system, as well as the design details of these two parts.
Overview of Reasoning System Design
The optimization goals of the DeepSeek V4.1 Flash inference system are: higher throughput and lower latency. To achieve these two goals, we use:
(Super)node Intranode MoE Partial Expert Parallelism (opens in new tab) (EP) : Increase batch size, improve GPU matrix multiplication (opens in new tab) efficiency, and thus increase throughput.
Vision Encoder / Prefill / Decode Separate Deployment : Decouple resource allocation for each stage, independently control and optimize TTFT and TBT respectively.
Inheriting the DeepSeek-V3/R1 inference system (opens in new tab) , the ATTN part uses data parallel (opens in new tab) DP (#DP=#EP). Load balancing is required between different DPs/EPs. For this purpose, we developed Prefill Load Balancer, Decode Load Balancer, Vision Encoder Load Balancer, and Expert- Parallel (opens in new tab) Load Balancer . To save the cost of repeated prefilling, we developed KVCache Storage .
Before and after a user requests inference, a conversion between the API protocol and the token sequence is required. DeepSeek Recipe (opens in new tab) supports efficient and accurate encoding and decoding between mainstream large model API protocols and model token sequences. Through special token segmentation and parallel strategies, it avoids special token injection while reducing the processing latency (opens in new tab) of long text tokenization (opens in new tab) .
Please refer to the following architecture diagram:

Prefill : EP2~8/DP2~8; Use the smallest possible EP/DP ratio when video memory is sufficient.
Decode : EP64~128/DP64~128 (Wide-EP); Use the largest possible EP/DP if the graphics card is available. (opens in new tab)
Vision Encoder : Generally, a single-card deployment (opens in new tab) is sufficient. It is recommended to use the cheapest available card for deployment.
Actual statistical data of the DeepSeek-V4.1-Flash online system

Offline performance benchmark data


DeepSeek Recipe
API multi-protocol adaptation
DeepSeek Recipe converts requests from the Chat Completions API, Responses API (opens in new tab) , and Messages API into a single dialogue format, sharing model templates and encoding logic, and generating prompt and token sequences based on the model version.
After parsing, the model output is divided into text, thought content, and tool calls, which are then converted into streaming responses (opens in new tab) for various protocols . Request transformation, prompt encoding, and output parsing are all implemented in Rust for efficient processing, while also supporting Python binding calls to improve ease of use.
Tokenizer processing and acceleration
In practice, Tokenizer faces two challenging issues: (1) As the model context lengthens, the linear tokenization method introduces significant latency, impacting the Time-To-Function (TTFT) and significantly increasing the CPU overhead of the service; (2) Special tokens are used to mark message roles, thought boundaries, and tool calls, but these tokens may also appear as ordinary text in the dialogue. When processing requests, it is necessary to avoid the special token text carried in user input from negatively impacting the model’s inference performance. To address these two issues, we implemented a solution based on special token boundary matching in DeepSeek Recipe.
Input Encoding : When constructing the prompt, explicitly inserted special tokens and their positions are recorded. During encoding, these are directly mapped to their corresponding token IDs, while ordinary text is encoded using the basic vocabulary. Strings such as `<input>` and `<output>` in user input, tool returns, and tool call parameters <|User|>are <|image|>encoded into corresponding ordinary token sequences to prevent text injection from controlling tokens and influencing model behavior. Output Parsing : When processing the response token stream, we identify special tokens </think>in the “End of Thought” marker and DSML tags based on their token IDs |DSML|. The same text concatenated by the model using ordinary tokens is correctly preserved as the original content and does not gain the ability to trigger the end of thought or tool call using special tokens. Segmentation Acceleration : The positions and boundaries of special tokens are already determined when constructing the prompt. Based on the encoding principle (opens in new tab) of BPE tokenizer , it can be found that the segments of ordinary text between special tokens can be encoded independently. Therefore, during encoding, we first skip the matching and segmentation of special tokens in the ordinary text, then encode each segment of ordinary text in parallel, and finally concatenate the results in the original order. This reduces the word segmentation time of ultra-long context requests and optimizes request latency.
Streaming output analysis
The inference backend returns chunks at the token (opens in new tab)level (opens in new tab) . A single UTF (opens in new tab) -8 code point, DSML tag, or tool call parameter may be split into multiple chunks. To correctly decode the model’s output and parse its structure as early as possible to reduce return latency, we need to implement a cross-chunk streaming parsing (opens in new tab) state machine. This machine should retain the processing progress and state of incomplete tasks in memory and return the determined content promptly.
Cross-chunk parsing : When encountering incomplete UTF-8 code points during decoding, the state machine buffers the relevant tokens. Once subsequent tokens arrive and the state is complete, the state is handed over to the parser. The parser maintains its state according to stages such as thought, main text, and tool parameters, incrementally matching delimiters using the KMP algorithm (opens in new tab) . If only part of the tag is found at the end of a chunk, the matching progress is temporarily preserved, and incremental recognition continues when the next chunk arrives. Confirmed content is output along with the parsing progress, eliminating the need to repeatedly scan previous text. Tool parameter conversion : Once the tool name is parsed, it returns immediately, and subsequent JSON fragments are directly generated based on DSML parameter boundaries. String values are JSON-escaped segment by segment, while non-string values such as arrays and objects are output as raw JSON text, eliminating the need to construct complete parameter objects and saving the overhead of repeated parsing and serialization. Even with long parameters, they can be returned segment by segment during generation, shortening the waiting time for the first parameter fragment.
Load balancing
Both data parallelism and expert parallelism require distributing a balanced load across each card as much as possible to reduce the overhead of idle time and waiting caused by card asynchrony. Specifically, the Prefill/Decode Load Balancer controls the load balancing of the attention through request (opens in new tab)routing (opens in new tab) and scheduling ; the Vision Encoder Load Balancer controls the load balancing between different vision encoders; and the Expert-Parallel Load Balancer controls the load balancing of the MoE sublayer through redundant experts and dynamic adjustments.
Prefill Load Balancer
The implementation mechanism of Prefill Load Balancer can be divided into 4 layers. The idea is to route and distribute requests based on the computational cost of the fitted attention sublayer (denoted as compute-quota, which is the pre-fitted quadratic cost model cost(q, c), q= the number of unhitted tokens in the sequence, c= the length of the prefix that has been computed in this sequence).
Bucketing: Requests are divided into four tiers based on the total input length: <16K / 16-64K / 64-256K / ≥256K. The number of engines allocated to each tier is dynamically redistributed based on the proportion of compute-quota demand in the previous round for that tier. Inter-DP Groups: Requests are pulled based on the remaining compute-quota. Since any engine can hit any prefix, prefix affinity routing is not performed. Intra-DP Groups: After a batch of requests is sent to an engine, each received request is distributed to the DP rank with the lowest current compute-quota load. Intra-card: Requests are dequeued FIFO. The chunk size is not fixed and is determined by inverse calculation of the budget. Each step has a time budget (in nanoseconds). During scheduling, a fitted quadratic cost model is used to predict how long each step will take . The maximum time budget that perfectly satisfies the remaining time budget is then calculated cost(q, c)by qinverse calculation of this quadratic equation (opens in new tab)q .
Decode Load Balancer
The Decode Load Balancer performs unified routing for all cards in all decode instances. The principle is to balance the number of requests and the total length of the KVCache at the same time. The former determines the batch size at each step, and the latter determines the cost of the attention part. The Decode Load Balancer heuristically sends short requests to cards with lower concurrency and larger total KVCache length, and long requests to cards with smaller total KVCache length. For details, please refer to the DualPath intra-group scheduling algorithm [1] (opens in new tab) .
Vision Encoder Load Balancer
Image routing is performed on a per-machine basis. The prefill machine randomly selects a machine with a number of vision encoders deployed on it. On this machine, the image is assigned to (width/patch_size) × (height/patch_size)the card with the fewest current image patches (calculated directly from the width and height) to generate the embedding.
Expert-Parallel Load Balancer
The implementation method of expert parallel load balancing can be summarized as follows: each card has several redundant experts in addition to the resident experts to house copies of popular experts. By statistically analyzing the load of each expert over the most recent N steps, experts are dynamically adjusted and redistributed to cards every M steps based on the statistical results. The expert weights are transferred within the parallel group (opens in new tab) in the scale-up network via symm_mem .
Prefill strategy : Start with 4 redundant experts per card (padded up to a multiple of ep_size in total number of experts); statistical window 100 steps, re-ranked every 1000 steps; online data measured moe_lb (slowest rank / mean) = 1.0320 (EP2/DP2) / 1.0618 (EP4/DP4). Decode strategy : Start with 1 redundant expert (opens in new tab) per card (padded up to a multiple of ep_size in total number of experts), for example, after rounding EP64, there are 7 experts per card (excluding shared experts (opens in new tab) ); statistical window 1000 steps, re-ranked every 10000 steps; online data measured moe_lb = 1.090.
Automatic expansion and contraction
After PD is separated, the resource demand ratio between the two changes dynamically with traffic. The resource mismatch under static allocation leads to queuing and idle time. Therefore, it is necessary to adjust the ratio of Prefill/Decode instances at the cluster level according to traffic through automatic scaling up and down (opens in new tab) .
Implementation : Prefill determines the required number of instances by counting the number of missing tokens, while decode determines the required number of instances by counting the number of requests being processed, adjusting the rate limit and upper and lower bounds accordingly. To address the resource waste caused by long-tail requests during large EP deployments, decode instances enter (opens in new tab)a grace period (opens in new tab) (1 minute) after receiving a shutdown notification. Requests and their KVCaches that are still incomplete after the grace period are forwarded to other instances to continue decoding. Redundant Quota (opens in new tab) : Redundant quotas are allocated during deployment to handle traffic fluctuations. This includes two aspects: first, unused ready machines are kept in the pool as expansion inventory; second, there are always more instances online than the calculated demand. Specifically, scaling down is prohibited when the quota occupancy rate is higher than 0.8. Fault Tolerance Mechanism : Error restarts occur on the order of minutes. Fault tolerance relies on a combination of quota redundancy and request-level rerouting, rather than shortening instance restart time. In large EP deployments, all cards of an instance constitute a single communication domain (opens in new tab) ; failure of any one card renders the entire instance unusable. During a restart, in order to continue using the remaining healthy cards of the instance, the platform redetermines the EP size based on all currently healthy cards, and the instance uses the actual number of cards it receives as the EP size.
KVCache Storage
With the growth of dialogue context and the widespread use of agents, KVCache prefix cache reuse can greatly reduce prefill recalculation, thereby significantly reducing costs. The DeepSeek-V4.1-Flash model’s KVCache storage is divided into two parts: Block KVCache and State KVCache, stored in the 3FS cluster and local memory respectively. Block KVCache : Managed in paginated blocks of 128 tokens, storing the main KV and indexer K across four layers, according to the model architecture. SWA KVCache : Each of the first 20 layers has a 134-token ring (window length = 128 + speculative backoff = 6), and does not store compressed intermediate states.

Block KVCache
Block KVCache persists to disk on a dedicated 3FS storage cluster, without using local memory storage. Compared to multi-level storage systems, the dedicated storage cluster ensures that all prefill machines can hit any prefix, avoiding conflicts between prefix affinity and load balancing routing strategies. It also decouples storage and compute nodes (opens in new tab) from automatic scaling, and KVCache is unaffected by compute node failures (compute node failure rates are orders of magnitude higher than storage failure rates). The KVCache system design considers two factors: increasing storage capacity to improve hit rate and increasing bandwidth (opens in new tab) to reduce read latency.
In terms of capacity, the 3FS distributed file system (opens in new tab)[2] (opens in new tab) provides approximately 100 PiB of SSD storage for KVCache. The TTL (opens in new tab) (Time-To-Live) of agent traffic KVCache reaches 3 days, and that of chatbot traffic reaches 7 days. For chatbot, the long-tail hit is more severe, and a small number of KVCaches will be read many times. After missing, the overall hit rate will be greatly reduced. According to experience, a TTL of about 3 days will cause the KVCache hit rate of chatbot traffic to decrease by 0.5%-1%. In terms of bandwidth, the 3FS storage cluster is optimized for random read and write at the KVCache block granularity (approximately 112 KiB). The peak aggregate read bandwidth reaches 10 TiB/s, and the peak aggregate write bandwidth is approximately 850 GiB/s.
At the inference framework level, Block KVCache is stored on disk by decode instances, allowing both prefill and decode outputs to be reused. Regarding KVCache hit overhead, the hit path is divided into three segments, with only the first segment on the critical path: matching is performed by the Entrypoint on the kvcache-store of the local CHECK role, using hash chain (opens in new tab) prefix matching. Only metadata is retrieved, but the hit length needs to be calculated before routing and cannot be masked; KV retrieval is done asynchronously (opens in new tab) in a separate process , moving the KVCache from 3FS to the host machine shm; the transfer from shm to HBM is done in a thread within the engine process, aiming to achieve overlapping of layer-by-layer H2D (Host-to-Device) transmission and forward computation. Due to the poor performance of the CUDA copy engine for fine-grained copying of KVCache Blocks, GDR is used for the transfer, allowing full utilization of PCIe bandwidth with an RDMA network card without consuming SM.
SWA KVCache
SWA KV is not stored in the persistent cache, but remains in the memory of the machine that generated it. A memory directory that is logically unified but actually stored separately by key on each Entrypoint records which machine it is on. If the local machine does not have the key when it is hit, it is retrieved from another machine. Since SWA KV is only reused within a minute-level window in a session, the total capacity of the pool is small but the turnover is fast, which is enough to cover simultaneously active sessions. Online data shows that its TTL is about 180 seconds and the hit rate is over 98%. Misses caused by eviction are recalculated by the above-mentioned Encoder SWA Bounded Replay [3] . (opens in new tab)
SWA KVCache only stores at positions that are multiples of 128 tokens. Storage occurs after the Prefill phase: the position is taken from the end of the sequence, leaving one token, and then aligned down to the position equal to 128. If the position is less than one 128 token, it is not stored in this round. Chunked prefill does not store during the process; only the last chunk is stored. During the Decode phase, a separate copy area is created in GPU memory. Each time a 128-token boundary is crossed, the current window of the request is copied from GPU memory into the copy area. When the request ends, it is stored at the position of the most recently copied element in the copy area, with the attachment point aligned down to the 128-token boundary. The copy position can cross a maximum of 5 tokens beyond the boundary, absorbed by the 6 extra positions in the ring compared to the window.
Hit Strategy
The hit principle is as follows: the hit length is determined by the Block KVCache, and whether the last block needs to be recalculated is determined by the SWA KVCache. If the SWA KVCache misses, a block is backed up and recalculated. Specifically, according to the YOCO truncation requirements, for an input length L, the maximum hit length is at most L - 134 (window size 128 + speculative backoff 6). Under this upper limit, the Block KVCache hit status is checked first, assuming the hit length is L1 (L1 is an integer multiple of block size = 128), and then the SWA KVCache hit status is checked. If the SWA KVCache exists within the length range of L1 - 128 to L1, the hit length is L1; otherwise, the hit length is L1 - 128, where 128 is the recalculation window.
3FS Distributed File System
As mentioned above, Block KVCache handles massive amounts of data and relies on the 3FS distributed file system for storage. To cope with node downtime or network fluctuations, it needs to balance high availability with high performance.
Data is stored on 3FS using erasure coding (10+2 Erasure Coding), and each EC group can tolerate the simultaneous failure of 2 nodes. (opens in new tab)
During reading, requests are sent to all 11 Storage nodes in the group simultaneously, and any 10 responses are sufficient to decode and return the data.
The GC Worker component is responsible for cleaning up expired caches: one instance is deployed on each Storage node, and the least accessed KVCache is periodically evicted according to the LRU policy, keeping disk usage below the waterline.
SWA Bounded Replay
SWA KVCache does not persist storage and recovers by recalculation when missing. This is the main change in KVCache management between V4 and V4. For details, please refer to the V4.1 technical report (opens in new tab)[2] (opens in new tab) . V4’s approach is disk write plus precise recalculation: SWA KVCache writes to persistent storage in two locations. When a miss occurs, it recalculates “level number × window” tokens to perform a complete forward pass and precisely recovers the window state layer by layer. This path is very costly. It is necessary to use a sufficiently large SSD capacity to reduce the miss rate to an extremely low level. The cost is that SWA KVCache occupies nearly half of the capacity of persistent KVCache and does not bring corresponding hit benefits: Block KVCache is long-tailed and reused. A prefix can be repeatedly hit within several days. SWA KVCache is only reused within minutes in active sessions and becomes invalid when the session ends or enters the next round. Using SSD capacity that can be retained for several days to carry data with a lifespan of minutes is particularly uneconomical in multi-round short dialogue scenarios. Therefore, Zero SWA Caching (opens in new tab) (i.e., no disk writes and only precise recalculation) proposed in the V4 technical report was not actually adopted in the V4.1 inference system because its cost was too high for production deployment.
The V4.1 inference system employs SWA Bounded Replay, recalculating only the most recent 128 tokens and truncating the attention window (opens in new tab) within the recalculation segment: replaying begins from position s, and the query at position i is calculated only with the keys in the interval from max(s, i − 127) to i, resulting in an approximate SWA KVCache. The recalculation cost is thus reduced to a constant independent of the context length and number of layers. This mechanism is used in two places:
Encoder SWA Bounded Replay : The prefix cache hit length now depends only on the Block KVCache. When the SWA KVCache is missing, the last 128 tokens of the hit prefix are recalculated and forwarded together with the missing suffix; the recalculated segment only rebuilds the SWA KVCache, while the Block KVCache is reused as is, without recalculation or overwriting.
Decoder SWA Bounded Replay : Prefill computation is performed only at the encoder layer. In the CED architecture, the decoder (opens in new tab) layer’s Block KVCache is projected from the encoder’s last layer’s hidden state. Only the decoder SWA KVCache needs to be calculated by each layer, requiring 20 decoder layers to process the last 2560 tokens for accurate reconstruction. Instead, each prefill operation only recalculates the last 128 tokens of the prompt at the decoder layer, and the output is only used for decoding and does not enter the prefix cache, nearly halving the total prefill computation.
DSpark Speculative Decoding
DSpark is a speculative decoding algorithm framework proposed by DeepSeek. Compared with DFlash, it superimposes a low-rank logit bias that only depends on the previous token ID on the output base logits. It uses extremely lightweight serial correction to fill in the local conditional information of adjacent tokens and improves the draft accuracy. For details, please refer to the DSpark paper [4] (opens in new tab) .
At the inference system level, DSpark employs variable-length validation. Since the acceptance probability of draft tokens varies across different requests and locations, fixed-length validation wastes computing power on low-probability tokens. DSpark predicts the acceptance rate of each draft token using the Confidence Head and allocates validation computing power to the token with the highest marginal benefit during scheduling. In online deployment, a draft generates 5 tokens at a time. Based on the hardware throughput curve from the profile and the acceptance rate of each draft token for each request, the draft tokens are sorted according to their expected acceptance rates, and the subset of prefixes with the largest effective throughput is selected for validation. Comparison shows that compared to fixed validation using 5 tokens, variable-length validation improves throughput by 6% in online deployment scenarios.
In conjunction with Asynchronous Scheduling: When schedule(N+1) is executed, the confidence of step N has not yet been returned. Therefore, when the CPU schedules, only the total budget is determined. The request length is recalculated on the GPU by a Tilelang kernel (cumulative multiplication, global sorting to take the top-k, and fusion with remap scatter), and the result is written back to the pinned buffer.
Asynchronous Scheduling (Zero-Overhead Batch Scheduler)
The Nth step of the Decode process is: schedule(N) → forward(N) → sample(N) → post-process(N). Schedule and post-process are pure CPU logic. Before post-process(N), the GPU and CPU need to synchronously obtain the result of sample(N). This results in the GPU being idle during post-process(N) and schedule(N+1). Referring to existing implementations [5] (opens in new tab)[6] (opens in new tab) , we overlap the pipeline through asynchronous scheduling between the GPU and the CPU: when the GPU executes forward(N) → sample(N), the CPU synchronously executes schedule(N+1) → launch_forward(N+1) → post-process(N-1) → launch_sample(N+1), and the GPU is not idle at all.
Structured output
Structured Output (SO) is used to constrain model output to strictly conform to a specified format (JSON schema, regular expression, EBNF (opens in new tab) , etc.).
Workflow:
Compilation phase (independent SO machine): Compiles the format constraints into a syntax automaton (opens in new tab) .
During runtime: At each decoding step, the automaton is updated, the set of valid tokens is determined, a mask is generated and applied to the sampling logic to force the output to conform to the target structure.
Compilation and derivation are decoupled; SO compilation is performed asynchronously, and the compilation results are retrieved before entering the engine.
refer to
^ (opens in new tab)Y. Wu, S. Chen, R. Huang, Y. Tan, Y. Zhong, M. Zhang, X. Jin, and P. Huang. DualPath: Accelerating Agentic LLM Inference by Harvesting Disaggregated KV-Cache Storage I/O. In Proceedings of the ACM SIGCOMM 2026 Conference, SIGCOMM ‘26, Denver, CO, USA, 2026. ACM. ISBN 979-8-4007-2467-1.
^ a (opens in new tab)b (opens in new tab) DeepSeek-AI. Deepseek-v4.1: DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression, 2026. URL https://arxiv.org/pdf/2609.19969 (opens in new tab).
^ (opens in new tab)DeepSeek-AI. A High-Performance Distributed File System Designed to Address the Challenges of AI Training and Inference Workloads, 2025. URL https://github.com/deepseek-ai/3FS (opens in new tab).
^ (opens in new tab)X. Cheng, X. Yu, C. Shao, J. Li, Y. Xiong, Y. Qian, J. Zhu, S. Ma, X. Zhang, J. Ye, et al. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. arXiv preprint arXiv:2607.05147, 2026a.
^ (opens in new tab)L. Zheng, L. Yin, Z. Xie, C. Sun, J. Huang, CH Yu, S. Cao, C. Kozyrakis, I. Stoica, JE Gonzalez, C. Barrett, and Y. Sheng. Sglang: Efficient Execution of Structured Language Model Programs. In Advances in Neural Information Processing Systems, volume 37, pages 62557–62583, 2024. doi: 10.52202/079017-2000.
^ (opens in new tab)K. Zhu, Y. Gao, Y. Zhao, L. Zhao, G. Zuo, Y. Gu, D. Xie, T. Tang, Q. Xu, Z. Ye, K. Kamahori, C.-Y. Lin, Z. Wang, S. Wang, A. Krishnamurthy, and B. Kasikci. Nanoflow: Towards Optimal Large Language Model Serving Throughput. In Proceedings of the 19th USENIX Conference on Operating Systems Design and Implementation, OSDI ‘25, USA, 2025. USENIX Association. ISBN 978-1-939133-47-2.
Edited on 2026-10-20 04:30
-
DeepSeek open-sourced DeepInfer, the production inference framework used for V4.1 Flash, with distributed inference, speculative decoding, SWA Bounded Replay and structured output; it runs on NVIDIA and Huawei Ascend and includes the
deepseek-recipeAPI/token conversion component. KVCache management and load balancing are not fully open-sourced. - Agent-serving cache pattern: retain reusable Block KVCache prefixes in shared 3FS storage. DeepSeek reports a 3-day TTL for agent traffic; shared storage lets any prefill machine hit any prefix without prefix-affinity routing, reducing repeated prefill work. SWA KV remains in machine memory (about 180 seconds’ TTL and over 98% hit rate); on a miss, SWA Bounded Replay recalculates only the latest 128 tokens to rebuild it.
-
deepseek-recipemaps prompt-builder-inserted special tokens to token IDs and treats user/tool text that merely resembles them as ordinary text, preventing token injection; it also parallel-encodes ordinary-text segments between known special-token boundaries to reduce long-context tokenization latency. DSpark allocates speculative-token validation according to per-token confidence; DeepSeek reports 6% higher online throughput than fixed five-token validation.
- DeepSeek open-sourced DeepInfer, its large-scale inference framework for V4.1 Flash. It supports distributed inference, speculative decoding and structured output, and runs on NVIDIA and Huawei Ascend hardware.
- DeepSeek Recipe handles encoding and decoding between mainstream LLM API protocols and model tokens, with processing aligned to DeepSeek’s official API for third-party deployments. For agent-serving systems, DeepSeek identifies prefix KVCache reuse as a way to cut repeated prefill as contexts grow; its stated agent-traffic cache TTL is 3 days. Its persistent Block KVCache is stored on 3FS, while SWA KVCache stays in machine memory and has a reported ~180-second TTL and over 98% hit rate.
- Caveat: DeepInfer’s KVCache-management and load-balancing components are coupled to DeepSeek’s internal infrastructure and are not yet fully open source.