vLLM Architecture and PagedAttention Internals

Mapping OS Virtual Memory Paging to the KV-Cache Problem

vLLM's architecture makes the most sense once you recognize it as an operating systems paper wearing a machine learning paper's clothes. When Woosuk Kwon and colleagues published PagedAttention in 2023, they presented the work at SOSP, the ACM Symposium on Operating Systems Principles, not at a machine learning venue. That choice tells you what the paper actually solved: memory management, using an idea operating systems designers have leaned on for decades, virtual memory paging. Most people who first hear about vLLM assume the breakthrough was a smarter attention kernel. Most people who first hear about vLLM assume the breakthrough was a smarter attention kernel, but it wasn't. The breakthrough was admitting that GPU memory allocation had been running on the same crude logic as malloc before a widely used operating system's engineers fixed it decades ago, and borrowing the fix wholesale.

The stakes are concrete. On an NVIDIA A100 with 40GB of memory, a 13-billion-parameter model consumes a large share of that memory just holding static weights. Everything else, the dynamic key-value cache that grows as a sequence generates tokens, has to compete for what's left. Before PagedAttention, systems handled that remaining memory so poorly that they wasted 60% to 80% of it on fragmentation and over-reservation. Three failure modes drove that waste, and each one maps to a specific design fix later on: internal fragmentation from reserving space for a maximum sequence length that requests rarely use in full (a 4096-token allocation serving a 300-token output wastes 93% of the block), zero sharing between requests with identical prompts, and all-or-nothing eviction that forces entire sequences to swap out to CPU over PCIe, stalling the GPU for a meaningful stretch of time. PagedAttention exists to close that gap. Everything built around it in vLLM's runtime exists to keep it fed.

Virtual memory works because it separates what a process thinks it has from what actually exists in physical memory. A process sees a clean, contiguous address space. Underneath, the operating system scatters that process's data across whatever physical pages happen to be free, translating between the two views with a page table. The process never needs to know its memory is fragmented in reality, because the abstraction hides it from view.

The KV cache has the exact same shape of problem. A sequence's logical view of its own token history is the virtual address space. The physical GPU memory blocks where key and value tensors actually live are the physical pages. The block table that vLLM maintains per sequence, translating logical token positions to physical block locations, is the page table. That's a structural match, not a loose analogy, and it's the reason the SOSP venue mattered: the argument being made there was that GPU memory management had ignored a problem operating systems research solved decades earlier. It's a structural match, and it's the reason the SOSP venue mattered: the argument being made there was that GPU memory management had ignored a problem operating systems research solved decades earlier, and kept ignoring it well past the point of excuse.

Copy-on-write comes along for the same reason. When two sequences share a block, both point to the same physical memory, and neither pays a duplication cost until one of them actually needs to write something different into that block. Only then does the system copy it, splitting the shared block into two private ones. It's the same trick a widely used operating system has used for copy-on-write memory since before modern GPU serving systems existed, applied here to attention keys and values instead of process memory pages.

PagedAttention: the algorithm in concrete terms

Mechanically, PagedAttention divides each sequence's KV cache into fixed-size blocks, where each block holds the keys and values for a set number of tokens. These blocks don't need to sit next to each other in GPU memory. They can scatter anywhere, exactly like physical pages, because the block table for that sequence knows where each one lives.

vLLM V1 sets the default block size at 16 tokens, and that number is a tradeoff, not a guess. Larger blocks mean the GPU processes more tokens in one parallel pass, which cuts latency. But larger blocks also mean coarser allocation granularity, and coarser granularity means more wasted space in the last, partially-filled block at the tail of a sequence. Sixteen tokens is where vLLM's V1 engine lands as the balance point between those two pressures. Anyone tuning it larger for a latency win should expect to pay for it in fragmentation on short sequences, and pretending that cost doesn't exist is how tuning efforts go sideways.

The actual memory footprint of a block follows directly from its shape: two (for key and value) times the block size (16) times the number of KV heads times the head dimension times the byte width of the data type (2 bytes for bf16, for instance). Multiplying that out across every layer of a model turns the block pool's total size into a straightforward accounting exercise, which is what lets vLLM plan allocation ahead of time instead of guessing at runtime.

The free_block_queue and KV cache manager: how blocks are tracked at runtime

None of this works without bookkeeping, and that's the KV cache manager's job. At its core sits a structure called the free_block_queue, a pool of block IDs sitting idle and ready to be handed out. Its size is determined by the available VRAM and the configured block size.

The scheduler doesn't reach into that pool directly. Instead it talks to a KVCacheBlocks object, a deliberate wall between the scheduler and the manager's internals. That separation keeps the scheduler ignorant of how blocks are tracked underneath, so the two layers can change independently without breaking each other. It's the same reason kernel code separates a syscall interface from the page allocator sitting behind it; skipping that separation means a scheduler rewrite ends up rewriting the allocator too.

The lifecycle runs in a loop. A request comes in, the scheduler works out how many blocks its current token budget needs, and the manager pulls that many off the free_block_queue and maps them into the sequence's block table. When the sequence finishes, its blocks go back to the queue, or get held in the prefix cache in case another request can reuse them. Under memory pressure, lower-priority sequences get preempted and their blocks reclaimed, allowing the system to service higher-priority requests without permanently dropping the preempted work.

Before any of this starts, a Worker Init Device step runs a few checks: it confirms VRAM availability against the configured gpu_memory_utilization, verifies the model's data type is supported (bf16, for example), and sets up both the model_runner and a CPU-side InputBatch object. Only after that does the block pool come into existence.

Continuous batching and the unified V1 scheduler: keeping paged memory saturated

Static batching has a structural flaw: every sequence in a batch has to finish before the next batch can begin, leaving the GPU idle while it waits on the slowest member of the batch. That's expensive silicon sitting around doing nothing, and no amount of clever kernel optimization fixes what is, at bottom, a scheduling problem.

Continuous batching, sometimes called iteration-level scheduling, fixes it by running the scheduler after every single forward pass instead of after every batch. The moment a sequence finishes, its blocks free up, and a waiting request grabs them for the very next step. The GPU never waits for a whole cohort to wrap up together.

V1's scheduler goes further, and this is where it breaks from V0 in a way that actually matters: it erases the split between prefill (processing the prompt) and decode (generating new tokens) that V0 treated as fundamental, with separate phases and separate scheduling logic for each. V1 represents both, uniformly, as a dictionary mapping request ID to a number of tokens, a token budget handed out per step. That single representation is what lets chunked prefill, prefix caching, and speculative decoding compose together without the codebase needing a special case for every pairwise interaction between features. Treating prefill and decode as fundamentally different phases, the way V0 did, is the design choice that made every later feature harder to bolt on. The scheduler runs under either a first-come-first-served policy or a priority policy, depending on how the deployment wants to weight incoming requests.

Chunked prefill: solving head-of-line blocking without sacrificing throughput

A long prompt creates a real problem for a shared GPU. Processing a 32,768-token prompt can occupy the GPU for one to several seconds of continuous computation on a large model, and every decode request queued behind it just waits. That's head-of-line blocking, and on a shared inference server it means one user's long document can visibly stall everyone else's token generation.

Chunked prefill breaks that long prefill into smaller, configurable pieces and interleaves those chunks with decode operations within the same batch step. Instead of the GPU running one giant prefill start to finish, it takes a bite of the prefill, serves some decode steps for other requests, takes another bite, and so on. vLLM V1 turns this on by default, with no flag to disable it, unlike V0, which required a flag (--enable-chunked-prefill) just to turn it on.

The design falls out naturally from the unified scheduler. Because the scheduler already thinks in terms of a token budget per request rather than a phase per request, giving a prefilling request a partial budget alongside decode budgets for other requests is just the normal operation of the same mechanism, applied twice in one step. Chunked prefill cuts Time To First Token by up to 30% and improves Inter-Token Latency by up to 1.4×. Running an earlier monolithic-prefill approach on a shared multi-tenant server, at this point, is choosing to stall every other user's decode loop for no benefit anyone asked for.

Prefix caching and copy-on-write sharing: the memory multiplier for repeated prompts

Plenty of production workloads send requests that share a long, identical prefix, such as a fixed system prompt, a common few-shot example set, or a repeated instruction block. Recomputing the KV cache for that shared text on every single request is pure waste, and prefix caching exists to stop it. When vLLM recognizes that a new request's leading tokens match a prefix it's already processed, it reuses the existing KV blocks instead of running the computation again. The new sequence's block table just points at the same physical blocks the earlier one used.

The matching works at the block level. Token sequences split into the same 16-token blocks used elsewhere in the system, each complete block gets hashed, and an incoming request's prefix hashes get checked against the cache before any GPU computation happens. If the hashes match, the work is already done, and skipping it is the entire point of the exercise.

Copy-on-write governs what happens when sequences that started out sharing a prefix begin to diverge. The shared blocks stay shared, no duplication, right up until one sequence needs to write a token that differs from what's already there. Only then does the system copy the block, giving the diverging sequence its own private version while the other sequence's pointer stays untouched. In workloads with heavy repetition, this produces a real throughput gain of up to 32%, alongside memory reductions of up to 90%. Turning prefix caching off on a workload dominated by shared system prompts throws away most of that gain for nothing, and there's no serious argument for doing it.

The execution loop: EngineCore, persistent batching, and CPU overhead elimination

As GPUs get faster, the bottleneck has a way of migrating somewhere else, and in vLLM's case it migrated to the CPU. Tokenization, scheduling, input preparation, de-tokenization, response streaming: none of that runs on the GPU, and none of it used to matter much when GPU execution was the slow part. But on a model like Llama-8B running on an H100, GPU execution time can drop to something like 5 milliseconds, fast enough that CPU-side overhead becomes visible and starts eating directly into the latency budget.

V0's answer was a multiprocessing API server using ZeroMQ for inter-process communication, letting the API server and the AsyncLLM component overlap their work. It kept the scheduler and Worker 0 colocated in the same process, specifically to cut down on broadcast overhead between them, and that produced an asymmetric design carrying its own complexity: a fix that solves one bottleneck by quietly creating a second one somewhere else in the stack.

V1 restructures this around an isolated EngineCore execution loop that handles only the scheduler and the model executor, nothing else. Everything CPU-intensive, tokenization, multimodal preprocessing, de-tokenization, streaming responses back to the client, gets pulled off that critical path entirely and overlapped with the core loop instead of blocking it. The scheduler and the executor run tight and undistracted, while the processing-heavy bookkeeping happens alongside rather than in the way. Colocating the scheduler with Worker 0 the way V0 did saves on broadcast overhead but reintroduces the exact CPU contention the redesign was meant to remove. V1 doesn't repeat it. That separation reflects the same instinct running through the rest of vLLM's design: find whichever resource is actually scarce, GPU memory or GPU execution time, and build every surrounding piece to protect it, not to look busy around it.