Scale Beyond One GPU: Replication, Tensor Parallelism, and Prefill/Decode Disaggregation

The one-sentence version

Independent replicas are the default multi-GPU candidate: they add no cross-GPU communication, but they do not fix prefill-decode interference. Move to tensor parallelism or prefill/decode disaggregation only after a measured replica baseline shows which SLO—TTFT or TPOT—is actually at risk, and only if the measured interconnect supports the required collectives or KV transfers.

---

1. The opening decision

You are an inference engineer on call. Production traffic for llama-2-7b-chat in BF16 has grown past what one GPU can hold under your latency SLO. You have been handed 8 x NVIDIA A100-80GB in a single node and an illustrative traffic target of roughly 32 requests/second. Your SLOs are client-observed: p95 TTFT ≤ 200 ms and p95 TPOT ≤ 35 ms.

Three teams hand you three architecture proposals:

Option	What it does	The pitch
A	8 independent replicas behind a load balancer	“Zero cross-GPU communication. Each replica is a full model. Aggregate throughput scales linearly.”
B	4 tensor-parallel pairs (TP=2) over NVLink	“Split the model across two GPUs per replica. Halve the weight bytes each GPU reads per token.”
C	2 prefill GPUs + 6 decode GPUs, disaggregated	“Never run prefill and decode on the same GPU. Dedicated pools eliminate interference.”

Which do you approve?

The correct answer is: start with Option A (replicas), measure it, and only promote to B or C if the measured baseline violates an SLO for a reason that topology can actually fix. Replicas are not automatically the winner, but they are the right null hypothesis. This lesson walks through that decision end-to-end, using one consistent scenario, and ends with a one-page topology decision record you can file before spending cloud budget.

Illustrative assumptions—labelled, not measured

All traffic rates, SLOs, GPU counts, interconnect bandwidths, and latency figures in this lesson are illustrative planning assumptions unless explicitly sourced. They define a teachable scenario; they are not benchmark results from this repository. Separate physical lower bounds (what the hardware cannot beat) from client-observed latency (what your users actually experience, including queueing, network, and framework overhead).

---

2. One consistent scenario

Every comparison in this lesson uses the same model, workload, hardware, and SLOs so you are never comparing different contracts.

2.1 Model: Llama-2-7B in BF16 (MHA)

Model choice matters—do not mix contracts

This lesson uses Llama-2-7B with Multi-Head Attention (MHA). Do not carry the Llama-3-8B GQA capacity rate or KV head count into this example; they are different model and workload contracts. The KV cache math below uses MHA deliberately because it stresses memory and interconnect more than GQA.

Parameter	Value	Source / status
Parameters	~6.74 x 10^9 (~7B)	Model specification
Layers (L)	32	Model specification
Hidden dimension	4096	Model specification
Query heads	32	Model specification
KV heads (H_kv)	32 (MHA)	Model specification
Head dimension	128	Model specification
Precision	BF16, 2 bytes	Deployment choice

Weight footprint: 6.74 x 10^9 x 2 bytes approx 13.5 GB (about 14 GB with embeddings and LM head). This fits on one A100-80GB with room for KV cache and workspace—so the question is not “can the model fit?” but “can one GPU sustain the arrival rate under the latency SLO?”

2.2 KV cache: 512 KiB per token

The KV cache formula for batch size B = 1:

Bytes per token = 2 × layers × KV heads × head dimension × bytes per element

Plugging in the MHA numbers:

2 × 32 × 32 × 128 × 2 = 524,288 bytes = 512 KiB per token

This is a physical payload bound, not a measured DRAM traffic figure. A paged or tiled engine may read less or more per token depending on block residency and tiling.

Checked calculation: KV cache for a 512-token prompt

For a 512-token prompt at 512 KiB/token:

512 × 512 KiB = 262,144 KiB = 256 MiB of KV data

If that tensor must move across the network from a prefill worker to a decode worker:

• 10 Gb/s Ethernet (raw line rate): ideal serialization floor approx 215 ms
• 100 Gb/s RDMA (approx 11.5 GB/s effective): ideal serialization floor approx 21.5 ms

These are ideal serialization floors, not measured TTFT. Real transfer adds protocol overhead, NIC contention, serialization, and orchestration delay. A 215 ms floor already consumes a 200 ms p95 TTFT budget before any prefill compute or queueing happens.

2.3 Workload and SLOs (illustrative)

Quantity	Value	Status
Arrival rate	~32 req/s	Illustrative planning assumption
p95 TTFT SLO	≤ 200 ms	Client-observed SLO
p95 TPOT SLO	≤ 35 ms	Client-observed SLO
Single-GPU sustainable rate	~8 req/s	Illustrative planning assumption (no measured evidence)
Hardware	8 x A100-80GB, single node	Available inventory
Interconnect	NVLink 3, 600 GB/s bidirectional per GPU	Hardware datasheet

The arrival rate is about 4x what one GPU sustained, so you need multi-GPU scale. The question is which topology.

---

3. Candidate A: independent replicas (start here)

Deploy all 8 GPUs as 8 independent full-model workers behind a load balancer. Each replica holds the complete 14 GB model and manages its own KV cache pool. The load balancer routes requests using least-connections or power-of-two-random-choices.

Why replicas are the right null hypothesis:

• Zero cross-GPU communication. No All-Reduce collectives, no NVLink dependency, no PCIe fallback risk. Each replica executes standard continuous batching independently.
• Linear aggregate capacity (illustrative assumption). If a single GPU could sustainably serve 8 req/s under your SLO, 8 replicas would theoretically provide up to 64 req/s, but this is an illustrative assumption and not measured.
• Fault isolation. One GPU crash removes 1/8th of capacity, not a whole TP pair or prefill pool.
• Simple operations. Standard load balancer plus independent vLLM/SGLang processes; no NCCL process groups, no coordinator, no dual scheduling loops.

Why replicas are not automatically the winner:

Each replica still mixes prefill and decode on the same GPU. When a long prompt (e.g., 512 tokens) is admitted to a replica, its prefill forward pass monopolizes Tensor Cores and can stall ongoing decode iterations on that same GPU, causing inter-token latency jitter for active generation streams. Continuous batching mitigates this but does not eliminate it. If your measured replica baseline shows p95 TPOT spiking above 35 ms during prefill bursts, replicas alone may not satisfy your SLO even though aggregate throughput looks fine.

Replication does not compress single-request latency

Spinning up 8 replicas raises aggregate request capacity. It does not make any individual request faster. The p95 TTFT of a single request on a replica is bounded by single-GPU hardware limits—weight streaming, prefill compute, and queueing on that card. If your problem is per-request latency rather than aggregate concurrency, adding replicas does not fix it.

---

4. When to test Tensor Parallelism (Option B)

Tensor Parallelism partitions weight matrices within each transformer layer across TP GPUs using 1D Megatron-LM partitioning. For every decode step, each GPU holds a fraction of each weight matrix, and two All-Reduce collectives per layer synchronize partial results.

4.1 The communication tax

For this 32-layer model, standard 1D tensor parallelism requires 64 All-Reduce collectives per decode step (2 per layer). Their cost depends on the implementation, message size, interconnect, and contention; measure it on the target system.

PCIe or cross-node networking may make these synchronizations expensive enough to erase the benefit of splitting weights. Compare measured tensor-parallel latency with the replica baseline before choosing that layout.

4.2 When TP helps (and when it does not)

TP is worth testing only when your measured replica baseline shows:

1. TPOT is the binding SLO (not TTFT), and
2. Weight streaming is the dominant decode cost (memory-bandwidth bound at your batch size), and
3. You have NVLink or equivalent high-speed intra-node interconnect.

TP halves the weight bytes each GPU reads per token, which can lower TPOT. But it does nothing for prefill-decode interference within a replica, and the 64 All-Reduce barriers per token introduce straggler risk: one slow GPU delays every other GPU in the TP group, 64 times per token.

TP does not guarantee a latency or throughput improvement

TP reduces per-GPU weight traffic but adds synchronous collectives. On a slow interconnect, or for a small model where weight traffic is already small relative to launch overhead, TP can increase TPOT. Compare TP against your measured replica baseline and a tuned chunked-prefill configuration before accepting it.

---

5. Pipeline Parallelism: not categorically unsuitable

Pipeline Parallelism partitions sequential transformer layers across GPUs (stage 1 = layers 1–8, stage 2 = layers 9–16, etc.). Unlike the older blanket statement that PP is unsuitable for inference, PP is not categorically ruled out. vLLM supports TP+PP combinations, and PP can be useful on hardware without NVLink where TP’s All-Reduce tax is prohibitive.

However, PP introduces pipeline bubbles and inter-stage activation transfer latency. In autoregressive decode, token t+1 cannot begin until token t traverses all PP stages. Unless concurrency is high enough to keep all stages occupied with different microbatches, GPUs sit idle between stages. For strict interactive TTFT and TPOT SLOs, PP usually adds latency inflation and scheduler complexity without reducing the memory-read time per token. Treat PP as a candidate when TP is interconnect-limited and you can sustain enough concurrency to amortize bubbles—not as the default.

---

6. When to test prefill/decode disaggregation (Option C)

Disaggregation separates prefill and decode onto independent GPU pools. Prefill workers process prompts and produce the first token plus the KV cache; decode workers receive the KV cache and generate the remaining tokens. The two pools never run prefill and decode on the same GPU, so prefill bursts cannot stall decode iterations.

6.1 The KV transfer tax

The governing physical bound is the KV cache transfer from prefill worker to decode worker. Using the checked calculation above:

• A 512-token prompt generates 256 MiB of KV data
• At 10 Gb/s: ideal serialization floor approx 215 ms (blows a 200 ms TTFT budget before compute)
• At 100 Gb/s: ideal serialization floor approx 21.5 ms (viable, but real transfer adds overhead)
• On intra-node NVLink: sub-millisecond for this payload

Disaggregation over a standard cloud network can destroy TTFT

Do not deploy P/D disaggregation over commodity 10 Gbps cloud networking for an MHA model with long prompts. The KV transfer alone can exceed your TTFT SLO. Measure your actual NIC effective bandwidth and KV serialization latency before choosing disaggregation.

6.2 When disaggregation helps (and when it does not)

Disaggregation is worth testing only when your measured replica baseline shows:

1. TTFT or TPOT tail is dominated by prefill-decode interference on shared GPUs, and
2. Your interconnect can absorb KV transfers within your TTFT budget, and
3. Your prompt/output ratio is relatively stationary so you do not strand prefill GPUs while decode GPUs queue.

Disaggregation does not guarantee improvement. If your workload is mostly short prompts and long generations, your decode pool saturates while prefill GPUs sit idle. The static M:N pool split cannot rebalance without orchestrator support. Compare disaggregation against measured replicas and a tuned chunked-prefill configuration before accepting it.

---

7. Physical lower bounds vs client-observed latency

This distinction determines whether you can approve a topology on paper or must measure it live.

Concept	What it means	Can you approve from it alone?
Physical lower bound	Ideal serialization, weight streaming floor, NVLink transfer time—what hardware cannot beat	No. These are necessary filters, not production proof. Passing a bound does not guarantee SLO compliance.
Client-observed latency	TTFT/TPOT measured at the client, including network, queueing, framework overhead, and scheduling jitter	Yes, with measurement. This is what your SLO contract actually binds.

For example, the 21.5 ms ideal KV transfer floor at 100 Gb/s is a physical bound. Your client-observed p95 TTFT will be higher because it includes NIC overhead, prefill queueing, decode queueing, and transport. A topology that passes every physical bound can still fail client SLOs under load.

---

8. First learner action: checkable sizing exercise

Before you choose a topology, run this arithmetic on your own numbers:

1. Check fit: Does the model + workspace fit on one GPU? (14 GB weights < 80 GB framebuffer → yes.)
2. Check KV per token: For your model, compute 2 x L x H_kv x d_head x bytes. For Llama-2-7B MHA BF16: 512 KiB/token.
3. Check transfer floor: For your longest prompt P, compute P x 512 KiB, then divide by your interconnect’s effective GB/s. Does the ideal serialization floor already exceed your TTFT SLO?
4. Check the replica baseline: Measure one GPU under your traffic mix. Record p95 TTFT and p95 TPOT. Which SLO is at risk?

If step 4 shows p95 TPOT is the problem and you have NVLink, test TP. If step 4 shows p95 TTFT is inflated by prefill queueing and step 3 shows your network can absorb the KV transfer, test disaggregation. Otherwise, stay with replicas and tune chunked prefill and admission control first.

---

9. Topology decision record (one page)

Complete this before requesting multi-GPU hardware. Each field must reference a measurement or a physical bound, not a vendor claim.

TOPOLOGY DECISION RECORD — scale-beyond-one-gpu
=================================================
Model:            llama-2-7b-chat BF16 (MHA, 32 layers, H_kv=32, d_head=128)
Weight footprint: ~14 GB (fits one A100-80GB: YES)
KV bytes/token:   512 KiB  [2 x 32 x 32 x 128 x 2]
Max prompt P:     512 tokens → 256 MiB KV
Workload:         ~32 req/s (illustrative; measure your own λ)
Single-GPU rate:  ~8 req/s sustainably under SLO (measured baseline date: ______)

SLOs (client-observed):
  p95 TTFT <= 200 ms   p95 TPOT <= 35 ms

Interconnect:     NVLink 3, 600 GB/s bidirectional (illustrative)
KV transfer floor @ P=512:
  NVLink: <1 ms        100 Gbps RDMA: ~21.5 ms        10 Gbps: ~215 ms

Measured replica baseline:
  p95 TTFT: ______ ms   p95 TPOT: ______ ms   Goodput: ______ %
  Binding SLO: TTFT / TPOT / neither

Candidate evaluated:
  [ ] Replicas (R=8)      [ ] TP=2 pairs (R=4)       [ ] P/D disaggregation (M=2, N=6)
  [ ] TP+PP over NVLink

Decision:         _________________________________________________
Rationale:        _________________________________________________
Must measure before approval:
  1. Replica p95 TPOT under prefill bursts (does interference breach 35 ms?)
  2. Effective NIC bandwidth and KV serialization latency (if considering disaggregation)
  3. NCCL All-Reduce p99 per token under TP (if considering tensor parallelism)
  4. Chunked-prefill-tuned replica baseline (does tuning alone satisfy SLO?)
  5. Client-observed p95 TTFT/TPOT under sustained λ=32 req/s for the chosen topology

Approval sign-off: ____________________  Date: __________

---

Try it yourself

1. 01For your own model, compute KV bytes per token using 2 x L x H_kv x d_head x bytes. Then compute the KV volume for your longest prompt and divide by your interconnect effective GB/s. Does the ideal serialization floor already exceed your TTFT SLO?

   Show a hint

   Llama-2-7B MHA BF16: 2 x 32 x 32 x 128 x 2 = 524,288 bytes = 512 KiB/token. A 512-token prompt = 256 MiB. At 10 Gb/s (~1.1 GB/s), floor approx 215 ms; at 100 Gb/s (~11.5 GB/s), floor approx 21.5 ms.

2. 02Measure one GPU serving your traffic mix under continuous batching. Record p95 TTFT and p95 TPOT. Which SLO is at risk? If TPOT is the problem and you have NVLink, TP is a candidate. If TTFT is inflated by prefill queueing, disaggregation is a candidate—provided your network passes step 1.

   Show a hint

   You need a running vLLM/SGLang endpoint and a client harness (httpx asyncio with semaphore concurrency). Stream responses and timestamp first chunk (TTFT) and last chunk (E2E); TPOT = (E2E - TTFT) / (completion_tokens - 1).

3. 03Explain why adding 8 replicas does not reduce the p95 TTFT of any single request, and name the condition under which replicas would still fail your SLO despite having 4x aggregate headroom.

   Show a hint

   Replicas add aggregate capacity, not per-request speed. They fail when prefill-decode interference on a shared GPU spikes TPOT above SLO during long-prompt bursts, or when a single request’s prefill compute exceeds the TTFT budget.

4. 04State why TP over PCIe or cross-node Ethernet without NVLink is an antipattern for interactive serving, using the 64 All-Reduce per token figure.

   Show a hint

   A 32-layer model executes 64 All-Reduce collectives per decode step. On slow interconnects, collective launch and synchronization latency per step can exceed the memory-bandwidth savings of splitting weights, increasing TPOT rather than reducing it.

---

Key takeaways

• Start with independent replicas as the default multi-GPU candidate: zero cross-GPU communication, linear aggregate capacity, simple operations, and clean fault isolation.
• Replicas do not compress single-request latency and do not remove prefill-decode interference on shared GPUs.
• Tensor Parallelism halves per-GPU weight traffic but inserts 64 All-Reduce collectives per decode step (32-layer model); it helps only when TPOT is binding, weight streaming dominates, and you have NVLink.
• Pipeline Parallelism is not categorically unsuitable for inference—vLLM supports TP+PP—but pipeline bubbles and inter-stage latency usually make it a poor fit for strict interactive TTFT/TPOT SLOs.
• Prefill/decode disaggregation eliminates interference but introduces a KV transfer tax: a 512-token Llama-2-7B MHA prompt produces 256 MiB of KV data, whose ideal serialization floor is ~215 ms at 10 Gb/s or ~21.5 ms at 100 Gb/s before real-world overhead.
• Neither TP nor disaggregation guarantees improvement. Compare both against measured replicas and a tuned chunked-prefill baseline before accepting either.
• Separate physical lower bounds (ideal serialization, weight streaming floors) from client-observed latency (queueing, network, framework overhead). Passing a bound does not prove SLO compliance.
• File a one-page topology decision record before requesting hardware: model footprint, KV bytes/token, measured replica baseline, binding SLO, interconnect bounds, and the specific measurements required for approval.

---

Interview questions

Why are independent replicas the correct null hypothesis for multi-GPU scaling, even when tensor parallelism is available?

Replicas add zero cross-GPU communication, scale aggregate capacity linearly, isolate failures, and are operationally simple. They should be the measured baseline against which TP and disaggregation are compared; neither alternative is automatically better.

A 512-token Llama-2-7B MHA prompt in BF16 produces 256 MiB of KV data. At 10 Gb/s effective bandwidth the ideal serialization floor is ~215 ms. Does this prove disaggregation is unviable over 10 Gbps networks?

It proves the physical transfer alone consumes a 200 ms TTFT budget before prefill compute, queueing, or transport. Real-world protocol, NIC, and orchestration overheads make it worse. Disaggregation over 10 Gbps is therefore physically non-viable for this model/prompt class, but the conclusion is about the bound, not a measured TTFT.

Does Tensor Parallelism guarantee lower TPOT?

No. TP reduces per-GPU weight bytes read but adds two synchronous All-Reduce collectives per transformer layer (64 per step for 32 layers). On slow interconnects or small models, communication overhead can exceed bandwidth savings, increasing TPOT. Compare against a measured replica baseline.

Is pipeline parallelism categorically unsuitable for inference serving?

No. vLLM supports TP+PP, and PP can be useful on hardware without NVLink where TP All-Reduce tax is prohibitive. However, pipeline bubbles and inter-stage activation transfer usually add latency for strict interactive TTFT/TPOT SLOs unless concurrency is high enough to amortize bubbles.

What must you measure before approving a multi-GPU topology, and why can analytical bounds alone not approve it?

Measure replica p95 TPOT under prefill bursts, effective NIC bandwidth and KV serialization latency (disaggregation), NCCL All-Reduce p99 per token (TP), and chunked-prefill-tuned replica baseline. Analytical bounds are necessary filters but omit queueing, framework overhead, and scheduling jitter that determine client-observed SLO compliance.

---

Related lessons

• Inference Optimisation & AI Infrastructure · Measure and Explain Inference PerformanceServing Workloads and Performance Metrics12 min
• Inference Optimisation & AI Infrastructure · Measure and Explain Inference PerformanceWhere Inference Time and Memory Go13 min
• Inference Optimisation & AI Infrastructure · Build a Serving SystemChoose a Serving Runtime and Hardware15 min
• Inference Optimisation & AI Infrastructure · Schedule Requests and Manage KV MemoryPrefill Optimisation: Prefix Caching, Chunked Prefill, and Disaggregation35 min
• Inference Optimisation & AI Infrastructure · Build a Serving SystemCapacity Planning: From Traffic Profile to GPU Count18 min

---

Sources and scope

Claim	Source	Status / limitation
1D Megatron TP: two All-Reduce collectives per transformer layer	Shoeybi et al., Megatron-LM (arXiv:1909.08053), §3.4	Communication pattern is standard; exact per-collective latency is hardware- and NCCL-version-dependent.
vLLM parallelism and scaling (TP, PP, TP+PP)	vLLM docs: Parallelism and Scaling	Describes TP+PP support and use cases including hardware without NVLink.
vLLM disaggregated serving examples	vLLM docs: Disaggregated Serving	Prefill/decode separation mechanics; production overheads (serialization, RPC) must be benchmarked.
DistServe: disaggregating prefill and decode for goodput	Zhong et al., DistServe (USENIX OSDI 2024)	Shows structural isolation of prefill/decode; KV transfer latency is the governing tax.
Sarathi-Serve: chunked prefill and prefill-decode interference	Agrawal et al., Sarathi-Serve (USENIX OSDI 2024, arXiv:2403.02310)	Defines prefill as producing the first output token; chunked prefill bounds per-iteration tokens.
NVIDIA A100 NVLink 3: 600 GB/s bidirectional per GPU	NVIDIA Ampere Architecture Whitepaper; CUDA Ampere Tuning Guide	Peak hardware specification; achieved throughput depends on NCCL configuration and topology.
KV cache formula and MHA vs GQA head-count impact	Internal: inference-time-memory-breakdown.mdx (Lesson 2)	512 KiB/token for Llama-2-7B MHA BF16; 128 KiB/token for Llama-3-8B GQA BF16.

All latency, throughput, goodput, and utilization figures in this lesson are illustrative planning assumptions unless explicitly cited as hardware specifications. No GPU cluster was benchmarked during drafting. Before production signoff, execute measurements on target hardware under synthetic traffic matching your workload profile.

---

Unresolved measurements

The following cannot be resolved from this lesson alone and must be measured on your target hardware before topology approval:

1. Replica p95 TPOT under prefill bursts — does continuous batching alone keep inter-token latency under your SLO when long prompts arrive?
2. Chunked-prefill-tuned replica baseline — does enabling chunked prefill (--enable-chunked-prefill in vLLM) satisfy the SLO without any multi-GPU topology change?
3. Effective NIC bandwidth and KV serialization latency — if considering disaggregation, measure actual transfer time for your longest prompt’s KV tensor, not the ideal serialization floor.
4. NCCL All-Reduce p99 per token under TP — if considering tensor parallelism, measure collective latency on your exact node topology (NVLink vs PCIe vs Ethernet).
5. Client-observed p95 TTFT/TPOT under sustained arrival rate — the only number that actually binds your SLO contract.

File these in the topology decision record (§9) before requesting multi-GPU hardware.