Multi-node Serving · Part 2
Prefill/Decode Disaggregation
The queueing-theoretic case for splitting prefill and decode into separate pools, and the runtime mechanics -- KV transfer, scheduling, failure handling -- that make the split work.
Multi-node Serving · Part 2
The queueing-theoretic case for splitting prefill and decode into separate pools, and the runtime mechanics -- KV transfer, scheduling, failure handling -- that make the split work.