A Routine Node Drain Turns Into a 14-Minute Outage
During a planned Kubernetes node drain, one of our consumer pods was evicted. Nothing about this was unusual: pods get rescheduled every day. Within seconds, however, end-to-end latency for the whole order-processing pipeline climbed from 38 ms to 4.2 seconds, and stayed there.
The Resolution: Switched the consumer group to the cooperative sticky assignor, raised the session timeout above worst-case pod startup time, and committed offsets before releasing partitions.
The dashboards showed a sawtooth: consumers would join, the group would rebalance, processing would stall, heartbeats would time out, and the group would rebalance again.
Eager Rebalancing Stops the Whole World
With the default eager protocol, every member of the group revokes all of its partitions at the start of a rebalance and only gets assignments back once the group has stabilized. One evicted pod therefore paused every consumer, not just the affected one.
Our session timeout was 10 seconds, shorter than the time a replacement pod needed to start and join. Each time a rebalance ran long, healthy consumers missed their heartbeats and were kicked out, which triggered the next rebalance. Offsets for in-flight batches were also uncommitted, so reassigned partitions replayed messages and added load.
“A rebalance is a distributed lock on your whole pipeline. If it takes longer than your session timeout, it feeds itself.”
Cooperative Rebalancing and Safer Timeouts
Cooperative sticky rebalancing moves only the partitions that actually need to move. Healthy consumers keep processing while the evicted pod's partitions are reassigned, so the stop-the-world pause disappears.
# move only the partitions that must movepartition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor # survive a pod restart without a rebalance stormsession.timeout.ms=45000heartbeat.interval.ms=15000max.poll.interval.ms=300000 # do not replay work on reassignmentenable.auto.commit=falseWe also added a ConsumerRebalanceListener that commits processed offsets synchronously when partitions are revoked, so a reassigned partition resumes exactly where the previous owner stopped.
Rebalance-Proofing Checklist
If you run Kafka consumers on Kubernetes, audit these checks before the next node drain:
Use The Cooperative Sticky Assignor
Avoid eager rebalancing so a single pod eviction never pauses the entire consumer group.
Set Session Timeouts Above Pod Startup Time
A timeout shorter than a replacement pod's startup turns one eviction into a rebalance storm.
Commit Offsets On Partition Revoke
Use a rebalance listener so reassigned partitions never replay already-processed messages.
Add PodDisruptionBudgets For Consumers
Limit how many consumers a drain can evict at once, and drain nodes one at a time.
Alert On Rebalance Frequency
Rebalances per minute is the earliest signal of a storm, long before latency alarms fire.
Rehearse Node Drains In Staging
Drain a node under production-shaped load. This incident was reproducible in an hour once we tried.