Update Kafka client version and change partition assignor #769
Conversation
| # Use the cooperative sticky assignor for consumer group rebalances. | ||
| # This *dramatically* reduces the time taken to rebalance the consumer group | ||
| # when the underlying topic metadata changes (leadership changes etc). | ||
| "partition.assignment.strategy": "cooperative-sticky", |
There was a problem hiding this comment.
This seems like this will change how things work for existing consumers. Will it make things better in all cases?
There was a problem hiding this comment.
Once they restart their consumer applications they will use this new assignment strategy. It will only be triggered on a consumer group rebalance. If that happens they will see a much quicker rebalance as fewer partitions will be reassigned between consumers. So, yes, we should see an across the board improvement.
5402f7f
to
b2c23a2
Compare
|
@KTAtkinson Since this PR was posted the confluent-kafka version has already been updated, but the config change to use cooperative rebalancing in the consumer is still an important change. What do we need to do to get this merged? |
There was a problem hiding this comment.
(copy-pasting same comment from #761)
I'm planning to release a baseplate 2.6.0 final version in the next couple days (and I'd like to avoid adding any more changes to this already-large release). After that, I think it'd be a good time to start working through the backlog of baseplate PRs to see what can be merged. Let's plan to merge this in a few days once 2.6.0 is out?
NOTE: Since this PR was posted the confluent-kafka version has already been updated, but the config change to use cooperative rebalancing in the consumer is still an important change.
💸 TL;DR
This PR updates the confluent kafka client version, which brings bug fixes and support for newer Kafka versions and features. It also updates the default partition assignor used by the Kafka consumers to use the "cooperative sticky" assigner, which is the recommended, faster option.
📜 Details
While Kafka servers are backwards compatible with older client versions, new features are limited to newer client versions so it is important to keep you clients up to date. Also, newer clients are backwards compatible with older server so there should not be any compatibility concerns. This, along with the obvious advantages of bug and performance fixes (including Mac M1 support) that come with new versions, makes this upgrade worthwhile.
The new client version also allows new configuration options. The newer consumer clients can use the "cooperative sticky" assignor (see KIP-726 for more details) which dramatically speeds up consumer group rebalances (by only changing partitions of consumers that have stopped rather that potentially shuffling the partitions of every consumer). This should make baseplate.py kafka consumer applications more stable.
✅ Checks