Open
Description
Description
While reading through the codebase, I came across a comment in src/confluent_kafka/src/Producer.c that states:
/**
* @brief KNOWN ISSUES
*
* - Partitioners will cause a dead-lock with librdkafka, because:
* GIL + topic lock in topic_new is different lock order than
* topic lock in msg_partitioner + GIL.
* This needs to be sorted out in librdkafka, preferably making the
* partitioner run without any locks taken.
* Until this is fixed the partitioner is ignored and librdkafka's
* default will be used.
*
*/
However, in tests/test_Producer.py, I noticed a test function named test_set_partitioner_murmur2_random()
which appears to validate the ability to set a partitioner without error.
This leads to some confusion on my end:
- Is the comment in
Producer.c
outdated and no longer valid? - Or is
test_set_partitioner_murmur2_random()
testing only the API call, but the partitioner is actually not used at runtime (due to being ignored internally)? - Could this represent a partial incompatibility, or is it just a legacy note that should be removed or clarified?
Environment
- Version:
confluent-kafka==2.10.0
- Platform: N/A (just reading the source)
What I'm looking for
Clarification on whether custom partitioners are actually used at runtime in the Python client, and if the warning about deadlocks is still applicable in the current version.
Metadata
Metadata
Assignees
Labels
No labels