Skip to content

Clarification on partitioner usage: discrepancy between known issue comment in Producer.c and test case test_set_partitioner_murmur2_random #1993

Open
@arossowork

Description

@arossowork

Description

While reading through the codebase, I came across a comment in src/confluent_kafka/src/Producer.c that states:

/**
 * @brief KNOWN ISSUES
 *
 *  - Partitioners will cause a dead-lock with librdkafka, because:
 *     GIL + topic lock in topic_new  is different lock order than
 *     topic lock in msg_partitioner + GIL.
 *     This needs to be sorted out in librdkafka, preferably making the
 *     partitioner run without any locks taken.
 *     Until this is fixed the partitioner is ignored and librdkafka's
 *     default will be used.
 *
 */

However, in tests/test_Producer.py, I noticed a test function named test_set_partitioner_murmur2_random() which appears to validate the ability to set a partitioner without error.

This leads to some confusion on my end:

  • Is the comment in Producer.c outdated and no longer valid?
  • Or is test_set_partitioner_murmur2_random() testing only the API call, but the partitioner is actually not used at runtime (due to being ignored internally)?
  • Could this represent a partial incompatibility, or is it just a legacy note that should be removed or clarified?

Environment

  • Version: confluent-kafka==2.10.0
  • Platform: N/A (just reading the source)

What I'm looking for

Clarification on whether custom partitioners are actually used at runtime in the Python client, and if the warning about deadlocks is still applicable in the current version.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions