Skip to content

AdminClient.create_topics call hanging #833

New issue
New issue
Open
Open
AdminClient.create_topics call hanging#833
Labels
bugReporting an unexpected or problematic behavior of the codebasecomponent:adminIssues tied specifically to admin logic or code pathsinvestigate furtherIt's unclear what the issue is at this time but there is enough interest to look into itstatus:needs-more-infoIssues that require more information to cleanup.
@jpelletier-dad

Description

@jpelletier-dad
jpelletier-dad
opened on Apr 13, 2020

Description

Calls to AdminClient.create_topic() seem to spin indefinitely (even with request_timeouts added). This is after previous calls to retrieve the topics succeed (so the configuration would appear to be correct).

How to reproduce

CODE:

    def create_topics(self, topics, partitions):
        new_topics = [
            confluent_kafka.admin.NewTopic(t.topic, partitions, 2) for t in topics
        ]
        # futures == dict(<topic_name, future>)
        futures = confluent_kafka.admin.AdminClient(self.conf).create_topics(
            new_topics, request_timeout=5
        )

        for topic, f in futures.items():
            try:
                # blocking
                f.result()
                print("Topic {} created".format(topic))
            except Exception as e:
                print("Failed to create topic {}: {}".format(topic, e))
        return

Checklist

Please provide the following information:

  • confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()):
>>> confluent_kafka.version()
('1.3.0', 16973824)
>>> confluent_kafka.libversion()
('1.2.2', 16909055)
  • Apache Kafka broker version: Unsure, we're a confluent client. lkc-nvgqz
  • Client configuration: { "sasl.username": key, "sasl.password": secret, "bootstrap.servers": ",".join(servers), "sasl.mechanism": "PLAIN", "request.timeout.ms": "20000", "retry.backoff.ms": "500", "security.protocol": "SASL_SSL", "group.id": group_id, "debug": "broker,admin", }
  • Operating system: linux (docker) python:3.8-alpine
  • Provide client logs (with 'debug': '..' as necessary)
create topics (yes/no)?: yes
partition count: 4
INFO:root:creating topics with 4 partitions
[dev.market.trades.gemini.backfill]
%7|1586797957.446|BROKER|rdkafka#producer-2| [thrd:app]: sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstrap: Added new broker with NodeId -1
%7|1586797957.446|BRKMAIN|rdkafka#producer-2| [thrd::0/internal]: :0/internal: Enter main broker thread
%7|1586797957.446|BRKMAIN|rdkafka#producer-2| [thrd:sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstr]: sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstrap: Enter main broker thread
%7|1586797957.446|CONNECT|rdkafka#producer-2| [thrd:app]: sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
%7|1586797957.446|CONNECT|rdkafka#producer-2| [thrd:sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstr]: sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstrap: Received CONNECT op
%7|1586797957.446|STATE|rdkafka#producer-2| [thrd:sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstr]: sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1586797957.446|INIT|rdkafka#producer-2| [thrd:app]: librdkafka v1.2.2 (0x10202ff) rdkafka#producer-2 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer, CC CXX PKGCONFIG GNULD C11THREADS LIBDL PLUGINS ZLIB SSL SASL_CYRUS ZSTD HDRHISTOGRAM LZ4_EXT RAPIDJSON SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0x4002)
%7|1586797957.446|CONNECT|rdkafka#producer-2| [thrd:sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstr]: sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstrap: broker in state TRY_CONNECT connecting
%7|1586797957.446|STATE|rdkafka#producer-2| [thrd:sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstr]: sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1586797957.447|DESTROY|rdkafka#producer-2| [thrd:app]: Terminating instance (destroy flags none (0x0))
%7|1586797957.447|CREATETOPICS|rdkafka#producer-2| [thrd:main]: CREATETOPICS worker called in state initializing: Success
%7|1586797957.447|ADMIN|rdkafka#producer-2| [thrd:main]: CREATETOPICS: looking up controller
%7|1586797957.447|CONNECT|rdkafka#producer-2| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: lookup controller
%7|1586797957.447|CONNECT|rdkafka#producer-2| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: lookup controller
%7|1586797957.447|DESTROY|rdkafka#producer-2| [thrd:main]: Destroy internal
%7|1586797957.447|CREATETOPICS|rdkafka#producer-2| [thrd:main]: CREATETOPICS worker called in state waiting for controller: handle is terminating: Local: Broker handle destroyed
%7|1586797957.447|DESTROY|rdkafka#producer-2| [thrd:main]: Waiting for background queue thread to terminate
%7|1586797957.447|DESTROY|rdkafka#producer-2| [thrd:main]: Removing all topics
%7|1586797957.447|DESTROY|rdkafka#producer-2| [thrd:main]: Sending TERMINATE to sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstrap
%7|1586797957.447|TERM|rdkafka#producer-2| [thrd::0/internal]: :0/internal: Received TERMINATE op in state INIT: 1 refcnts, 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs
%7|1586797957.447|BROKERFAIL|rdkafka#producer-2| [thrd::0/internal]: :0/internal: failed: err: Local: Broker handle destroyed: (errno: No error information)
%7|1586797957.447|FAIL|rdkafka#producer-2| [thrd::0/internal]: :0/internal: Client is terminating (after 0ms in state INIT)
%7|1586797957.447|STATE|rdkafka#producer-2| [thrd::0/internal]: :0/internal: Broker changed state INIT -> DOWN
%7|1586797957.447|TERMINATE|rdkafka#producer-2| [thrd::0/internal]: :0/internal: Handle is terminating in state DOWN: 1 refcnts (0x557ce477cf30), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
%7|1586797957.447|BROKERFAIL|rdkafka#producer-2| [thrd::0/internal]: :0/internal: failed: err: Local: Broker handle destroyed: (errno: No error information)
%7|1586797957.454|CONNECT|rdkafka#producer-2| [thrd:sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstr]: sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstrap: Connecting to ipv4#54.165.163.127:9092 (sasl_ssl) with socket 3
%7|1586797957.454|TERMINATE|rdkafka#producer-2| [thrd:sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstr]: sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstrap: Handle is terminating in state CONNECT: 1 refcnts (0x557ce477da90), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
%7|1586797957.454|BROKERFAIL|rdkafka#producer-2| [thrd:sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstr]: sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstrap: failed: err: Local: Broker handle destroyed: (errno: Operation in progress)
%7|1586797957.454|STATE|rdkafka#producer-2| [thrd:sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstr]: sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstrap: Broker changed state CONNECT -> DOWN
%7|1586797957.454|TERM|rdkafka#producer-2| [thrd:sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstr]: sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstrap: Received TERMINATE op in state DOWN: 1 refcnts, 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs
%7|1586797957.454|BROKERFAIL|rdkafka#producer-2| [thrd:sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstr]: sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstrap: failed: err: Local: Broker handle destroyed: (errno: Operation in progress)
%7|1586797957.454|FAIL|rdkafka#producer-2| [thrd:sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstr]: sasl_ssl://pkc-l9pm0.us-east-1.aws.confluent.cloud:9092/bootstrap: Client is terminating (after 0ms in state DOWN)
  • Provide broker log excerpts
  • Critical issue

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugReporting an unexpected or problematic behavior of the codebasecomponent:adminIssues tied specifically to admin logic or code pathsinvestigate furtherIt's unclear what the issue is at this time but there is enough interest to look into itstatus:needs-more-infoIssues that require more information to cleanup.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions