-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
bigquery: Big Query Managed Writer flow controller can dead lock when connections error out #9540
Labels
api: bigquery
Issues related to the BigQuery API.
priority: p2
Moderately-important priority. Fix may not be included in next release.
type: bug
Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Comments
hongalex
changed the title
pubsub: Big Query Managed Writer flow controller can dead lock when connections error out
bigquery: Big Query Managed Writer flow controller can dead lock when connections error out
Mar 14, 2024
hongalex
added
api: bigquery
Issues related to the BigQuery API.
and removed
api: pubsub
Issues related to the Pub/Sub API.
labels
Mar 14, 2024
alvarowolfx
added
priority: p2
Moderately-important priority. Fix may not be included in next release.
and removed
triage me
I really want to be triaged.
labels
Mar 14, 2024
shollyman
added
the
type: bug
Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
label
Mar 26, 2024
Thanks for the report here, sorry for the delay. The correct behavior is to ensure that connection's |
shollyman
added a commit
to shollyman/google-cloud-go
that referenced
this issue
Mar 26, 2024
Previously, connection's `lockingAppend` did not properly refund the connection's flow controller if the send response errored. This PR addresses that issue, and includes a test to ensure the correct behavior. Fixes: googleapis#9540
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
api: bigquery
Issues related to the BigQuery API.
priority: p2
Moderately-important priority. Fix may not be included in next release.
type: bug
Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Background
We have a service that reads a message from a pub sub topic, performs some computations and then uploads data to a Bigquery table using the
managedwriter
Big query package.We are periodically getting
io.EOF
errors from theAppendRows
API which eventually causes the API to dead lock. We have an open Google Cloud support ticket to root cause why we are receiving EOF errors from Bigquery backend.Reproduction
I was able to reproduce the issue in the following PR by simulating the connection always returning EOFs with a small in-flight request limit:
#9539
It is likely that there are other scenarios where the connection handling logic can error and lead to in flight requests not being properly released.
Workaround
The deadlock can be avoided by disabling the in-flight request limit.
The text was updated successfully, but these errors were encountered: