Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Panic when connecting through alloydb auth proxy - panic: runtime error: slice bounds out of range [:1157627904] with capacity 16384 #504

Closed
andriihrachov opened this issue Nov 17, 2023 · 14 comments
Closed

Panic when connecting through alloydb auth proxy - panic: runtime error: slice bounds out of range [:1157627904] with capacity 16384 #504

andriihrachov opened this issue Nov 17, 2023 · 14 comments
Assignees
@enocom
Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@andriihrachov
Copy link

Bug Description

alloydb proxy crashes with error, when deployed as a sidecar on kubernetes

2023/11/17 15:52:42 [********] accepted connection from 127.0.0.1:51470
panic: runtime error: slice bounds out of range [:1157627904] with capacity 16384

Example code (or command)

No response

Stacktrace

goroutine 9 [running]:
cloud.google.com/go/alloydbconn.(*Dialer).metadataExchange(0xc0000bf320, {0x12687f8?, 0xc000510000})
	/go/pkg/mod/cloud.google.com/go/[email protected]/dialer.go:337 +0x777
cloud.google.com/go/alloydbconn.(*Dialer).Dial(0xc0000bf320, {0x1263f08, 0xc0004cc000}, {0x7ffd5eaa8c14, 0x52}, {0x0, 0x0, 0x0?})
	/go/pkg/mod/cloud.google.com/go/[email protected]/dialer.go:247 +0xc34
github.com/GoogleCloudPlatform/alloydb-auth-proxy/internal/proxy.(*Client).serveSocketMount.func1()
	/go/src/alloydb-auth-proxy/internal/proxy/proxy.go:651 +0x2b1
created by github.com/GoogleCloudPlatform/alloydb-auth-proxy/internal/proxy.(*Client).serveSocketMount in goroutine 8
	/go/src/alloydb-auth-proxy/internal/proxy/proxy.go:631 +0x1da

Steps to reproduce?

  1. 1.5.0 version of gcr.io/alloydb-connectors/alloydb-auth-proxy
  2. deploy on kubernetes as a sidecar
  3. run psql to connect from a neighbor conatiner / port forward from local

Environment

GKE

Additional Details

No response
@andriihrachov andriihrachov added the type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. label Nov 17, 2023
@andriihrachov
Copy link
Author

not reproduceable on 1.4.1

@enocom
Copy link
Member

enocom commented Nov 17, 2023

Thanks @andriihrachov. I'll take a look and report back.

@enocom enocom transferred this issue from GoogleCloudPlatform/alloydb-go-connector Nov 17, 2023
@enocom
Copy link
Member

enocom commented Nov 17, 2023

What region is your cluster in?

@enocom enocom added the priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. label Nov 17, 2023
@enocom
Copy link
Member

enocom commented Nov 17, 2023

The new Go Connector version (and Proxy version) introduces a metadata exchange check prior to the database protocol taking over. You're seeing this panic because your instance doesn't understand the metadata exchange. This is not expected behavior.

Talking with some folks on the AlloyDB team, I'm told a recent rollout may have failed for a very small number of instances. Your instance seems to be one. If you have a support contract, you can open a ticket to get a fix sooner than the next rollout (tell them to cc me and I'll help get the right people to look at this). Otherwise, I'd suggest staying on v1.4.1 and I'll report back when I see the few stragglers fixed.

@enocom enocom added priority: p2 Moderately-important priority. Fix may not be included in next release. and removed priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. labels Nov 17, 2023
@andriihrachov
Copy link
Author

What region is your cluster in?

us-east4

Thanks a lot, I downgraded to 1.4.1

@cthumuluru
Copy link

Can you point us to the time window when you saw these errors?

@andriihrachov
Copy link
Author

17.11.2023 ~17:00 CET

@enocom
Copy link
Member

enocom commented Nov 18, 2023

FWIW I tested this on a newly created instance and didn't see any issues. It's possible some existing instances will be affected.

@andriihrachov
Copy link
Author

thanks. it was test db anyway, so I'll recreate

@enocom enocom reopened this Nov 20, 2023
@enocom
Copy link
Member

enocom commented Nov 20, 2023

This remains an unexpected problem. I'll leave this open for anyone else who runs into it and we'll close it once the underlying issue has been fixed.

@enocom
Copy link
Member

enocom commented Nov 20, 2023

FYI I've moved our latest container tags back to v1.4.1 for now too to try to minimize disruption.

@enocom
Copy link
Member

enocom commented Nov 21, 2023 •
edited

Quick update here after digging into this a bit more. If anyone runs into this issue here's how to work around it:

  • Downgrade to v1.4.1 which is unaffected. Downside here is that the last version doesn't work with Auto IAM AuthN.

To get Auto IAM AuthN working:

  • Consider updating any static database flag that requires a restart. Once the instance is updated, it will correctly understand the metadata exchange and work with v1.5.0

If you're still hitting this error, reach out to support for more help. We expect this issue will be fully resolved during our next data plane rollout. Sorry for the inconvenience.

@StevenACoffman
Copy link

StevenACoffman commented Nov 21, 2023 •
edited

For those playing the home edition of "fix that bug!", these are the explicit steps to temporarily revert:

go mod edit -replace cloud.google.com/go/alloydbconn=cloud.google.com/go/[email protected]
go mod tidy

@enocom
Copy link
Member

enocom commented Jan 29, 2024

This has now been fixed in the latest rollout.

@enocom enocom closed this as completed Jan 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@enocom enocom

Labels
priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@enocom @StevenACoffman @cthumuluru @andriihrachov