I'm encountering a peculiar issue with my setup. I've deployed my service on Cloud Run, and to forward requests to GKE pods, I'm utilizing a Serverless VPC Connector, all within the same region. This VPC connector is also facilitating connections from my Cloud Run service to Redis and some other resources external to GKE. However, I'm noticing significant delays, sometimes taking 10-15 seconds to forward a single request from Cloud Run to a GKE pod. I've investigated but haven't been able to pinpoint the cause. Can anyone provide insight or suggestions on how to troubleshoot and resolve this issue?
Two things to look for:
- The VPC Connectors have a particular number of instances and they scale up with traffic (but not back down). Make sure you haven't disabled automatic scaling. I believe you can also select an instance size; the larger instances have better throughput, and might have better latency if I recall correctly (it's been a while since I worked on this feature).
- Make sure you're reusing network connections, not establishing new ones for each request. Latency increases greatly if there is too much connection churn.
Agreed, that VM instance size also matters. But I have sufficient min and max instances. And other requests re not getting affected, only few requests every hour or so retries or takes lot of time to forward it to GKE pod. Not sure why it's happening
And one last note: Try moving to Direct VPC. It's free (unlike VPC Connectors) and has far better performance. The downside is that is can use a lot of IPs, and has a few feature compatibility limitations as it's still fairly new.
Yes this is something we could look at it, thanks.
LinkedIn
Twitter
