Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ContainerCluster: Update call failed: cannot make changes to immutable field(s) (Autopilot) #576

Open
2 of 3 tasks
flunderpero opened this issue Nov 26, 2021 · 5 comments
Open
2 of 3 tasks

ContainerCluster: Update call failed: cannot make changes to immutable field(s) (Autopilot) #576

flunderpero opened this issue Nov 26, 2021 · 5 comments
Labels
bug Something isn't working

Comments

@flunderpero
Copy link

Checklist

Bug Description

I create an Autopilot Cluster like this:

---
apiVersion: container.cnrm.cloud.google.com/v1beta1
kind: ContainerCluster
metadata:
  name: beta
spec:
  enableAutopilot: true
  location: europe-west1
  releaseChannel:
    channel: REGULAR
  minMasterVersion: 1.21.5-gke.1302

Everything is fine, but after a while, the resource moves to status UpdateFailed with the message:
Update call failed: cannot make changes to immutable field(s): [nodeConfig.0.Taint.0.Value nodeConfig.0.Taint.# nodeConfig.0.Taint.0.Effect nodeConfig.0.Taint.0.Key].

As you can see, I did not specify any taints. I use Spot-VMs and as a result, the nodes receive a taint during runtime. But that's nothing I could specify here because it relies purely on the actual pods/deployments running in the cluster.

I upgraded ConfigConnector to the latest version, but the problem still exists.

Additional Diagnostic Information

Kubernetes Cluster Version

Client Version: v1.20.8-dispatcher
Server Version: v1.20.10-gke.1600

Config Connector Version

1.67.0

Config Connector Mode

cluster

Steps to Reproduce

See above.
@flunderpero flunderpero added the bug Something isn't working label Nov 26, 2021
@jcanseco
Copy link
Member

jcanseco commented Dec 1, 2021

Hi @flunderpero, apologies for the late reply. I believe this might be due to a known issue where users cannot let list fields (and only list fields) be externally managed due to technical limitations with K8s that are a bit too complex to get into here.

Today, Config Connector takes ownership of list fields even if the user did not specify them in their original YAML, and pins the list to the last-read value. This means that if the list were to change under-the-hood, Config Connector will try to change it back, which results in an error in this case.

We have a feature that should allow you to bypass this issue, but we haven't rolled it out to all resources yet, so I'll note down this issue as another problem that can be fixed by the feature.

The feature allows users to annotate resources with state-into-spec: absent which tells Config Connector not to reflect non-user-specified state into the spec (i.e. keep the desired and observed states separate). This should fix your problem since it'll allow you tell Config Connector not to reflect list fields you did not specify yourself into the spec, allowing you to keep them externally managed.

Please let us know if that works for you. We'll update this thread when we've rolled out the feature to ContainerCluster.

@flunderpero
Copy link
Author

Hi @jcanseco, that would certainly work for us and in many cases, this would be exactly the behavior I'd expect.

@jcanseco
Copy link
Member

jcanseco commented Dec 2, 2021

Sounds great, thanks for confirming. I'll raise your need up internally and will let you know when we have rolled out the feature for ContainerCluster.

@jcanseco
Copy link
Member

jcanseco commented Dec 8, 2021

@flunderpero FYI: it seems there is an issue with the latest release (KCC v1.68.0) that causes your config above to fail to create. Please avoid upgrading to KCC v1.68.0. We'll see if we can put out a fix soon.

We should hopefully also be able to put out support for state-into-spec: absent for ContainerCluster soon which is currently in the works.

@xiaobaitusi
Copy link
Contributor

@flunderpero, with the release of 1.69.0, we have rolled out support for "state-into-spec: absent" to ContainerCluster. Feel free to give it try, and let me know if that satisfies your use case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@xiaobaitusi @jcanseco @flunderpero