Description
What happened?
When I was attempting to upgrade my kubernetes cluster from version 1.29.15 to 1.30.14 using kubeadm in a completely offline environment, I found that kubeadm will always pull the images of the control plane. So it will get stuck due to the network problem and my kubernetes cannot be upgraded normally.
mally.
Here is the log:
[root]# kubeadm upgrade apply --config /root/kubeadm-upgrade.yaml --yes --v=5
I0630 15:12:33.327224 1939880 apply.go:111] [upgrade/apply] verifying health of cluster
I0630 15:12:33.327323 1939880 apply.go:112] [upgrade/apply] retrieving configuration from cluster
W0630 15:12:33.327851 1939880 upgradeconfiguration.go:44] [config] WARNING: YAML document with GroupVersionKind kubeadm.k8s.io/v1beta4, Kind=InitConfiguration is deprecated for upgrade, please use config file with kind of UpgradeConfiguration instead
W0630 15:12:33.328261 1939880 upgradeconfiguration.go:60] error unmarshaling configuration schema.GroupVersionKind{Group:"kubeadm.k8s.io", Version:"v1beta4", Kind:"UpgradeConfiguration"}: strict decoding error: unknown field "apply.IgnorePreflightErrors"
I0630 15:12:33.330758 1939880 common.go:94] running preflight checks
[preflight] Running pre-flight checks.
I0630 15:12:33.330825 1939880 preflight.go:77] validating if there are any unsupported CoreDNS plugins in the Corefile
I0630 15:12:33.347832 1939880 preflight.go:109] validating if migration can be done for the current CoreDNS release.
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
I0630 15:12:33.363538 1939880 kubeproxy.go:55] attempting to download the KubeProxyConfiguration from ConfigMap "kube-proxy"
I0630 15:12:33.368961 1939880 kubelet.go:74] attempting to download the KubeletConfiguration from ConfigMap "kubelet-config"
[upgrade] Running cluster health checks
I0630 15:12:33.401298 1939880 health.go:176] Creating a Job with the prefix "upgrade-health-check" in the namespace "kube-system"
I0630 15:12:33.431488 1939880 health.go:207] Job "upgrade-health-check-fdp9c" in the namespace "kube-system" is not yet complete, retrying
I0630 15:12:34.433639 1939880 health.go:207] Job "upgrade-health-check-fdp9c" in the namespace "kube-system" is not yet complete, retrying
I0630 15:12:35.433735 1939880 health.go:207] Job "upgrade-health-check-fdp9c" in the namespace "kube-system" is not yet complete, retrying
I0630 15:12:36.437820 1939880 health.go:207] Job "upgrade-health-check-fdp9c" in the namespace "kube-system" is not yet complete, retrying
I0630 15:12:37.433578 1939880 health.go:214] Job "upgrade-health-check-fdp9c" in the namespace "kube-system" completed
I0630 15:12:37.439396 1939880 apply.go:119] [upgrade/apply] validating requested and actual version
I0630 15:12:37.439457 1939880 apply.go:135] [upgrade/version] enforcing version skew policies
[upgrade/version] You have chosen to change the cluster version to "v1.30.14"
[upgrade/versions] Cluster version: v1.29.15
[upgrade/versions] kubeadm version: v1.30.14
[upgrade/prepull] Pulling images required for setting up a Kubernetes cluster
[upgrade/prepull] This might take a minute or two, depending on the speed of your internet connection
[upgrade/prepull] You can also perform this action in beforehand using 'kubeadm config images pull'
I0630 15:12:37.445667 1939880 checks.go:830] using image pull policy: IfNotPresent
I0630 15:12:37.492320 1939880 checks.go:870] pulling: registry.k8s.io/kube-apiserver:v1.30.14
......And here is my UpgradeConfiguration:
apiVersion: kubeadm.k8s.io/v1beta4
kind: UpgradeConfiguration
apply:
certificateRenewal: false
kubernetesVersion: v1.30.14
In the previous version of kubeadm, such as v1.29. it was possible not to pull the images while upgrading. Use the InitConfiguration, like:
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
nodeRegistration:
imagePullPolicy: neverHowever, after version 1.30 of kubernetes, InitConfiguration was deprecated so there is no place to configure imagePullPolicy. And the UpgradeConfiguration does not provide a similar configuration items neither. This leads to the fact that the control plane images will always be pulled during the upgrade.
What did you expect to happen?
Run the kubeadm upgrade apply command to no longer pull the control plane images
How can we reproduce it (as minimally and precisely as possible)?
Install a kubernetets cluster of version 1.29.15, manually import the control plane images of version 1.30.14, and use kubeadm v1.30.14 bykubeadm upgrade apply command for upgrading the cluster to 1.30.14 in a toltally offline enviroment. It will always be unable to upgrade due to the failure to pull the images.
Anything else we need to know?
No response
Kubernetes version
Cloud provider
OS version
# cat /etc/os-release
NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
# uname -a
Linux 10-9-8-200 4.18.0-348.el8.x86_64 #1 SMP Tue Oct 19 15:14:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux