태그 보관물: kubernetes

IT, K8S

Kubernetes cluster upgrade

2026년 1월 15일 Jason 댓글 남기기

minor 버전 한 단계씩, Control Plane부터 올리는 원칙

보안 패치와 기능 개선을 받으려면 Kubernetes cluster upgrade 절차를 주기적으로 거쳐야 합니다. Kubernetes 업그레이드는 한 번에 한 개의 minor 버전씩만 가능하며, 예를 들어 1.32에서 1.35로 올리려면 1.32 → 1.33 → 1.34 → 1.35 순서로 각 버전을 단계별로 진행해야 합니다.

업그레이드 도구는 kubeadm이며, 진행 순서는 반드시 Control Plane 노드 먼저, 이후 Worker 노드 순차적으로 진행해야 합니다. Control Plane 노드가 여러 개인 HA 구성에서는 각 master 노드를 한 번에 하나씩 업그레이드합니다.

노드 업그레이드 전후에는 kubectl drain과 kubectl uncordon으로 해당 노드의 스케줄링을 일시 중지하고, 업그레이드 완료 후 다시 활성화합니다. Ubuntu 환경에서는 apt-mark hold로 kubeadm, kubelet, kubectl 패키지를 고정해 두었다가 업그레이드 시점에만 해제하는 방식을 권장합니다.

이 가이드는 Ubuntu 24.04 기반의 단일 Control Plane 클러스터(master 1대, worker 2대)에서 1.32.x → 1.33.7 업그레이드를 진행한 절차를 기록합니다. kubeadm upgrade plan으로 가능한 업그레이드 대상 버전을 확인하고, kubeadm upgrade apply로 Control Plane을 업그레이드한 뒤, 각 Worker 노드에서 kubelet과 kubectl을 교체합니다.

Kubernetes cluster upgrade 순서를 나타낸 개념도로, Control Plane을 kubeadm upgrade apply로 먼저 올린 뒤 Worker 노드 두 대를 drain과 uncordon으로 한 대씩 v1.33.7로 맞추는 흐름 — Control Plane을 먼저 올리고 Worker는 한 대씩 drain·uncordon하며, 단계마다 kubectl get node로 버전을 확인합니다.

Kubernetes는 기능적/보안적인 이유로 주기적인 Cluster upgrade가 필연적입니다.

현재 버전인 1.32.*에서 1.33.* 버전으로 Upgrade가 목표이며 만일 1.35.* 같은 몇 단계 상위 버전 Upgrade를 계획하더라도 1.33, 1.34, 1.35 버전을 단계별로 순차 진행해야 되는 것은 동일합니다.

Master node가 이중화된 환경은 추가 과정이 있지만 과정 숙달을 목표로 메뉴얼을 기반하여 기본적인 순서로 진행해보겠습니다.

Master Node (Control Plane) 업그레이드

Control Plane 노드에서 kubeadm을 먼저 업그레이드하고, kubeadm upgrade plan으로 가능한 버전을 확인합니다. 이후 kubeadm upgrade apply로 API 서버, etcd, CoreDNS 등 핵심 컴포넌트를 일괄 업그레이드하고, kubelet/kubectl도 새 버전으로 교체합니다.

# 사용 명령어
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

sudo apt update
sudo apt-cache madison kubeadm

sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.33.7-*' && \
sudo apt-mark hold kubeadm

kubeadm version

sudo kubeadm upgrade plan

sudo kubeadm upgrade apply v1.33.7

kubectl drain <master-node> --ignore-daemonsets

sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.7-*' kubectl='1.33.7-*' && \
sudo apt-mark hold kubelet kubectl

sudo systemctl daemon-reload
sudo systemctl restart kubelet

kubectl uncordon <master-node>

# Log
test@<master-node>:~$ echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /
test@<master-node>:~$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
File '/etc/apt/keyrings/kubernetes-apt-keyring.gpg' exists. Overwrite? (y/N) y
test@<master-node>:~$ sudo apt update
Hit:1 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:4 https://download.docker.com/linux/ubuntu noble InRelease
Hit:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
230 packages can be upgraded. Run 'apt list --upgradable' to see them.
test@<master-node>:~$ sudo apt-cache madison kubeadm
   kubeadm | 1.33.7-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.6-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.5-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.4-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.3-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
test@<master-node>:~$ sudo apt-mark unhold kubeadm && \
> sudo apt-get update && sudo apt-get install -y kubeadm='1.33.7-*' && \
> sudo apt-mark hold kubeadm
kubeadm was already not on hold.
Hit:1 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:4 https://download.docker.com/linux/ubuntu noble InRelease
Hit:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubeadm'
The following packages will be upgraded:
  kubeadm
1 upgraded, 0 newly installed, 0 to remove and 229 not upgraded.
Need to get 12.7 MB of archives.
After this operation, 3,600 kB of additional disk space will be used.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubeadm 1.33.7-1.1 [12.7 MB]
Fetched 12.7 MB in 0s (33.8 MB/s)
(Reading database ... 196041 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.33.7-1.1_amd64.deb ...
Unpacking kubeadm (1.33.7-1.1) over (1.32.7-1.1) ...
Setting up kubeadm (1.33.7-1.1) ...
kubeadm set on hold.
test@<master-node>:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"33", EmulationMajor:"", EmulationMinor:"", MinCompatibilityMajor:"", MinCompatibilityMinor:"", GitVersion:"v1.33.7", GitCommit:"a7245cdf3f69e11356c7e8f92b3e78ca4ee4e757", GitTreeState:"clean", BuildDate:"2025-12-09T14:41:01Z", GoVersion:"go1.24.11", Compiler:"gc", Platform:"linux/amd64"}
test@<master-node>:~$ sudo kubeadm upgrade plan
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade/config] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: 1.32.7
[upgrade/versions] kubeadm version: v1.33.7
I0113 17:54:44.730671 2116008 version.go:261] remote version is much newer: v1.35.0; falling back to: stable-1.33
[upgrade/versions] Target version: v1.33.7
[upgrade/versions] Latest version in the v1.32 series: v1.32.11

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE                CURRENT   TARGET
kubelet     <master-node>   v1.32.7   v1.32.11
kubelet     <worker-node-01>   v1.32.7   v1.32.11
kubelet     <worker-node-02>   v1.32.7   v1.32.11

Upgrade to the latest version in the v1.32 series:

COMPONENT                 NODE                CURRENT    TARGET
kube-apiserver            <master-node>   v1.32.7    v1.32.11
kube-controller-manager   <master-node>   v1.32.7    v1.32.11
kube-scheduler            <master-node>   v1.32.7    v1.32.11
kube-proxy                                    1.32.7     v1.32.11
CoreDNS                                       v1.11.3    v1.12.0
etcd                      <master-node>   3.5.16-0   3.5.24-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.32.11

_____________________________________________________________________

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE                CURRENT   TARGET
kubelet     <master-node>   v1.32.7   v1.33.7
kubelet     <worker-node-01>   v1.32.7   v1.33.7
kubelet     <worker-node-02>   v1.32.7   v1.33.7

Upgrade to the latest stable version:

COMPONENT                 NODE                CURRENT    TARGET
kube-apiserver            <master-node>   v1.32.7    v1.33.7
kube-controller-manager   <master-node>   v1.32.7    v1.33.7
kube-scheduler            <master-node>   v1.32.7    v1.33.7
kube-proxy                                    1.32.7     v1.33.7
CoreDNS                                       v1.11.3    v1.12.0
etcd                      <master-node>   3.5.16-0   3.5.24-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.33.7

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________

test@<master-node>:~$ sudo kubeadm upgrade apply v1.33.7
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it.
[upgrade/preflight] Running preflight checks
[upgrade] Running cluster health checks
[upgrade/preflight] You have chosen to upgrade the cluster version to "v1.33.7"
[upgrade/versions] Cluster version: v1.32.7
[upgrade/versions] kubeadm version: v1.33.7
[upgrade] Are you sure you want to proceed? [y/N]: y
[upgrade/preflight] Pulling images required for setting up a Kubernetes cluster
[upgrade/preflight] This might take a minute or two, depending on the speed of your internet connection
[upgrade/preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[upgrade/control-plane] Upgrading your static Pod-hosted control plane to version "v1.33.7" (timeout: 5m0s)...
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests3418553132"
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Renewing etcd-peer certificate
[upgrade/staticpods] Renewing etcd-healthcheck-client certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-01-13-17-57-46/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 1 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-01-13-17-57-46/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-01-13-17-57-46/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-01-13-17-57-46/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade/control-plane] The control plane instance for this node was successfully upgraded!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upgrade/kubeconfig] The kubeconfig files for this node were successfully upgraded!
W0113 18:01:48.219276 2117131 postupgrade.go:117] Using temporary directory /etc/kubernetes/tmp/kubeadm-kubelet-config1781268223 for kubelet config. To override it set the environment variable KUBEADM_UPGRADE_DRYRUN_DIR
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config1781268223/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded!
[upgrade/bootstrap-token] Configuring bootstrap token and cluster-info RBAC rules
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

[upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.33.7".

[upgrade] Now please proceed with upgrading the rest of the nodes by following the right order.
test@<master-node>:~$ kubectl drain <master-node> --ignore-daemonsets
node/<master-node> cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/calico-node-km28t, kube-system/kube-proxy-ghl52
evicting pod kube-system/calico-kube-controllers-6d5bc68bd-vv654
pod/calico-kube-controllers-6d5bc68bd-vv654 evicted
node/<master-node> drained
test@<master-node>:~$ kubectl get node -o wide
NAME                STATUS                     ROLES           AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
<master-node>   Ready,SchedulingDisabled   control-plane   166d   v1.32.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
<worker-node-01>   Ready                      <none>          166d   v1.32.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
<worker-node-02>   Ready                      <none>          166d   v1.32.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
test@<master-node>:~$ sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.7-*' kubectl='1.33.7-*' && \
sudo apt-mark hold kubelet kubectl
Canceled hold on kubelet.
Canceled hold on kubectl.
Hit:1 https://download.docker.com/linux/ubuntu noble InRelease
Hit:2 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:4 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Hit:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubelet'
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubectl'
The following package was automatically installed and is no longer required:
  conntrack
Use 'sudo apt autoremove' to remove it.
The following packages will be upgraded:
  kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 227 not upgraded.
Need to get 27.6 MB of archives.
After this operation, 7,115 kB of additional disk space will be used.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubectl 1.33.7-1.1 [11.7 MB]
Get:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubelet 1.33.7-1.1 [15.9 MB]
Fetched 27.6 MB in 1s (51.6 MB/s)
(Reading database ... 196041 files and directories currently installed.)
Preparing to unpack .../kubectl_1.33.7-1.1_amd64.deb ...
Unpacking kubectl (1.33.7-1.1) over (1.32.7-1.1) ...
Preparing to unpack .../kubelet_1.33.7-1.1_amd64.deb ...
Unpacking kubelet (1.33.7-1.1) over (1.32.7-1.1) ...
Setting up kubectl (1.33.7-1.1) ...
Setting up kubelet (1.33.7-1.1) ...
kubelet set on hold.
kubectl set on hold.
test@<master-node>:~$ sudo systemctl daemon-reload
test@<master-node>:~$ sudo systemctl restart kubelet
test@<master-node>:~$ kubectl uncordon <master-node>
node/<master-node> uncordoned
test@<master-node>:~$ kubectl get node -o wide
NAME                STATUS   ROLES           AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
<master-node>   Ready    control-plane   166d   v1.33.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
<worker-node-01>   Ready    <none>          166d   v1.32.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
<worker-node-02>   Ready    <none>          166d   v1.32.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3

Worker Node 업그레이드 (#1)

Worker 노드 업그레이드는 Control Plane에서 kubectl drain으로 해당 Worker 노드를 비운 뒤 진행합니다. Worker 노드에서 kubeadm/kubelet/kubectl을 새 버전으로 교체하고, kubelet 재시작 후 Control Plane에서 kubectl uncordon으로 복귀시킵니다. 노드가 여러 개인 경우 동일 절차를 반복합니다.

# 사용 명령어
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

sudo apt update
sudo apt-cache madison kubeadm

sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.33.7-*' && \
sudo apt-mark hold kubeadm

kubeadm version

kubectl drain <worker-node-01> \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60 \
  --timeout=15m

sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.7-*' kubectl='1.33.7-*' && \
sudo apt-mark hold kubelet kubectl

sudo systemctl daemon-reload
sudo systemctl restart kubelet

kubectl uncordon <worker-node-01>

# Log
test@<worker-node-01>:~$ echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /
test@<worker-node-01>:~$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
File '/etc/apt/keyrings/kubernetes-apt-keyring.gpg' exists. Overwrite? (y/N) y
test@<worker-node-01>:~$ sudo apt update
Hit:1 https://download.docker.com/linux/ubuntu noble InRelease
Get:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease [1,230 B]
Get:3 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  Packages [11.3 kB]
Hit:4 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Get:5 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
Get:6 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
Get:7 http://security.ubuntu.com/ubuntu noble-security InRelease [126 kB]
Get:8 http://kr.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [1,693 kB]
Get:9 http://kr.archive.ubuntu.com/ubuntu noble-updates/main Translation-en [313 kB]
Get:10 http://kr.archive.ubuntu.com/ubuntu noble-updates/main amd64 Components [175 kB]
Get:11 http://kr.archive.ubuntu.com/ubuntu noble-updates/main amd64 c-n-f Metadata [15.9 kB]
Get:12 http://kr.archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Packages [2,426 kB]
Get:13 http://kr.archive.ubuntu.com/ubuntu noble-updates/restricted Translation-en [554 kB]
Get:14 http://kr.archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Components [212 B]
Get:15 http://kr.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [1,510 kB]
Get:16 http://kr.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Components [377 kB]
Get:17 http://kr.archive.ubuntu.com/ubuntu noble-updates/universe amd64 c-n-f Metadata [31.4 kB]
Get:18 http://kr.archive.ubuntu.com/ubuntu noble-updates/multiverse amd64 Components [940 B]
Get:19 http://kr.archive.ubuntu.com/ubuntu noble-backports/main amd64 Components [7,300 B]
Get:20 http://kr.archive.ubuntu.com/ubuntu noble-backports/restricted amd64 Components [216 B]
Get:21 http://kr.archive.ubuntu.com/ubuntu noble-backports/universe amd64 Components [10.5 kB]
Get:22 http://kr.archive.ubuntu.com/ubuntu noble-backports/multiverse amd64 Components [212 B]
Get:23 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages [1,404 kB]
Get:24 http://security.ubuntu.com/ubuntu noble-security/main Translation-en [228 kB]
Get:25 http://security.ubuntu.com/ubuntu noble-security/main amd64 Components [21.5 kB]
Get:26 http://security.ubuntu.com/ubuntu noble-security/restricted amd64 Packages [2,302 kB]
Get:27 http://security.ubuntu.com/ubuntu noble-security/restricted Translation-en [527 kB]
Get:28 http://security.ubuntu.com/ubuntu noble-security/restricted amd64 Components [208 B]
Get:29 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Components [71.4 kB]
Get:30 http://security.ubuntu.com/ubuntu noble-security/multiverse amd64 Components [208 B]
Fetched 12.1 MB in 3s (3,513 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
241 packages can be upgraded. Run 'apt list --upgradable' to see them.
test@<worker-node-01>:~$ sudo apt-cache madison kubeadm
   kubeadm | 1.33.7-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.6-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.5-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.4-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.3-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
test@<worker-node-01>:~$ sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.33.7-*' && \
sudo apt-mark hold kubeadm
Canceled hold on kubeadm.
Hit:1 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:4 https://download.docker.com/linux/ubuntu noble InRelease
Hit:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubeadm'
The following packages will be upgraded:
  kubeadm
1 upgraded, 0 newly installed, 0 to remove and 240 not upgraded.
Need to get 12.7 MB of archives.
After this operation, 3,600 kB of additional disk space will be used.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubeadm 1.33.7-1.1 [12.7 MB]
Fetched 12.7 MB in 0s (28.4 MB/s)
(Reading database ... 196036 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.33.7-1.1_amd64.deb ...
Unpacking kubeadm (1.33.7-1.1) over (1.32.7-1.1) ...
Setting up kubeadm (1.33.7-1.1) ...
kubeadm set on hold.
test@<worker-node-01>:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"33", EmulationMajor:"", EmulationMinor:"", MinCompatibilityMajor:"", MinCompatibilityMinor:"", GitVersion:"v1.33.7", GitCommit:"a7245cdf3f69e11356c7e8f92b3e78ca4ee4e757", GitTreeState:"clean", BuildDate:"2025-12-09T14:41:01Z", GoVersion:"go1.24.11", Compiler:"gc", Platform:"linux/amd64"}

test@<master-node>:~$ kubectl drain <worker-node-01> \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60 \
  --timeout=15m
node/<worker-node-01> cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/calico-node-fdff8, kube-system/kube-proxy-2rz9d
evicting pod kube-system/coredns-674b8bbfcf-7jxkk
pod/coredns-674b8bbfcf-7jxkk evicted
node/<worker-node-01> drained
test@<master-node>:~$ kubectl get node -o wide
NAME                STATUS                     ROLES           AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
<master-node>   Ready                      control-plane   166d   v1.33.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
<worker-node-01>   Ready,SchedulingDisabled   <none>          166d   v1.32.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
<worker-node-02>   Ready                      <none>          166d   v1.32.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3

test@<worker-node-01>:~$ sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.7-*' kubectl='1.33.7-*' && \
sudo apt-mark hold kubelet kubectl
Canceled hold on kubelet.
Canceled hold on kubectl.
Hit:1 https://download.docker.com/linux/ubuntu noble InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:4 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:5 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubelet'
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubectl'
The following package was automatically installed and is no longer required:
  conntrack
Use 'sudo apt autoremove' to remove it.
The following packages will be upgraded:
  kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 238 not upgraded.
Need to get 27.6 MB of archives.
After this operation, 7,115 kB of additional disk space will be used.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubectl 1.33.7-1.1 [11.7 MB]
Get:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubelet 1.33.7-1.1 [15.9 MB]
Fetched 27.6 MB in 1s (49.7 MB/s)
(Reading database ... 196036 files and directories currently installed.)
Preparing to unpack .../kubectl_1.33.7-1.1_amd64.deb ...
Unpacking kubectl (1.33.7-1.1) over (1.32.7-1.1) ...
Preparing to unpack .../kubelet_1.33.7-1.1_amd64.deb ...
Unpacking kubelet (1.33.7-1.1) over (1.32.7-1.1) ...
Setting up kubectl (1.33.7-1.1) ...
Setting up kubelet (1.33.7-1.1) ...
kubelet set on hold.
kubectl set on hold.
test@<worker-node-01>:~$ sudo systemctl daemon-reload
test@<worker-node-01>:~$ sudo systemctl restart kubelet
test@<worker-node-01>:~$ dpkg -l | grep kube
hi  kubeadm                                       1.33.7-1.1                               amd64        Command-line utility for administering a Kubernetes cluster
hi  kubectl                                       1.33.7-1.1                               amd64        Command-line utility for interacting with a Kubernetes cluster
hi  kubelet                                       1.33.7-1.1                               amd64        Node agent for Kubernetes clusters
ii  kubernetes-cni                                1.6.0-1.1                                amd64        Binaries required to provision kubernetes container networking

test@<master-node>:~$ kubectl uncordon <worker-node-01>
node/<worker-node-01> uncordoned
test@<master-node>:~$ kubectl get node -o wide
NAME                STATUS   ROLES           AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
<master-node>   Ready    control-plane   166d   v1.33.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
<worker-node-01>   Ready    <none>          166d   v1.33.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
<worker-node-02>   Ready    <none>          166d   v1.32.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3

Worker Node 업그레이드 (#2)

Worker 노드 #1과 동일한 절차를 Worker 노드 #2에 반복합니다. Control Plane에서 kubectl drain <worker-node-02>로 노드를 비우고, Worker 노드에서 kubeadm/kubelet/kubectl을 업그레이드한 뒤 kubectl uncordon으로 복귀시킵니다. 모든 Worker 노드 업그레이드 완료 후 kubectl get nodes -o wide로 전체 노드 버전이 동일한지 확인합니다. 세 노드가 모두 v1.33.7로 표시되면서 이번 Kubernetes cluster upgrade 과정을 마쳤습니다.

# 사용 명령어
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

sudo apt update
sudo apt-cache madison kubeadm

sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.33.7-*' && \
sudo apt-mark hold kubeadm

kubeadm version

kubectl drain <worker-node-02> \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60 \
  --timeout=15m

sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.7-*' kubectl='1.33.7-*' && \
sudo apt-mark hold kubelet kubectl

sudo systemctl daemon-reload
sudo systemctl restart kubelet

kubectl uncordon <worker-node-02>

# Log
test@<worker-node-02>:~$ echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /
test@<worker-node-02>:~$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
File '/etc/apt/keyrings/kubernetes-apt-keyring.gpg' exists. Overwrite? (y/N) y
test@<worker-node-02>:~$ sudo apt update
Hit:1 https://download.docker.com/linux/ubuntu noble InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Get:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease [1,230 B]
Get:4 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
Get:5 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
Get:6 http://security.ubuntu.com/ubuntu noble-security InRelease [126 kB]
Get:7 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  Packages [11.3 kB]
Get:8 http://kr.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [1,693 kB]
Get:9 http://kr.archive.ubuntu.com/ubuntu noble-updates/main Translation-en [313 kB]
Get:10 http://kr.archive.ubuntu.com/ubuntu noble-updates/main amd64 Components [175 kB]
Get:11 http://kr.archive.ubuntu.com/ubuntu noble-updates/main amd64 c-n-f Metadata [15.9 kB]
Get:12 http://kr.archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Packages [2,426 kB]
Get:13 http://kr.archive.ubuntu.com/ubuntu noble-updates/restricted Translation-en [554 kB]
Get:14 http://kr.archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Components [212 B]
Get:15 http://kr.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [1,510 kB]
Get:16 http://kr.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Components [377 kB]
Get:17 http://kr.archive.ubuntu.com/ubuntu noble-updates/universe amd64 c-n-f Metadata [31.4 kB]
Get:18 http://kr.archive.ubuntu.com/ubuntu noble-updates/multiverse amd64 Components [940 B]
Get:19 http://kr.archive.ubuntu.com/ubuntu noble-backports/main amd64 Components [7,300 B]
Get:20 http://kr.archive.ubuntu.com/ubuntu noble-backports/restricted amd64 Components [216 B]
Get:21 http://kr.archive.ubuntu.com/ubuntu noble-backports/universe amd64 Components [10.5 kB]
Get:22 http://kr.archive.ubuntu.com/ubuntu noble-backports/multiverse amd64 Components [212 B]
Get:23 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages [1,404 kB]
Get:24 http://security.ubuntu.com/ubuntu noble-security/main Translation-en [228 kB]
Get:25 http://security.ubuntu.com/ubuntu noble-security/main amd64 Components [21.5 kB]
Get:26 http://security.ubuntu.com/ubuntu noble-security/restricted amd64 Packages [2,302 kB]
Get:27 http://security.ubuntu.com/ubuntu noble-security/restricted Translation-en [527 kB]
Get:28 http://security.ubuntu.com/ubuntu noble-security/restricted amd64 Components [208 B]
Get:29 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Components [71.4 kB]
Get:30 http://security.ubuntu.com/ubuntu noble-security/universe amd64 c-n-f Metadata [19.7 kB]
Get:31 http://security.ubuntu.com/ubuntu noble-security/multiverse amd64 Components [208 B]
Fetched 12.1 MB in 4s (3,410 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
229 packages can be upgraded. Run 'apt list --upgradable' to see them.
test@<worker-node-02>:~$ sudo apt-cache madison kubeadm
   kubeadm | 1.33.7-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.6-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.5-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.4-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.3-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
test@<worker-node-02>:~$ sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.33.7-*' && \
sudo apt-mark hold kubeadm
Canceled hold on kubeadm.
Hit:1 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:4 https://download.docker.com/linux/ubuntu noble InRelease
Hit:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubeadm'
The following packages will be upgraded:
  kubeadm
1 upgraded, 0 newly installed, 0 to remove and 228 not upgraded.
Need to get 12.7 MB of archives.
After this operation, 3,600 kB of additional disk space will be used.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubeadm 1.33.7-1.1 [12.7 MB]
Fetched 12.7 MB in 1s (23.5 MB/s)
(Reading database ... 196021 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.33.7-1.1_amd64.deb ...
Unpacking kubeadm (1.33.7-1.1) over (1.32.7-1.1) ...
Setting up kubeadm (1.33.7-1.1) ...
kubeadm set on hold.
test@<worker-node-02>:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"33", EmulationMajor:"", EmulationMinor:"", MinCompatibilityMajor:"", MinCompatibilityMinor:"", GitVersion:"v1.33.7", GitCommit:"a7245cdf3f69e11356c7e8f92b3e78ca4ee4e757", GitTreeState:"clean", BuildDate:"2025-12-09T14:41:01Z", GoVersion:"go1.24.11", Compiler:"gc", Platform:"linux/amd64"}

test@<master-node>:~$ kubectl drain <worker-node-02> \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60 \
  --timeout=15m
node/<worker-node-02> cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/calico-node-drhtt, kube-system/kube-proxy-hpkd8
evicting pod kube-system/coredns-674b8bbfcf-f6txf
evicting pod kube-system/calico-kube-controllers-6d5bc68bd-7mppz
pod/calico-kube-controllers-6d5bc68bd-7mppz evicted
pod/coredns-674b8bbfcf-f6txf evicted
node/<worker-node-02> drained
test@<master-node>:~$ kubectl get node -o wide
NAME                STATUS                     ROLES           AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
<master-node>   Ready                      control-plane   166d   v1.33.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
<worker-node-01>   Ready                      <none>          166d   v1.33.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
<worker-node-02>   Ready,SchedulingDisabled   <none>          166d   v1.32.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3

test@<worker-node-02>:~$ sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.7-*' kubectl='1.33.7-*' && \
sudo apt-mark hold kubelet kubectl
Canceled hold on kubelet.
Canceled hold on kubectl.
Hit:1 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:4 https://download.docker.com/linux/ubuntu noble InRelease
Hit:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubelet'
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubectl'
The following package was automatically installed and is no longer required:
  conntrack
Use 'sudo apt autoremove' to remove it.
The following packages will be upgraded:
  kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 226 not upgraded.
Need to get 27.6 MB of archives.
After this operation, 7,115 kB of additional disk space will be used.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubectl 1.33.7-1.1 [11.7 MB]
Get:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubelet 1.33.7-1.1 [15.9 MB]
Fetched 27.6 MB in 1s (47.2 MB/s)
(Reading database ... 196021 files and directories currently installed.)
Preparing to unpack .../kubectl_1.33.7-1.1_amd64.deb ...
Unpacking kubectl (1.33.7-1.1) over (1.32.7-1.1) ...
Preparing to unpack .../kubelet_1.33.7-1.1_amd64.deb ...
Unpacking kubelet (1.33.7-1.1) over (1.32.7-1.1) ...
Setting up kubectl (1.33.7-1.1) ...
Setting up kubelet (1.33.7-1.1) ...
kubelet set on hold.
kubectl set on hold.
test@<worker-node-02>:~$ sudo systemctl daemon-reload
test@<worker-node-02>:~$ sudo systemctl restart kubelet
test@<worker-node-02>:~$ dpkg -l | grep kube
hi  kubeadm                                       1.33.7-1.1                               amd64        Command-line utility for administering a Kubernetes cluster
hi  kubectl                                       1.33.7-1.1                               amd64        Command-line utility for interacting with a Kubernetes cluster
hi  kubelet                                       1.33.7-1.1                               amd64        Node agent for Kubernetes clusters
ii  kubernetes-cni                                1.6.0-1.1                                amd64        Binaries required to provision kubernetes container networking

test@<master-node>:~$ kubectl uncordon <worker-node-02>
node/<worker-node-02> uncordoned
test@<master-node>:~$ kubectl get node -o wide
NAME                STATUS   ROLES           AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
<master-node>   Ready    control-plane   166d   v1.33.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
<worker-node-01>   Ready    <none>          166d   v1.33.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
<worker-node-02>   Ready    <none>          166d   v1.33.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3

업그레이드 주의사항

Kubernetes 클러스터 업그레이드 시 주의해야 할 사항을 정리합니다.

버전 건너뛰기 불가 — minor 버전은 반드시 한 단계씩 업그레이드해야 합니다. 예를 들어 1.31 → 1.33으로 직접 업그레이드할 수 없습니다.
etcd 백업 선행 — Control Plane 업그레이드 전 etcd 스냅샷을 백업해 두면 문제 발생 시 롤백이 가능합니다.
drain 전 PodDisruptionBudget 확인 — kubectl drain 시 PDB(PodDisruptionBudget)에 의해 eviction이 차단될 수 있습니다. --delete-emptydir-data와 --grace-period 옵션을 적절히 설정합니다.
CNI 플러그인 호환성 확인 — Calico, Flannel 등 CNI 플러그인이 업그레이드 대상 Kubernetes 버전과 호환되는지 릴리스 노트를 미리 확인합니다.
HA 클러스터 — master 노드가 여러 개인 경우 한 번에 하나씩 업그레이드하며, 각 master 업그레이드 후 etcd quorum이 유지되는지 확인합니다.

참고

Ceph MDS Pod Anti-Affinity Troubleshoot

2025년 12월 29일 Jason 댓글 남기기

drain 이후 한 노드로 몰리는 MDS 파드

Kubernetes 클러스터에서 노드 업그레이드나 drain 작업을 수행할 때, Ceph MDS Pod Anti-Affinity 설정이 없으면 MDS 파드 두 개가 동일한 워커 노드에 집중 배치되는 문제가 발생할 수 있습니다. 이 글에서는 node drain 후 MDS 파드가 한 노드에 몰린 상황을 확인하고, CephFilesystem 리소스에 podAntiAffinity 규칙을 적용하여 MDS 파드를 분산 배치하는 과정을 정리합니다. 이어서 함께 발생한 mgr crash 알람 처리 방법도 설명합니다.

Ceph MDS Pod Anti-Affinity 적용 전에는 MDS 파드 두 개가 워커 노드 2에 몰려 있고, CephFilesystem 패치 후에는 active와 standby-replay가 서로 다른 워커 노드로 나뉘는 배치를 비교한 개념도 — podAntiAffinity 적용 후에는 drain 중에도 active와 standby-replay MDS가 서로 다른 노드에 배치됩니다.

문제 상황

워커 노드 1번에 대해 drain을 수행한 후 Ceph 상태를 확인하면 HEALTH_WARN이 발생하고, MDS 파드 2개가 모두 워커 노드 2번에서 기동 중임을 확인할 수 있습니다.

# ceph 상태 확인
<admin-user>@<master-node>:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status
  cluster:
    id:     xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    health: HEALTH_WARN
            4 mgr modules have recently crashed

  services:
    mon: 3 daemons, quorum a,b,c (age 10h)
    mgr: b(active, since 5M), standbys: a
    mds: 1/1 daemons up, 1 hot standby
    osd: 3 osds: 3 up (since 10h), 3 in (since 18M)

# MDS pod 위치 확인 — 두 파드 모두 <worker-node-2>에 집중
<admin-user>@<master-node>:~$ kubectl -n rook-ceph get pod -o wide | egrep 'mds'
rook-ceph-mds-myfs-a-77d484dc4-jddf9  2/2  Running  0  18s  172.16.x.x  <worker-node-2>  <none>  <none>
rook-ceph-mds-myfs-b-bd6ddc59b-l2b4t  2/2  Running  0  18s  172.16.x.x  <worker-node-2>  <none>  <none>

원인 분석

ceph fs status에서 확인하면 active MDS와 standby-replay MDS 모두 정상 동작 중이지만, 두 파드가 같은 노드에 배치되어 있어 해당 노드에 장애가 발생하면 CephFS 서비스 전체가 중단될 위험이 있습니다. Anti-Affinity 규칙이 설정되어 있지 않으면 Kubernetes 스케줄러가 가용 자원이 충분한 노드에 임의로 배치하기 때문에 이런 현상이 발생합니다.

# cephFS 상태 확인
<admin-user>@<master-node>:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph fs status
myfs - 2 clients
====
RANK      STATE          MDS     ACTIVITY     DNS    INOS   DIRS   CAPS
 0        active        myfs-b  Reqs:  0 /s  35.9k  18.0k  4301      2
0-s   standby-replay   myfs-a  Evts:  0 /s  35.9k  18.0k  4301      0
MDS version: ceph version 18.2.2 reef (stable)

Ceph MDS Pod Anti-Affinity 적용

CephFilesystem 리소스에 podAntiAffinity를 설정하면 동일 레이블의 MDS 파드가 같은 노드에 배치되지 않도록 강제할 수 있습니다. requiredDuringSchedulingIgnoredDuringExecution을 사용하면 조건을 만족하지 못할 경우 파드가 아예 스케줄링되지 않으므로 강하게 분산을 보장합니다.

# CephFilesystem에 podAntiAffinity 패치 적용
<admin-user>@<master-node>:~$ kubectl -n rook-ceph patch cephfilesystem myfs --type='merge' -p '
spec:
  metadataServer:
    placement:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values: ["rook-ceph-mds"]
            - key: rook_file_system
              operator: In
              values: ["myfs"]
          topologyKey: kubernetes.io/hostname
'
cephfilesystem.ceph.rook.io/myfs patched

패치 후 kubectl -n rook-ceph get cephfilesystem myfs -o yaml에서 spec.metadataServer.placement.podAntiAffinity 항목이 반영되었는지 확인합니다.

워커 노드 2번 drain으로 분산 배치 검증

워커 노드 1번을 uncordon하고 워커 노드 2번을 drain하면 MDS 파드가 Ceph MDS Pod Anti-Affinity 규칙에 따라 서로 다른 노드(worker-01, worker-03)에 분산 배치됩니다.

# Anti-Affinity 적용 후 MDS 파드 분산 확인
<admin-user>@<master-node>:~$ kubectl -n rook-ceph get pod -o wide | egrep 'mds'
rook-ceph-mds-myfs-a-58846844d6-nd5mk  2/2  Running  0  53s  172.16.x.x  <worker-node-1>  <none>  <none>
rook-ceph-mds-myfs-b-6b4d9476cb-q6b6p  2/2  Running  0  38s  172.16.x.x  <worker-node-3>  <none>  <none>

# cephFS 상태 정상 확인
<admin-user>@<master-node>:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph fs status
myfs - 2 clients
====
RANK      STATE          MDS     ACTIVITY     DNS    INOS   DIRS   CAPS
 0        active        myfs-a  Reqs:  0 /s  35.9k  18.0k  4301      2
0-s   standby-replay   myfs-b  Evts:  0 /s  35.9k  18.0k  4301      0
MDS version: ceph version 18.2.2 reef (stable)

mgr crash 알람 처리

MDS 분산 이후에도 4 mgr modules have recently crashed 알람이 남아 있을 수 있습니다. MDS 이슈와는 무관하게 mgr 파드가 재시작되며 남은 crash 이력이므로, ceph mgr stat에서 available: true이면 서비스는 정상입니다. ceph crash archive-all로 이력을 정리하면 알람이 해소됩니다.

# mgr 상태 정상 확인 (available: true)
<admin-user>@<master-node>:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph mgr stat
{
    "epoch": 476,
    "available": true,
    "active_name": "b",
    "num_standby": 1
}

# crash 이력 목록 확인
<admin-user>@<master-node>:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph crash ls
ID                                                                ENTITY  NEW
2025-12-26T09:01:17.354121Z_c76c6eaf-4bf7-4cf9-a9ec-f646fe857b76  mgr.b    *
2025-12-26T09:01:32.345473Z_4dfd271c-3d5b-4c89-88cf-13ba096f327b  mgr.b    *
2025-12-26T09:01:47.357321Z_0f938fb6-4c50-4b58-815d-5990fbe4bbb7  mgr.b    *
2025-12-26T09:02:02.329492Z_43d344a7-b71f-442e-a664-1852dda3a3f3  mgr.b    *

# crash 이력 아카이브 후 HEALTH_OK 확인
<admin-user>@<master-node>:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph crash archive-all
<admin-user>@<master-node>:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status
  cluster:
    health: HEALTH_OK
  services:
    mds: 1/1 daemons up, 1 hot standby
    osd: 3 osds: 3 up, 3 in

node drain / uncordon 작업 중 일시적으로 mon quorum 이탈이나 rebalancing이 발생할 수 있으나, 일정 시간 후 재확인하면 HEALTH_OK 상태로 복구됩니다.

참고

Ceph 관련 운영 내용은 아래 포스트도 참고하시기 바랍니다.

Ceph OSD down 알람 처리 — 불필요한 OSD 제거 및 클러스터 복구
Ceph 구축 — Rook-Ceph 초기 구축 가이드
Ceph Storage Class를 활용한 WordPress/MySQL Pod 생성

참고 문서: Rook CephFilesystem CRD 공식 문서 · Kubernetes Pod Anti-Affinity 공식 문서

K8S

Kubernetes 기본 설치(Version 1.28)

2023년 12월 20일 Jason 댓글 남기기

cri-dockerd와 Calico로 구성하는 단일 마스터 클러스터

이 글은 Ubuntu 22.04 환경에서 kubeadm으로 1.28 클러스터를 구성한 Kubernetes 기본 설치 기록입니다. Kubernetes 1.24부터 Docker shim이 제거되었기 때문에 Docker 런타임을 계속 사용하려면 cri-dockerd를 컨테이너 런타임 인터페이스(CRI)로 별도 설치해야 합니다. CNI 플러그인은 Calico를 사용하며, 마스터 노드 1개와 워커 노드 2개로 구성하는 것을 기준으로 작성하였습니다.

버전 범위 안내: 이 글은 Kubernetes 1.28.2와 Ubuntu 22.04.3에서 검증한 과거 구축 기록입니다. Kubernetes 1.28은 현재 지원이 종료된 버전이므로 신규 구축 절차로 그대로 사용하지 말고, Kubernetes 공식 지원 상태와 최신 설치 문서를 먼저 확인하시기 바랍니다.

이 Kubernetes 기본 설치 가이드는 단일 마스터 클러스터 기준입니다. HA(고가용성) 멀티 마스터 구성은 HAProxy 로드밸런서와 etcd 클러스터 구성이 추가로 필요합니다.

모든 노드 공통 준비 후 마스터의 kubeadm init과 Calico 적용, 워커 2대의 kubeadm join, kubectl get nodes 확인으로 이어지는 Kubernetes 기본 설치 순서 구성도 — Kubernetes 기본 설치 순서 — 모든 노드에 공통 준비를 마친 뒤 마스터에서 init과 Calico를, 워커에서 join을 실행합니다.

구성 내역

설치 환경은 다음과 같습니다.

Kubernetes 1.28.2
Ubuntu 22.04.3 LTS
Container : cri-dockerd
CNI : calico
구성용 계정 : <admin-user>
Master node : <master-node>
Worker node : <worker-node-01> ~ <worker-node-02>

시작 전 모든 노드에서 swap을 비활성화해야 합니다. kubelet은 swap이 활성화된 환경에서 기본적으로 동작을 거부합니다.

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#/g' /etc/fstab

[공통]

cri-docker 설치

Kubernetes 1.24 이후 Docker shim이 제거되면서 Docker를 컨테이너 런타임으로 사용하려면 cri-dockerd를 별도로 설치해야 합니다. cri-dockerd는 Mirantis가 관리하는 오픈소스 프로젝트로, Docker Engine과 kubelet 사이의 CRI(Container Runtime Interface) 어댑터 역할을 합니다. 먼저 Docker Engine을 설치한 뒤 cri-dockerd 바이너리를 받아 systemd 서비스로 등록합니다.

Docker cgroup driver를 systemd로 맞추는 것이 중요합니다. kubelet도 기본적으로 systemd cgroup driver를 사용하므로, driver 불일치 시 노드가 NotReady 상태가 됩니다.

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo systemctl enable --now docker && sudo systemctl status docker --no-pager
sudo usermod -aG docker <admin-user>
sudo docker container ls

# cri-docker Install
VER=$(curl -s https://api.github.com/repos/Mirantis/cri-dockerd/releases/latest|grep tag_name | cut -d '"' -f 4|sed 's/v//g')
echo $VER
wget https://github.com/Mirantis/cri-dockerd/releases/download/v${VER}/cri-dockerd-${VER}.amd64.tgz
tar xvf cri-dockerd-${VER}.amd64.tgz
sudo mv cri-dockerd/cri-dockerd /usr/local/bin/

# cri-docker Version Check
cri-dockerd --version

wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.service
wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/master/packaging/systemd/cri-docker.socket
sudo mv cri-docker.socket cri-docker.service /etc/systemd/system/
sudo sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service

sudo systemctl daemon-reload
sudo systemctl enable cri-docker.service
sudo systemctl enable --now cri-docker.socket

# cri-docker Active Check
sudo systemctl restart docker && sudo systemctl restart cri-docker
sudo systemctl status cri-docker.socket --no-pager

# Docker cgroup Change Require to Systemd
sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF

sudo systemctl restart docker && sudo systemctl restart cri-docker
sudo docker info | grep Cgroup

환경 설정

Kubernetes 네트워킹이 정상 동작하려면 커널 모듈과 sysctl 파라미터를 설정해야 합니다. br_netfilter 모듈은 Linux Bridge를 통과하는 패킷을 iptables/ip6tables에서 처리할 수 있게 해주며, overlay 모듈은 컨테이너 파일시스템에 사용하는 OverlayFS를 위해 필요합니다. net.ipv4.ip_forward = 1은 노드가 패킷을 다른 노드로 포워딩할 수 있도록 허용합니다. 이 설정은 모든 노드(마스터 + 워커)에 적용해야 합니다.

# Kernel Forwarding
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

sudo sysctl --system

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# 필요한 sysctl 파라미터를 설정하면, 재부팅 후에도 값이 유지된다.
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF

# 재부팅하지 않고 sysctl 파라미터 적용하기
sudo sysctl --system

Package 설치

kubeadm, kubelet, kubectl 세 패키지를 설치합니다. 각 패키지의 역할은 다음과 같습니다.

kubeadm — 클러스터 초기화(init)와 노드 합류(join)를 담당하는 부트스트랩 도구
kubelet — 각 노드에서 실행되며 Pod 생명주기를 관리하는 에이전트
kubectl — 클러스터를 제어하는 CLI 클라이언트

apt-mark hold로 패키지를 고정하면 apt upgrade로 인한 의도치 않은 버전 업그레이드를 방지할 수 있습니다. Kubernetes는 컴포넌트 버전을 맞춰 관리해야 하므로 반드시 hold 설정을 유지하고, 업그레이드 시에는 kubeadm upgrade 절차를 따릅니다.

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl

sudo curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://dl.k8s.io/apt/doc/apt-key.gpg && echo "deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

sudo systemctl daemon-reload
sudo systemctl restart kubelet

위 방법으로 패키지를 찾지 못하는 경우(E: Unable to locate package kubelet), pkgs.k8s.io 신규 리포지토리를 사용합니다.

$ sudo apt-get install -y kubelet kubeadm kubectl
E: Unable to locate package kubelet
E: Unable to locate package kubeadm
E: Unable to locate package kubectl

# kubelet kubeadm kubectl 설치시 위 에러 발생할 경우 아래 방안으로 설치
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.28/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.28/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

sudo systemctl daemon-reload
sudo systemctl restart kubelet

[Master node]

kubeadm init으로 마스터 노드를 초기화합니다. --cri-socket 옵션으로 cri-dockerd 소켓을 명시하지 않으면 kubeadm이 런타임을 자동 감지하는 과정에서 오류가 발생할 수 있습니다. 초기화가 완료되면 출력 마지막에 워커 노드 join 명령어가 표시됩니다. 이 명령어의 토큰은 24시간 후 만료되므로 즉시 복사해 두거나 kubeadm token create --print-join-command로 재발급합니다.

kubeconfig 설정 후 Calico CNI를 설치합니다. CNI가 설치되기 전까지는 노드 상태가 NotReady로 표시되며, calico.yaml 적용 후 수 분 내에 Ready로 전환됩니다.

sudo kubeadm config images pull --cri-socket unix:///run/cri-dockerd.sock
sudo kubeadm init --cri-socket /var/run/cri-dockerd.sock

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

kubectl get nodes -o wide
kubectl get pods -A
kubectl describe node <master-node>

# Calico CNI 설치
curl https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml -O
kubectl apply -f calico.yaml

kubectl get nodes
kubectl get pod --all-namespaces

[worker node]

워커 노드에서 kubeadm join 명령어를 실행하여 클러스터에 합류합니다. kubeadm init 출력에서 복사한 명령어를 그대로 사용하되, --cri-socket 옵션을 추가해야 합니다. join이 완료된 후 마스터 노드에서 kubectl get nodes를 실행하여 워커 노드가 Ready 상태로 등록되는지 확인합니다.

# kubeadm init 실행시 마지막 출력되는 명령어 사용
sudo kubeadm join x.x.x.x:6443 --token xxxxxxxxxxxx --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxx --cri-socket /var/run/cri-dockerd.sock

kubectl get nodes

kubectl 명령어 자동 완성

kubectl bash completion을 설정하면 Tab 키로 리소스명, 네임스페이스 등을 자동 완성할 수 있어 운영 편의성이 크게 향상됩니다. alias k=kubectl과 함께 설정하면 짧은 명령어로 빠르게 조작할 수 있습니다. 자세한 내용은 kubectl 자동 완성 설정 — Kubernetes 공식 문서를 참고하세요.

echo 'source <(kubectl completion bash)' >> ~/.bashrc
echo 'alias k=kubectl' >> ~/.bashrc
echo 'complete -o default -F __start_kubectl k' >> ~/.bashrc
source ~/.bashrc

노드 Ready와 kube-system 파드 상태 확인

모든 노드가 정상적으로 등록되고 kube-system 파드들이 Running 상태가 되면 클러스터 구성이 완료된 것입니다. 아래 명령어로 최종 상태를 확인합니다.

# 노드 상태 확인 — 모든 노드 Ready 여야 정상
kubectl get nodes -o wide

# 시스템 파드 상태 확인 — 모두 Running 또는 Completed
kubectl get pods -n kube-system

# 클러스터 정보 확인
kubectl cluster-info

워커 노드가 NotReady로 유지될 경우 해당 노드에서 sudo systemctl status kubelet과 journalctl -u kubelet -n 50으로 에러 메시지를 확인합니다. 대부분은 cgroup driver 불일치 또는 cri-dockerd 미기동으로 인한 문제입니다.

참고

참고 문서: kubeadm으로 클러스터 생성 — Kubernetes 공식 문서 · Calico 온프레미스 설치 문서

IT Garage