카테고리 보관물: Container

Certificate, Container, K8S, WEB

Migrating Ingress Controller: ingress-nginx EOL to Traefik v3.7.5 with Wildcard TLS Automation

댓글 남기기

개요

kubernetes/ingress-nginx 프로젝트가 2026년 3월 31일부로 EOL(End of Life)을 선언하고 아카이브되었습니다. 이를 계기로 ingress-nginx에서 Traefik으로 마이그레이션을 진행하여 자체 관리형 Kubernetes 클러스터의 Ingress 컨트롤러를 Traefik v3.7.5로 교체하고, 모니터링 스택(Prometheus / Alertmanager / Grafana) 도메인을 *.sierracloud.dev로 전환하였습니다. 아울러 HAProxy 서버에서 관리하는 Let’s Encrypt 와일드카드 인증서를 Kubernetes 클러스터에 자동 동기화하는 CronJob을 구성하였습니다.

대안으로 Contour, Kong, HAProxy Ingress 등을 검토하였으나, Traefik을 선택한 이유는 다음과 같습니다. Helm chart가 잘 관리되고 있으며, TLSStore를 통한 와일드카드 인증서 중앙 관리가 가능합니다. 또한 CRD(IngressRoute, Middleware 등)를 통한 고급 라우팅 설정과 Kubernetes Ingress 표준 오브젝트와의 호환성을 동시에 지원합니다. HTTP → HTTPS 강제 리다이렉트도 values.yaml 설정 한 줄로 처리됩니다.

환경

  • Kubernetes v1.33.7 (HA: Control Plane 3대 + Worker 3대)
  • MetalLB — LoadBalancer IP 풀: 192.168.x.x/29
  • HAProxy (192.168.x.x:6443) — K8s API LB 및 HTTPS 리버스 프록시 겸용
  • NAS — Let’s Encrypt 인증서 원본 보관, NFS export
  • 교체 전: kubernetes/ingress-nginx v1.10.1 (EOL)
  • 교체 후: Traefik v3.7.5 (Helm chart 41.0.0)

단계별 절차

1. ingress-nginx 설정 백업

삭제 전에 기존 설정을 코드로 백업하여 재설치 시 활용할 수 있도록 보존합니다.

# IngressClass, ConfigMap, Ingress 리소스 백업
kubectl get ingressclass nginx -o yaml > backup/ingress-nginx/ingressclass-nginx.yaml
kubectl get cm ingress-nginx-controller -n ingress-nginx -o yaml > backup/ingress-nginx/configmap-controller.yaml
kubectl get ingress -n monitoring -o yaml > backup/ingress-nginx/ingress-monitoring.yaml

# Helm values 백업
helm get values ingress-nginx -n ingress-nginx -o yaml > backup/ingress-nginx/helm-ingress-nginx-values.yaml

# 삭제 절차 문서화 후 제거
helm uninstall ingress-nginx -n ingress-nginx

2. Traefik v3.7.5 설치

ingress-nginx가 사용하던 MetalLB IP를 그대로 유지하여 HAProxy의 백엔드 설정 변경 없이 전환합니다.

helm repo add traefik https://traefik.github.io/charts
helm repo update

helm upgrade --install traefik traefik/traefik \
  -f helm/traefik-values.yaml \
  -n traefik --create-namespace

helm/traefik-values.yaml 핵심 설정:

service:
  annotations:
    metallb.universe.tf/loadBalancerIPs: "192.168.x.x"   # 기존 IP 유지

ingressClass:
  enabled: true
  isDefaultClass: true

ports:
  web:
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
          permanent: true   # HTTP → HTTPS 전체 리다이렉트

3. 모니터링 도메인 변경

*.sierracloud.kro.kr에서 *.sierracloud.dev로 전환하고 ingressClassName을 교체합니다. Helm values 수정 후 upgrade를 적용합니다.

# helm/monitoring-values.yaml (변경 부분)
prometheus:
  ingress:
    ingressClassName: traefik    # nginx → traefik
    hosts:
      - prometheus.sierracloud.dev

grafana:
  ingress:
    ingressClassName: traefik
    hosts:
      - grafana.sierracloud.dev
  grafana.ini:
    server:
      domain: grafana.sierracloud.dev
      root_url: https://grafana.sierracloud.dev
      protocol: http             # Traefik이 TLS 종료, Grafana 내부는 HTTP
helm upgrade monitoring prometheus-community/kube-prometheus-stack \
  -f helm/monitoring-values.yaml -n monitoring

4. 와일드카드 TLS 인증서 자동화

HAProxy 서버에서 certbot이 Let’s Encrypt 인증서를 주기적으로 갱신하고 NAS에 복사합니다. Kubernetes CronJob이 이후 NAS NFS를 마운트하여 SHA256 비교 후 변경 시에만 Secret을 갱신합니다.

certbot (HAProxy)
  └─→ NAS (NFS)
        └─→ K8s CronJob (SHA256 비교)
              └─→ traefik/wildcard-sierracloud-dev Secret
                    └─→ Traefik TLSStore default → 모든 서비스 자동 적용

TLSStore를 사용하면 각 네임스페이스의 Ingress 리소스에 secretName을 지정할 필요 없이 모든 HTTPS 라우트에 와일드카드 인증서가 자동으로 적용됩니다.

# manifests/traefik/tls-store.yaml
apiVersion: traefik.io/v1alpha1
kind: TLSStore
metadata:
  name: default
  namespace: traefik
spec:
  defaultCertificate:
    secretName: wildcard-sierracloud-dev

CronJob은 매주 갱신 주기에 맞춰 실행되며 SHA256 비교를 통해 인증서가 변경된 경우에만 Secret을 업데이트합니다.

# manifests/traefik/tls-secret-sync.yaml (핵심 부분)
schedule: "30 6 * * 1"    # 매주 월요일
timeZone: "Asia/Seoul"
concurrencyPolicy: Forbid

volumes:
  - name: certs
    nfs:
      server: 192.168.x.x
      path: /data/cert
      readOnly: true
# CronJob 컨테이너 스크립트 (요약)
CURRENT_SHA=$(kubectl get secret wildcard-sierracloud-dev -n traefik \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | sha256sum | cut -d' ' -f1)
NEW_SHA=$(sha256sum < /certs/sierracloud.dev/fullchain.pem | cut -d' ' -f1)

if [ "$CURRENT_SHA" != "$NEW_SHA" ]; then
  kubectl create secret tls wildcard-sierracloud-dev \
    --cert=/certs/sierracloud.dev/fullchain.pem \
    --key=/certs/sierracloud.dev/privkey.pem \
    -n traefik --dry-run=client -o yaml | kubectl apply -f -
fi

트러블슈팅

① bitnami/kubectl 이미지 태그 없음

증상: CronJob 컨테이너 이미지 bitnami/kubectl:1.33이 ImagePullBackOff 발생.

원인: 2025년 12월 이후 bitnami/kubectl Docker Hub 레포지토리에서 버전 태그가 삭제됨 (GitHub issue #88999). latest 태그만 존재.

해결: alpine/k8s:1.33.10으로 교체. Alpine 기반으로 kubectl + 기본 유틸리티(sha256sum, base64 등) 포함, K8s 1.33.x 버전과 동일 minor version으로 완전 호환.

image: alpine/k8s:1.33.10    # bitnami/kubectl:1.33 → 교체

② 도메인 변경 후 Traefik 503 (약 5초)

증상: Helm upgrade 직후 prometheus.sierracloud.dev, grafana.sierracloud.dev 접속 시 503 반환.

원인: Traefik이 새로운 Ingress 라우트를 동기화하는 데 약 5초 소요.

해결: 별도 조치 없이 자연 해소. Traefik의 정상적인 라우트 갱신 동작.

검증

# Traefik LoadBalancer IP 확인
kubectl get svc -n traefik

# TLSStore 적용 확인
kubectl get tlsstore -n traefik

# Secret 인증서 유효기간 확인
kubectl get secret wildcard-sierracloud-dev -n traefik \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject -dates

# HTTPS 접속 확인
curl -skI https://grafana.sierracloud.dev | head -3
curl -skI https://prometheus.sierracloud.dev | head -3
# 출력 예시
HTTP/2 302       ← Grafana 로그인 리다이렉트 (정상)
HTTP/2 405       ← Prometheus (정상)

subject=CN = *.sierracloud.dev
notAfter=Sep 17 13:39:09 2026 GMT

결론

ingress-nginx EOL 전환을 계기로 Traefik v3.7.5를 도입하고, Let’s Encrypt 와일드카드 인증서의 자동 갱신 파이프라인을 구성하였습니다. Secret을 traefik 네임스페이스에 단일 관리하고 TLSStore default로 노출함으로써 향후 신규 서비스 추가 시 Ingress에 secretName을 별도 지정할 필요 없이 자동으로 와일드카드 인증서가 적용됩니다.

기존 ingress-nginx와의 전환 과정에서 MetalLB LoadBalancer IP를 그대로 유지했기 때문에 HAProxy 백엔드 설정 변경 없이 완전한 무중단 전환이 가능하였습니다. Traefik v3 계열의 Kubernetes Gateway API 지원, 향상된 observability, 그리고 CRD 기반의 미들웨어 체인 설정은 장기적인 운영에서도 ingress-nginx 대비 유리한 점이 많습니다.

참고

관련 포스트:

참고 문서: Traefik TLSStore Default Certificate (공식 문서) · Traefik Helm Chart 설치 가이드

 IT, K8S

Kubernetes cluster upgrade

댓글 남기기

개요

Kubernetes는 보안 패치 및 기능 개선을 위해 Kubernetes cluster upgrade를 주기적으로 수행해야 합니다. Kubernetes 업그레이드는 한 번에 한 개의 minor 버전씩만 가능하며, 예를 들어 1.32에서 1.35로 올리려면 1.32 → 1.33 → 1.34 → 1.35 순서로 각 버전을 단계별로 진행해야 합니다.

업그레이드 도구는 kubeadm이며, 진행 순서는 반드시 Control Plane 노드 먼저, 이후 Worker 노드 순차적으로 진행해야 합니다. Control Plane 노드가 여러 개인 HA 구성에서는 각 master 노드를 한 번에 하나씩 업그레이드합니다.

노드 업그레이드 전후에는 kubectl drain과 kubectl uncordon으로 해당 노드의 스케줄링을 일시 중지하고, 업그레이드 완료 후 다시 활성화합니다. Ubuntu 환경에서는 apt-mark hold로 kubeadm, kubelet, kubectl 패키지를 고정해 두었다가 업그레이드 시점에만 해제하는 방식을 권장합니다.

이 가이드는 Ubuntu 24.04 기반의 단일 Control Plane 클러스터(master 1대, worker 2대)에서 1.32.x → 1.33.7 업그레이드를 진행한 절차를 기록합니다. kubeadm upgrade plan으로 가능한 업그레이드 대상 버전을 확인하고, kubeadm upgrade apply로 Control Plane을 업그레이드한 뒤, 각 Worker 노드에서 kubelet과 kubectl을 교체합니다.

Kubernetes는 기능적/보안적인 이유로 주기적인 Cluster upgrade가 필연적입니다.

현재 버전인 1.32.*에서 1.33.* 버전으로 Upgrade가 목표이며 만일 1.35.* 같은 몇 단계 상위 버전 Upgrade를 계획하더라도 1.33, 1.34, 1.35 버전을 단계별로 순차 진행해야 되는 것은 동일합니다.

Master node가 이중화된 환경은 추가 과정이 있지만 과정 숙달을 목표로 메뉴얼을 기반하여 기본적인 순서로 진행해보겠습니다.

Master Node (Control Plane) 업그레이드

Control Plane 노드에서 kubeadm을 먼저 업그레이드하고, kubeadm upgrade plan으로 가능한 버전을 확인합니다. 이후 kubeadm upgrade apply로 API 서버, etcd, CoreDNS 등 핵심 컴포넌트를 일괄 업그레이드하고, kubelet/kubectl도 새 버전으로 교체합니다.

# 사용 명령어
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

sudo apt update
sudo apt-cache madison kubeadm

sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.33.7-*' && \
sudo apt-mark hold kubeadm

kubeadm version

sudo kubeadm upgrade plan

sudo kubeadm upgrade apply v1.33.7

kubectl drain  --ignore-daemonsets

sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.7-*' kubectl='1.33.7-*' && \
sudo apt-mark hold kubelet kubectl

sudo systemctl daemon-reload
sudo systemctl restart kubelet

kubectl uncordon 
# Log
test@:~$ echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /
test@:~$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
File '/etc/apt/keyrings/kubernetes-apt-keyring.gpg' exists. Overwrite? (y/N) y
test@:~$ sudo apt update
Hit:1 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:4 https://download.docker.com/linux/ubuntu noble InRelease
Hit:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
230 packages can be upgraded. Run 'apt list --upgradable' to see them.
test@:~$ sudo apt-cache madison kubeadm
   kubeadm | 1.33.7-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.6-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.5-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.4-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.3-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
test@:~$ sudo apt-mark unhold kubeadm && \
> sudo apt-get update && sudo apt-get install -y kubeadm='1.33.7-*' && \
> sudo apt-mark hold kubeadm
kubeadm was already not on hold.
Hit:1 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:4 https://download.docker.com/linux/ubuntu noble InRelease
Hit:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubeadm'
The following packages will be upgraded:
  kubeadm
1 upgraded, 0 newly installed, 0 to remove and 229 not upgraded.
Need to get 12.7 MB of archives.
After this operation, 3,600 kB of additional disk space will be used.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubeadm 1.33.7-1.1 [12.7 MB]
Fetched 12.7 MB in 0s (33.8 MB/s)
(Reading database ... 196041 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.33.7-1.1_amd64.deb ...
Unpacking kubeadm (1.33.7-1.1) over (1.32.7-1.1) ...
Setting up kubeadm (1.33.7-1.1) ...
kubeadm set on hold.
test@:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"33", EmulationMajor:"", EmulationMinor:"", MinCompatibilityMajor:"", MinCompatibilityMinor:"", GitVersion:"v1.33.7", GitCommit:"a7245cdf3f69e11356c7e8f92b3e78ca4ee4e757", GitTreeState:"clean", BuildDate:"2025-12-09T14:41:01Z", GoVersion:"go1.24.11", Compiler:"gc", Platform:"linux/amd64"}
test@:~$ sudo kubeadm upgrade plan
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade/config] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: 1.32.7
[upgrade/versions] kubeadm version: v1.33.7
I0113 17:54:44.730671 2116008 version.go:261] remote version is much newer: v1.35.0; falling back to: stable-1.33
[upgrade/versions] Target version: v1.33.7
[upgrade/versions] Latest version in the v1.32 series: v1.32.11

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE                CURRENT   TARGET
kubelet        v1.32.7   v1.32.11
kubelet        v1.32.7   v1.32.11
kubelet        v1.32.7   v1.32.11

Upgrade to the latest version in the v1.32 series:

COMPONENT                 NODE                CURRENT    TARGET
kube-apiserver               v1.32.7    v1.32.11
kube-controller-manager      v1.32.7    v1.32.11
kube-scheduler               v1.32.7    v1.32.11
kube-proxy                                    1.32.7     v1.32.11
CoreDNS                                       v1.11.3    v1.12.0
etcd                         3.5.16-0   3.5.24-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.32.11

_____________________________________________________________________

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE                CURRENT   TARGET
kubelet        v1.32.7   v1.33.7
kubelet        v1.32.7   v1.33.7
kubelet        v1.32.7   v1.33.7

Upgrade to the latest stable version:

COMPONENT                 NODE                CURRENT    TARGET
kube-apiserver               v1.32.7    v1.33.7
kube-controller-manager      v1.32.7    v1.33.7
kube-scheduler               v1.32.7    v1.33.7
kube-proxy                                    1.32.7     v1.33.7
CoreDNS                                       v1.11.3    v1.12.0
etcd                         3.5.16-0   3.5.24-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.33.7

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________

test@:~$ sudo kubeadm upgrade apply v1.33.7
[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it.
[upgrade/preflight] Running preflight checks
[upgrade] Running cluster health checks
[upgrade/preflight] You have chosen to upgrade the cluster version to "v1.33.7"
[upgrade/versions] Cluster version: v1.32.7
[upgrade/versions] kubeadm version: v1.33.7
[upgrade] Are you sure you want to proceed? [y/N]: y
[upgrade/preflight] Pulling images required for setting up a Kubernetes cluster
[upgrade/preflight] This might take a minute or two, depending on the speed of your internet connection
[upgrade/preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[upgrade/control-plane] Upgrading your static Pod-hosted control plane to version "v1.33.7" (timeout: 5m0s)...
[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests3418553132"
[upgrade/staticpods] Preparing for "etcd" upgrade
[upgrade/staticpods] Renewing etcd-server certificate
[upgrade/staticpods] Renewing etcd-peer certificate
[upgrade/staticpods] Renewing etcd-healthcheck-client certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/etcd.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-01-13-17-57-46/etcd.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 1 Pods for label selector component=etcd
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/etcd] Waiting for etcd to become available
[upgrade/staticpods] Preparing for "kube-apiserver" upgrade
[upgrade/staticpods] Renewing apiserver certificate
[upgrade/staticpods] Renewing apiserver-kubelet-client certificate
[upgrade/staticpods] Renewing front-proxy-client certificate
[upgrade/staticpods] Renewing apiserver-etcd-client certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-apiserver.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-01-13-17-57-46/kube-apiserver.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 1 Pods for label selector component=kube-apiserver
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-controller-manager" upgrade
[upgrade/staticpods] Renewing controller-manager.conf certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-controller-manager.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-01-13-17-57-46/kube-controller-manager.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 1 Pods for label selector component=kube-controller-manager
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Preparing for "kube-scheduler" upgrade
[upgrade/staticpods] Renewing scheduler.conf certificate
[upgrade/staticpods] Moving new manifest to "/etc/kubernetes/manifests/kube-scheduler.yaml" and backing up old manifest to "/etc/kubernetes/tmp/kubeadm-backup-manifests-2026-01-13-17-57-46/kube-scheduler.yaml"
[upgrade/staticpods] Waiting for the kubelet to restart the component
[upgrade/staticpods] This can take up to 5m0s
[apiclient] Found 1 Pods for label selector component=kube-scheduler
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade/control-plane] The control plane instance for this node was successfully upgraded!
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upgrade/kubeconfig] The kubeconfig files for this node were successfully upgraded!
W0113 18:01:48.219276 2117131 postupgrade.go:117] Using temporary directory /etc/kubernetes/tmp/kubeadm-kubelet-config1781268223 for kubelet config. To override it set the environment variable KUBEADM_UPGRADE_DRYRUN_DIR
[upgrade] Backing up kubelet config file to /etc/kubernetes/tmp/kubeadm-kubelet-config1781268223/config.yaml
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded!
[upgrade/bootstrap-token] Configuring bootstrap token and cluster-info RBAC rules
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

[upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.33.7".

[upgrade] Now please proceed with upgrading the rest of the nodes by following the right order.
test@:~$ kubectl drain  --ignore-daemonsets
node/ cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/calico-node-km28t, kube-system/kube-proxy-ghl52
evicting pod kube-system/calico-kube-controllers-6d5bc68bd-vv654
pod/calico-kube-controllers-6d5bc68bd-vv654 evicted
node/ drained
test@:~$ kubectl get node -o wide
NAME                STATUS                     ROLES           AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
   Ready,SchedulingDisabled   control-plane   166d   v1.32.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
   Ready                      <none>          166d   v1.32.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
   Ready                      <none>          166d   v1.32.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
test@:~$ sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.7-*' kubectl='1.33.7-*' && \
sudo apt-mark hold kubelet kubectl
Canceled hold on kubelet.
Canceled hold on kubectl.
Hit:1 https://download.docker.com/linux/ubuntu noble InRelease
Hit:2 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:4 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Hit:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubelet'
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubectl'
The following package was automatically installed and is no longer required:
  conntrack
Use 'sudo apt autoremove' to remove it.
The following packages will be upgraded:
  kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 227 not upgraded.
Need to get 27.6 MB of archives.
After this operation, 7,115 kB of additional disk space will be used.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubectl 1.33.7-1.1 [11.7 MB]
Get:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubelet 1.33.7-1.1 [15.9 MB]
Fetched 27.6 MB in 1s (51.6 MB/s)
(Reading database ... 196041 files and directories currently installed.)
Preparing to unpack .../kubectl_1.33.7-1.1_amd64.deb ...
Unpacking kubectl (1.33.7-1.1) over (1.32.7-1.1) ...
Preparing to unpack .../kubelet_1.33.7-1.1_amd64.deb ...
Unpacking kubelet (1.33.7-1.1) over (1.32.7-1.1) ...
Setting up kubectl (1.33.7-1.1) ...
Setting up kubelet (1.33.7-1.1) ...
kubelet set on hold.
kubectl set on hold.
test@:~$ sudo systemctl daemon-reload
test@:~$ sudo systemctl restart kubelet
test@:~$ kubectl uncordon 
node/ uncordoned
test@:~$ kubectl get node -o wide
NAME                STATUS   ROLES           AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
   Ready    control-plane   166d   v1.33.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
   Ready    <none>          166d   v1.32.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
   Ready    <none>          166d   v1.32.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3

Worker Node 업그레이드 (#1)

Worker 노드 업그레이드는 Control Plane에서 kubectl drain으로 해당 Worker 노드를 비운 뒤 진행합니다. Worker 노드에서 kubeadm/kubelet/kubectl을 새 버전으로 교체하고, kubelet 재시작 후 Control Plane에서 kubectl uncordon으로 복귀시킵니다. 노드가 여러 개인 경우 동일 절차를 반복합니다.

# 사용 명령어
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

sudo apt update
sudo apt-cache madison kubeadm

sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.33.7-*' && \
sudo apt-mark hold kubeadm

kubeadm version

kubectl drain  \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60 \
  --timeout=15m

sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.7-*' kubectl='1.33.7-*' && \
sudo apt-mark hold kubelet kubectl

sudo systemctl daemon-reload
sudo systemctl restart kubelet

kubectl uncordon 
# Log
test@:~$ echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /
test@:~$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
File '/etc/apt/keyrings/kubernetes-apt-keyring.gpg' exists. Overwrite? (y/N) y
test@:~$ sudo apt update
Hit:1 https://download.docker.com/linux/ubuntu noble InRelease
Get:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease [1,230 B]
Get:3 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  Packages [11.3 kB]
Hit:4 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Get:5 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
Get:6 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
Get:7 http://security.ubuntu.com/ubuntu noble-security InRelease [126 kB]
Get:8 http://kr.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [1,693 kB]
Get:9 http://kr.archive.ubuntu.com/ubuntu noble-updates/main Translation-en [313 kB]
Get:10 http://kr.archive.ubuntu.com/ubuntu noble-updates/main amd64 Components [175 kB]
Get:11 http://kr.archive.ubuntu.com/ubuntu noble-updates/main amd64 c-n-f Metadata [15.9 kB]
Get:12 http://kr.archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Packages [2,426 kB]
Get:13 http://kr.archive.ubuntu.com/ubuntu noble-updates/restricted Translation-en [554 kB]
Get:14 http://kr.archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Components [212 B]
Get:15 http://kr.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [1,510 kB]
Get:16 http://kr.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Components [377 kB]
Get:17 http://kr.archive.ubuntu.com/ubuntu noble-updates/universe amd64 c-n-f Metadata [31.4 kB]
Get:18 http://kr.archive.ubuntu.com/ubuntu noble-updates/multiverse amd64 Components [940 B]
Get:19 http://kr.archive.ubuntu.com/ubuntu noble-backports/main amd64 Components [7,300 B]
Get:20 http://kr.archive.ubuntu.com/ubuntu noble-backports/restricted amd64 Components [216 B]
Get:21 http://kr.archive.ubuntu.com/ubuntu noble-backports/universe amd64 Components [10.5 kB]
Get:22 http://kr.archive.ubuntu.com/ubuntu noble-backports/multiverse amd64 Components [212 B]
Get:23 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages [1,404 kB]
Get:24 http://security.ubuntu.com/ubuntu noble-security/main Translation-en [228 kB]
Get:25 http://security.ubuntu.com/ubuntu noble-security/main amd64 Components [21.5 kB]
Get:26 http://security.ubuntu.com/ubuntu noble-security/restricted amd64 Packages [2,302 kB]
Get:27 http://security.ubuntu.com/ubuntu noble-security/restricted Translation-en [527 kB]
Get:28 http://security.ubuntu.com/ubuntu noble-security/restricted amd64 Components [208 B]
Get:29 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Components [71.4 kB]
Get:30 http://security.ubuntu.com/ubuntu noble-security/multiverse amd64 Components [208 B]
Fetched 12.1 MB in 3s (3,513 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
241 packages can be upgraded. Run 'apt list --upgradable' to see them.
test@:~$ sudo apt-cache madison kubeadm
   kubeadm | 1.33.7-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.6-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.5-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.4-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.3-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
test@:~$ sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.33.7-*' && \
sudo apt-mark hold kubeadm
Canceled hold on kubeadm.
Hit:1 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:4 https://download.docker.com/linux/ubuntu noble InRelease
Hit:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubeadm'
The following packages will be upgraded:
  kubeadm
1 upgraded, 0 newly installed, 0 to remove and 240 not upgraded.
Need to get 12.7 MB of archives.
After this operation, 3,600 kB of additional disk space will be used.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubeadm 1.33.7-1.1 [12.7 MB]
Fetched 12.7 MB in 0s (28.4 MB/s)
(Reading database ... 196036 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.33.7-1.1_amd64.deb ...
Unpacking kubeadm (1.33.7-1.1) over (1.32.7-1.1) ...
Setting up kubeadm (1.33.7-1.1) ...
kubeadm set on hold.
test@:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"33", EmulationMajor:"", EmulationMinor:"", MinCompatibilityMajor:"", MinCompatibilityMinor:"", GitVersion:"v1.33.7", GitCommit:"a7245cdf3f69e11356c7e8f92b3e78ca4ee4e757", GitTreeState:"clean", BuildDate:"2025-12-09T14:41:01Z", GoVersion:"go1.24.11", Compiler:"gc", Platform:"linux/amd64"}

test@:~$ kubectl drain  \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60 \
  --timeout=15m
node/ cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/calico-node-fdff8, kube-system/kube-proxy-2rz9d
evicting pod kube-system/coredns-674b8bbfcf-7jxkk
pod/coredns-674b8bbfcf-7jxkk evicted
node/ drained
test@:~$ kubectl get node -o wide
NAME                STATUS                     ROLES           AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
   Ready                      control-plane   166d   v1.33.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
   Ready,SchedulingDisabled   <none>          166d   v1.32.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
   Ready                      <none>          166d   v1.32.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3

test@:~$ sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.7-*' kubectl='1.33.7-*' && \
sudo apt-mark hold kubelet kubectl
Canceled hold on kubelet.
Canceled hold on kubectl.
Hit:1 https://download.docker.com/linux/ubuntu noble InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:4 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:5 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubelet'
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubectl'
The following package was automatically installed and is no longer required:
  conntrack
Use 'sudo apt autoremove' to remove it.
The following packages will be upgraded:
  kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 238 not upgraded.
Need to get 27.6 MB of archives.
After this operation, 7,115 kB of additional disk space will be used.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubectl 1.33.7-1.1 [11.7 MB]
Get:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubelet 1.33.7-1.1 [15.9 MB]
Fetched 27.6 MB in 1s (49.7 MB/s)
(Reading database ... 196036 files and directories currently installed.)
Preparing to unpack .../kubectl_1.33.7-1.1_amd64.deb ...
Unpacking kubectl (1.33.7-1.1) over (1.32.7-1.1) ...
Preparing to unpack .../kubelet_1.33.7-1.1_amd64.deb ...
Unpacking kubelet (1.33.7-1.1) over (1.32.7-1.1) ...
Setting up kubectl (1.33.7-1.1) ...
Setting up kubelet (1.33.7-1.1) ...
kubelet set on hold.
kubectl set on hold.
test@:~$ sudo systemctl daemon-reload
test@:~$ sudo systemctl restart kubelet
test@:~$ dpkg -l | grep kube
hi  kubeadm                                       1.33.7-1.1                               amd64        Command-line utility for administering a Kubernetes cluster
hi  kubectl                                       1.33.7-1.1                               amd64        Command-line utility for interacting with a Kubernetes cluster
hi  kubelet                                       1.33.7-1.1                               amd64        Node agent for Kubernetes clusters
ii  kubernetes-cni                                1.6.0-1.1                                amd64        Binaries required to provision kubernetes container networking

test@:~$ kubectl uncordon 
node/ uncordoned
test@:~$ kubectl get node -o wide
NAME                STATUS   ROLES           AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
   Ready    control-plane   166d   v1.33.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
   Ready    <none>          166d   v1.33.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
   Ready    <none>          166d   v1.32.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3

Worker Node 업그레이드 (#2)

Worker 노드 #1과 동일한 절차를 Worker 노드 #2에 반복합니다. Control Plane에서 kubectl drain <worker-node-02>로 노드를 비우고, Worker 노드에서 kubeadm/kubelet/kubectl을 업그레이드한 뒤 kubectl uncordon으로 복귀시킵니다. 모든 Worker 노드 업그레이드 완료 후 kubectl get nodes -o wide로 전체 노드 버전이 동일한지 확인합니다.

# 사용 명령어
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

sudo apt update
sudo apt-cache madison kubeadm

sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.33.7-*' && \
sudo apt-mark hold kubeadm

kubeadm version

kubectl drain  \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60 \
  --timeout=15m

sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.7-*' kubectl='1.33.7-*' && \
sudo apt-mark hold kubelet kubectl

sudo systemctl daemon-reload
sudo systemctl restart kubelet

kubectl uncordon 
# Log
test@:~$ echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /" | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /
test@:~$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
File '/etc/apt/keyrings/kubernetes-apt-keyring.gpg' exists. Overwrite? (y/N) y
test@:~$ sudo apt update
Hit:1 https://download.docker.com/linux/ubuntu noble InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Get:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease [1,230 B]
Get:4 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
Get:5 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
Get:6 http://security.ubuntu.com/ubuntu noble-security InRelease [126 kB]
Get:7 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  Packages [11.3 kB]
Get:8 http://kr.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [1,693 kB]
Get:9 http://kr.archive.ubuntu.com/ubuntu noble-updates/main Translation-en [313 kB]
Get:10 http://kr.archive.ubuntu.com/ubuntu noble-updates/main amd64 Components [175 kB]
Get:11 http://kr.archive.ubuntu.com/ubuntu noble-updates/main amd64 c-n-f Metadata [15.9 kB]
Get:12 http://kr.archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Packages [2,426 kB]
Get:13 http://kr.archive.ubuntu.com/ubuntu noble-updates/restricted Translation-en [554 kB]
Get:14 http://kr.archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Components [212 B]
Get:15 http://kr.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [1,510 kB]
Get:16 http://kr.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Components [377 kB]
Get:17 http://kr.archive.ubuntu.com/ubuntu noble-updates/universe amd64 c-n-f Metadata [31.4 kB]
Get:18 http://kr.archive.ubuntu.com/ubuntu noble-updates/multiverse amd64 Components [940 B]
Get:19 http://kr.archive.ubuntu.com/ubuntu noble-backports/main amd64 Components [7,300 B]
Get:20 http://kr.archive.ubuntu.com/ubuntu noble-backports/restricted amd64 Components [216 B]
Get:21 http://kr.archive.ubuntu.com/ubuntu noble-backports/universe amd64 Components [10.5 kB]
Get:22 http://kr.archive.ubuntu.com/ubuntu noble-backports/multiverse amd64 Components [212 B]
Get:23 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages [1,404 kB]
Get:24 http://security.ubuntu.com/ubuntu noble-security/main Translation-en [228 kB]
Get:25 http://security.ubuntu.com/ubuntu noble-security/main amd64 Components [21.5 kB]
Get:26 http://security.ubuntu.com/ubuntu noble-security/restricted amd64 Packages [2,302 kB]
Get:27 http://security.ubuntu.com/ubuntu noble-security/restricted Translation-en [527 kB]
Get:28 http://security.ubuntu.com/ubuntu noble-security/restricted amd64 Components [208 B]
Get:29 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Components [71.4 kB]
Get:30 http://security.ubuntu.com/ubuntu noble-security/universe amd64 c-n-f Metadata [19.7 kB]
Get:31 http://security.ubuntu.com/ubuntu noble-security/multiverse amd64 Components [208 B]
Fetched 12.1 MB in 4s (3,410 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
229 packages can be upgraded. Run 'apt list --upgradable' to see them.
test@:~$ sudo apt-cache madison kubeadm
   kubeadm | 1.33.7-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.6-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.5-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.4-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.3-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
test@:~$ sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.33.7-*' && \
sudo apt-mark hold kubeadm
Canceled hold on kubeadm.
Hit:1 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:4 https://download.docker.com/linux/ubuntu noble InRelease
Hit:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubeadm'
The following packages will be upgraded:
  kubeadm
1 upgraded, 0 newly installed, 0 to remove and 228 not upgraded.
Need to get 12.7 MB of archives.
After this operation, 3,600 kB of additional disk space will be used.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubeadm 1.33.7-1.1 [12.7 MB]
Fetched 12.7 MB in 1s (23.5 MB/s)
(Reading database ... 196021 files and directories currently installed.)
Preparing to unpack .../kubeadm_1.33.7-1.1_amd64.deb ...
Unpacking kubeadm (1.33.7-1.1) over (1.32.7-1.1) ...
Setting up kubeadm (1.33.7-1.1) ...
kubeadm set on hold.
test@:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"33", EmulationMajor:"", EmulationMinor:"", MinCompatibilityMajor:"", MinCompatibilityMinor:"", GitVersion:"v1.33.7", GitCommit:"a7245cdf3f69e11356c7e8f92b3e78ca4ee4e757", GitTreeState:"clean", BuildDate:"2025-12-09T14:41:01Z", GoVersion:"go1.24.11", Compiler:"gc", Platform:"linux/amd64"}

test@:~$ kubectl drain  \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60 \
  --timeout=15m
node/ cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/calico-node-drhtt, kube-system/kube-proxy-hpkd8
evicting pod kube-system/coredns-674b8bbfcf-f6txf
evicting pod kube-system/calico-kube-controllers-6d5bc68bd-7mppz
pod/calico-kube-controllers-6d5bc68bd-7mppz evicted
pod/coredns-674b8bbfcf-f6txf evicted
node/ drained
test@:~$ kubectl get node -o wide
NAME                STATUS                     ROLES           AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
   Ready                      control-plane   166d   v1.33.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
   Ready                      <none>          166d   v1.33.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
   Ready,SchedulingDisabled   <none>          166d   v1.32.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3

test@:~$ sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.7-*' kubectl='1.33.7-*' && \
sudo apt-mark hold kubelet kubectl
Canceled hold on kubelet.
Canceled hold on kubectl.
Hit:1 http://kr.archive.ubuntu.com/ubuntu noble InRelease
Hit:2 http://kr.archive.ubuntu.com/ubuntu noble-updates InRelease
Hit:3 http://kr.archive.ubuntu.com/ubuntu noble-backports InRelease
Hit:4 https://download.docker.com/linux/ubuntu noble InRelease
Hit:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  InRelease
Hit:6 http://security.ubuntu.com/ubuntu noble-security InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubelet'
Selected version '1.33.7-1.1' (isv:kubernetes:core:stable:v1.33:pkgs.k8s.io [amd64]) for 'kubectl'
The following package was automatically installed and is no longer required:
  conntrack
Use 'sudo apt autoremove' to remove it.
The following packages will be upgraded:
  kubectl kubelet
2 upgraded, 0 newly installed, 0 to remove and 226 not upgraded.
Need to get 27.6 MB of archives.
After this operation, 7,115 kB of additional disk space will be used.
Get:1 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubectl 1.33.7-1.1 [11.7 MB]
Get:2 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.33/deb  kubelet 1.33.7-1.1 [15.9 MB]
Fetched 27.6 MB in 1s (47.2 MB/s)
(Reading database ... 196021 files and directories currently installed.)
Preparing to unpack .../kubectl_1.33.7-1.1_amd64.deb ...
Unpacking kubectl (1.33.7-1.1) over (1.32.7-1.1) ...
Preparing to unpack .../kubelet_1.33.7-1.1_amd64.deb ...
Unpacking kubelet (1.33.7-1.1) over (1.32.7-1.1) ...
Setting up kubectl (1.33.7-1.1) ...
Setting up kubelet (1.33.7-1.1) ...
kubelet set on hold.
kubectl set on hold.
test@:~$ sudo systemctl daemon-reload
test@:~$ sudo systemctl restart kubelet
test@:~$ dpkg -l | grep kube
hi  kubeadm                                       1.33.7-1.1                               amd64        Command-line utility for administering a Kubernetes cluster
hi  kubectl                                       1.33.7-1.1                               amd64        Command-line utility for interacting with a Kubernetes cluster
hi  kubelet                                       1.33.7-1.1                               amd64        Node agent for Kubernetes clusters
ii  kubernetes-cni                                1.6.0-1.1                                amd64        Binaries required to provision kubernetes container networking

test@:~$ kubectl uncordon 
node/ uncordoned
test@:~$ kubectl get node -o wide
NAME                STATUS   ROLES           AGE    VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
   Ready    control-plane   166d   v1.33.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
   Ready    <none>          166d   v1.33.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3
   Ready    <none>          166d   v1.33.7   x.x.x.x   <none>        Ubuntu 24.04.2 LTS   6.14.0-37-generic   docker://28.3.3

업그레이드 주의사항

Kubernetes 클러스터 업그레이드 시 주의해야 할 사항을 정리합니다.

  • 버전 건너뛰기 불가 — minor 버전은 반드시 한 단계씩 업그레이드해야 합니다. 예를 들어 1.31 → 1.33으로 직접 업그레이드할 수 없습니다.
  • etcd 백업 선행 — Control Plane 업그레이드 전 etcd 스냅샷을 백업해 두면 문제 발생 시 롤백이 가능합니다.
  • drain 전 PodDisruptionBudget 확인 — kubectl drain 시 PDB(PodDisruptionBudget)에 의해 eviction이 차단될 수 있습니다. --delete-emptydir-data와 --grace-period 옵션을 적절히 설정합니다.
  • CNI 플러그인 호환성 확인 — Calico, Flannel 등 CNI 플러그인이 업그레이드 대상 Kubernetes 버전과 호환되는지 릴리스 노트를 미리 확인합니다.
  • HA 클러스터 — master 노드가 여러 개인 경우 한 번에 하나씩 업그레이드하며, 각 master 업그레이드 후 etcd quorum이 유지되는지 확인합니다.

참고

관련 포스트:

참고 문서: kubeadm upgrade 공식 문서 (kubernetes.io) · Kubernetes 패키지 저장소 변경 가이드

K8S, Object

Ceph MDS Pod Anti-Affinity Troubleshoot

댓글 남기기

개요

Kubernetes 클러스터에서 노드 업그레이드나 drain 작업을 수행할 때, Ceph MDS Pod Anti-Affinity 설정이 없으면 MDS 파드 두 개가 동일한 워커 노드에 집중 배치되는 문제가 발생할 수 있습니다. 이 글에서는 node drain 후 MDS 파드가 한 노드에 몰린 상황을 확인하고, CephFilesystem 리소스에 podAntiAffinity 규칙을 적용하여 MDS 파드를 분산 배치하는 과정을 정리합니다. 이어서 함께 발생한 mgr crash 알람 처리 방법도 설명합니다.

문제 상황

워커 노드 1번에 대해 drain을 수행한 후 Ceph 상태를 확인하면 HEALTH_WARN이 발생하고, MDS 파드 2개가 모두 워커 노드 2번에서 기동 중임을 확인할 수 있습니다.

# ceph 상태 확인
test@test-master-01:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status
  cluster:
    id:     d874b4ea-8deb-4aa3-a3ac-e750180a6a5b
    health: HEALTH_WARN
            4 mgr modules have recently crashed

  services:
    mon: 3 daemons, quorum a,b,c (age 10h)
    mgr: b(active, since 5M), standbys: a
    mds: 1/1 daemons up, 1 hot standby
    osd: 3 osds: 3 up (since 10h), 3 in (since 18M)

# MDS pod 위치 확인 — 두 파드 모두 test-worker-02에 집중
test@test-master-01:~$ kubectl -n rook-ceph get pod -o wide | egrep 'mds'
rook-ceph-mds-myfs-a-77d484dc4-jddf9  2/2  Running  0  18s  172.16.x.x  test-worker-02  <none>  <none>
rook-ceph-mds-myfs-b-bd6ddc59b-l2b4t  2/2  Running  0  18s  172.16.x.x  test-worker-02  <none>  <none>

원인 분석

ceph fs status에서 확인하면 active MDS와 standby-replay MDS 모두 정상 동작 중이지만, 두 파드가 같은 노드에 배치되어 있어 해당 노드에 장애가 발생하면 CephFS 서비스 전체가 중단될 위험이 있습니다. Anti-Affinity 규칙이 설정되어 있지 않으면 Kubernetes 스케줄러가 가용 자원이 충분한 노드에 임의로 배치하기 때문에 이런 현상이 발생합니다.

# cephFS 상태 확인
test@test-master-01:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph fs status
myfs - 2 clients
====
RANK      STATE          MDS     ACTIVITY     DNS    INOS   DIRS   CAPS
 0        active        myfs-b  Reqs:  0 /s  35.9k  18.0k  4301      2
0-s   standby-replay   myfs-a  Evts:  0 /s  35.9k  18.0k  4301      0
MDS version: ceph version 18.2.2 reef (stable)

Ceph MDS Pod Anti-Affinity 적용

CephFilesystem 리소스에 podAntiAffinity를 설정하면 동일 레이블의 MDS 파드가 같은 노드에 배치되지 않도록 강제할 수 있습니다. requiredDuringSchedulingIgnoredDuringExecution을 사용하면 조건을 만족하지 못할 경우 파드가 아예 스케줄링되지 않으므로 강하게 분산을 보장합니다.

# CephFilesystem에 podAntiAffinity 패치 적용
test@test-master-01:~$ kubectl -n rook-ceph patch cephfilesystem myfs --type='merge' -p '
spec:
  metadataServer:
    placement:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values: ["rook-ceph-mds"]
            - key: rook_file_system
              operator: In
              values: ["myfs"]
          topologyKey: kubernetes.io/hostname
'
cephfilesystem.ceph.rook.io/myfs patched

패치 후 kubectl -n rook-ceph get cephfilesystem myfs -o yaml에서 spec.metadataServer.placement.podAntiAffinity 항목이 반영되었는지 확인합니다.

결과 확인

워커 노드 1번을 uncordon하고 워커 노드 2번을 drain하면 MDS 파드가 Ceph MDS Pod Anti-Affinity 규칙에 따라 서로 다른 노드(worker-01, worker-03)에 분산 배치됩니다.

# Anti-Affinity 적용 후 MDS 파드 분산 확인
test@test-master-01:~$ kubectl -n rook-ceph get pod -o wide | egrep 'mds'
rook-ceph-mds-myfs-a-58846844d6-nd5mk  2/2  Running  0  53s  172.16.x.x  test-worker-01  <none>  <none>
rook-ceph-mds-myfs-b-6b4d9476cb-q6b6p  2/2  Running  0  38s  172.16.x.x  test-worker-03  <none>  <none>

# cephFS 상태 정상 확인
test@test-master-01:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph fs status
myfs - 2 clients
====
RANK      STATE          MDS     ACTIVITY     DNS    INOS   DIRS   CAPS
 0        active        myfs-a  Reqs:  0 /s  35.9k  18.0k  4301      2
0-s   standby-replay   myfs-b  Evts:  0 /s  35.9k  18.0k  4301      0
MDS version: ceph version 18.2.2 reef (stable)

mgr crash 알람 처리

MDS 분산 이후에도 4 mgr modules have recently crashed 알람이 남아 있을 수 있습니다. 이는 MDS 이슈와 무관하게 mgr 파드가 재시작되며 발생한 crash 이력으로, ceph mgr stat에서 available: true이면 서비스는 정상입니다. ceph crash archive-all로 이력을 정리하면 알람이 해소됩니다.

# mgr 상태 정상 확인 (available: true)
test@test-master-01:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph mgr stat
{
    "epoch": 476,
    "available": true,
    "active_name": "b",
    "num_standby": 1
}

# crash 이력 목록 확인
test@test-master-01:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph crash ls
ID                                                                ENTITY  NEW
2025-12-26T09:01:17.354121Z_c76c6eaf-4bf7-4cf9-a9ec-f646fe857b76  mgr.b    *
2025-12-26T09:01:32.345473Z_4dfd271c-3d5b-4c89-88cf-13ba096f327b  mgr.b    *
2025-12-26T09:01:47.357321Z_0f938fb6-4c50-4b58-815d-5990fbe4bbb7  mgr.b    *
2025-12-26T09:02:02.329492Z_43d344a7-b71f-442e-a664-1852dda3a3f3  mgr.b    *

# crash 이력 아카이브 후 HEALTH_OK 확인
test@test-master-01:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph crash archive-all
test@test-master-01:~$ kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status
  cluster:
    health: HEALTH_OK
  services:
    mds: 1/1 daemons up, 1 hot standby
    osd: 3 osds: 3 up, 3 in

node drain / uncordon 작업 중 일시적으로 mon quorum 이탈이나 rebalancing이 발생할 수 있으나, 일정 시간 후 재확인하면 HEALTH_OK 상태로 복구됩니다.

참고

Ceph 관련 운영 내용은 아래 포스트도 참고하시기 바랍니다.

참고 문서: Rook CephFilesystem CRD 공식 문서 · Kubernetes Pod Anti-Affinity 공식 문서