기본 node maintenance
- 이번엔 우리의 pod 중 일부가 node 에서 퇴출되고 다른 곳으로 스케줄이 변경되도록 할 것이다.
- 이것은 기본적인 maintenance 또는 롤링 OS 업데이트 등으로 있을 수 있다.
먼저 worker1번 노드를 더이상 스케쥴 되지 않도록 한다.
- 이때, worker1번 노드에 있는 pod들을 다른 노드들로 퇴출된다.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
# 현재 master node 이미지 수 확인
ps0107@k8smaster1:~$ sudo docker ps | wc -l
22
# 현재 worker node 이미지 수 확인
ps0107@k8sworker1:~$ sudo docker ps | wc -l
28
# maintenance를 완료하기 위해 노드에서 컨테이너를 이동하고 새 컨테이너가 배치되는 것을 막아야 한다.
# - drain : 비우다
# - cordon : 저지선을 치다
ps0107@k8smaster1:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8smaster1 Ready master 44h v1.15.1
k8sworker1 Ready <none> 44h v1.15.1
# drain 명령을 통해 해당 노드의 object를 비워봅니다.
# daemonSet Pod 가 있으면 error가 있고 taint만 설정됩니다.
# taint 설정 되면 해당 노드에 스케쥴링하지 않음.
ps0107@k8smaster1:~$ kubectl drain k8sworker1
node/k8sworker1 cordoned
error: unable to drain node "k8sworker1", aborting command...
There are pending nodes to be drained:
k8sworker1
cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): kube-system/calico-node-zxkxk, kube-system/kube-proxy-tnmtr
cannot delete Pods with local storage (use --delete-local-data to override): sock-shop/carts-56c6fb966b-44xdb, sock-shop/carts-db-5678cc578f-nh5l4, sock-shop/orders-749cdc8c9-7rntw, sock-shop/orders-db-5cfc68c4cf-6v2hb, sock-shop/shipping-78794fdb4f-dl6wk, sock-shop/user-db-99685d75b-6dvzs
# taint 설정 되면 해당 노드에 스케쥴링하지 않음.
ps0107@k8smaster1:~$ kubectl describe node | grep -i taint
Taints: <none>
Taints: node.kubernetes.io/unschedulable:NoSchedule
# 이번엔 daemonSet 무시하고 drain 해봅니다.
# 그런데 몽고 디비가 로컬 스토리지를 사용중이라 또 에러 발생
ps0107@k8smaster1:~$ kubectl drain k8sworker1 --ignore-daemonsets
node/k8sworker1 already cordoned
error: unable to drain node "k8sworker1", aborting command...
There are pending nodes to be drained:
k8sworker1
error: cannot delete Pods with local storage (use --delete-local-data to override): sock-shop/carts-56c6fb966b-44xdb, sock-shop/carts-db-5678cc578f-nh5l4, sock-shop/orders-749cdc8c9-7rntw, sock-shop/orders-db-5cfc68c4cf-6v2hb, sock-shop/shipping-78794fdb4f-dl6wk, sock-shop/user-db-99685d75b-6dvzs
# 이번엔 기존 연결된 몽고 디비 데이터 까지 삭제후 진행 하도록 옵션을 주고 drain 실행합니다.
# 오... 정상적으로 처리 되었습니다.
ps0107@k8smaster1:~$ kubectl drain k8sworker1 --ignore-daemonsets --delete-local-data
node/k8sworker1 already cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-zxkxk, kube-system/kube-proxy-tnmtr
evicting pod "user-db-99685d75b-6dvzs"
evicting pod "catalogue-db-6ddc796b66-8fws2"
evicting pod "carts-56c6fb966b-44xdb"
evicting pod "carts-db-5678cc578f-nh5l4"
evicting pod "catalogue-644549d46f-g8d4n"
evicting pod "orders-db-5cfc68c4cf-6v2hb"
evicting pod "front-end-5594987df6-252k8"
evicting pod "payment-54f55b96b9-hcc77"
evicting pod "orders-749cdc8c9-7rntw"
evicting pod "rabbitmq-bdfd84d55-87nrj"
evicting pod "shipping-78794fdb4f-dl6wk"
pod/catalogue-644549d46f-g8d4n evicted
pod/payment-54f55b96b9-hcc77 evicted
pod/user-db-99685d75b-6dvzs evicted
pod/orders-db-5cfc68c4cf-6v2hb evicted
pod/catalogue-db-6ddc796b66-8fws2 evicted
pod/rabbitmq-bdfd84d55-87nrj evicted
pod/carts-db-5678cc578f-nh5l4 evicted
pod/carts-56c6fb966b-44xdb evicted
pod/shipping-78794fdb4f-dl6wk evicted
pod/front-end-5594987df6-252k8 evicted
pod/orders-749cdc8c9-7rntw evicted
node/k8sworker1 evicted
# image 수 확인 한 결과 master 노드로 다 옮겨져 갔습니다.
ps0107@k8smaster1:~$ sudo docker ps | wc -l
42
ps0107@k8sworker1:~$ sudo docker ps | wc -l
6
# node 상태를 확인 해보면 SchedulingDisabled 상태로 변경이 되었습니다.
ps0107@k8smaster1:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8smaster1 Ready master 44h v1.15.1
k8sworker1 Ready,SchedulingDisabled <none> 44h v1.15.1
worker1번 노드가 drain 처리 되었고, pod들이 다른 노드로 옮겨져 갔으므로 worker 1번 노드의 작업을 해줍니다. (OS 업데이트, maintenance 등 처리)
다시 uncordon 처리하여 worker1번 노드를 사용하게 끔 설정
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
# 다시 worker 노드로 스케쥴 될수 있도록 uncordon 명령으로 처리 해줍니다.
ps0107@k8smaster1:~$ kubectl uncordon k8sworker1
node/k8sworker1 uncordoned
ps0107@k8smaster1:~$ kubectl describe nodes | grep -i taint
Taints: <none>
Taints: <none>
# 현재 worker node의 컨테이너 갯수를 봅니다. 역시나 아직 6개 입니다.
ps0107@k8sworker1:~$ sudo docker ps | wc -l
6
# node 정보를 보면 Ready 상태로 변경되었습니다.
ps0107@k8smaster1:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8smaster1 Ready master 44h v1.15.1
k8sworker1 Ready <none> 44h v1.15.1
# 현재 pod들도 전부 다른 노드에 running 중입니다.
ps0107@k8smaster1:~$ kubectl -n sock-shop get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
carts-56c6fb966b-tghrs 1/1 Running 0 4m39s 192.168.0.18 k8smaster1 <none> <none>
carts-db-5678cc578f-ngvxj 1/1 Running 0 4m39s 192.168.0.20 k8smaster1 <none> <none>
catalogue-644549d46f-p6crp 1/1 Running 0 4m39s 192.168.0.21 k8smaster1 <none> <none>
catalogue-db-6ddc796b66-pqwzf 1/1 Running 0 4m39s 192.168.0.24 k8smaster1 <none> <none>
front-end-5594987df6-64rlt 1/1 Running 0 4m39s 192.168.0.16 k8smaster1 <none> <none>
orders-749cdc8c9-js5v8 1/1 Running 0 4m39s 192.168.0.19 k8smaster1 <none> <none>
orders-db-5cfc68c4cf-bnqmg 1/1 Running 0 4m39s 192.168.0.17 k8smaster1 <none> <none>
payment-54f55b96b9-98jcw 1/1 Running 0 4m39s 192.168.0.25 k8smaster1 <none> <none>
queue-master-6fff667867-qffqg 1/1 Running 0 8m50s 192.168.0.14 k8smaster1 <none> <none>
rabbitmq-bdfd84d55-fhltf 1/1 Running 0 4m39s 192.168.0.22 k8smaster1 <none> <none>
shipping-78794fdb4f-g2wgm 1/1 Running 0 4m39s 192.168.0.23 k8smaster1 <none> <none>
user-77cff48476-vcwqt 1/1 Running 0 8m50s 192.168.0.15 k8smaster1 <none> <none>
user-db-99685d75b-6k5wp 1/1 Running 0 4m39s 192.168.0.26 k8smaster1 <none> <none>
# 테스트를 위해 몇개의 pod를 다시 띄워보기 위해 삭제를 해봅니다.
ps0107@k8smaster1:~$ kubectl -n sock-shop delete pod carts-56c6fb966b-tghrs orders-749cdc8c9-js5v8 front-end-5594987df6-64rlt user-77cff48476-vcwqt
pod "carts-56c6fb966b-tghrs" deleted
pod "orders-749cdc8c9-js5v8" deleted
pod "front-end-5594987df6-64rlt" deleted
pod "user-77cff48476-vcwqt" deleted
# pod 상태를 보니 이제 worker1번 노드에서 pod가 생성됨을 볼수 있습니다.
ps0107@k8smaster1:~$ kubectl -n sock-shop get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
carts-56c6fb966b-rlwwg 1/1 Running 0 54s 192.168.1.37 k8sworker1 <none> <none>
carts-db-5678cc578f-ngvxj 1/1 Running 0 8m2s 192.168.0.20 k8smaster1 <none> <none>
catalogue-644549d46f-p6crp 1/1 Running 0 8m2s 192.168.0.21 k8smaster1 <none> <none>
catalogue-db-6ddc796b66-pqwzf 1/1 Running 0 8m2s 192.168.0.24 k8smaster1 <none> <none>
front-end-5594987df6-v2xvq 1/1 Running 0 54s 192.168.1.38 k8sworker1 <none> <none>
orders-749cdc8c9-7gdht 1/1 Running 0 54s 192.168.1.36 k8sworker1 <none> <none>
orders-db-5cfc68c4cf-bnqmg 1/1 Running 0 8m2s 192.168.0.17 k8smaster1 <none> <none>
payment-54f55b96b9-98jcw 1/1 Running 0 8m2s 192.168.0.25 k8smaster1 <none> <none>
queue-master-6fff667867-qffqg 1/1 Running 0 12m 192.168.0.14 k8smaster1 <none> <none>
rabbitmq-bdfd84d55-fhltf 1/1 Running 0 8m2s 192.168.0.22 k8smaster1 <none> <none>
shipping-78794fdb4f-g2wgm 1/1 Running 0 8m2s 192.168.0.23 k8smaster1 <none> <none>
user-77cff48476-pnkzq 1/1 Running 0 54s 192.168.1.39 k8sworker1 <none> <none>
user-db-99685d75b-6k5wp 1/1 Running 0 8m2s 192.168.0.26 k8smaster1 <none> <none>
# 테스트가 끝났으니, 리소스 정리하자
ps0107@k8smaster1:~$ kubectl delete -f complete-demo.yaml