Home [kubernetes-실습] 로깅과 트러블슈팅 : 로그위치와 로그 출력 보기
Post
Cancel

[kubernetes-실습] 로깅과 트러블슈팅 : 로그위치와 로그 출력 보기

ps.jang on Jan 262021-01-26T09:00:00+09:00
Updated Aug 262021-08-26T00:00:00+09:00 8 min read

로그 파일들의 위치 알아보기

  • 다양한 로그 파일과 명령 출력 이외에도 journalctl을 사용하여 노드 관점에서 로그를 볼 수 있다. 우리는 로그 파일의 공통적인 위치를 보고, 컨테이너 로그를 보는 명령을 볼 것이다. 다른 컨테이너의 로그를 pod에 적재하는데 전용으로 사용되는 sidecar container의 사용과 같은 다른 로깅 옵션이 있다. Kubernet에서 전체 클러스터 로깅을 아직 사용할 수 없다. 따라서 kubernetes 같이 CNCF 프로젝트 다른 멤버인 fluentd 같은 외부 software를 사용된다.

  • 다음의 로그 파일과 웹 사이트를 간단히 살펴보십시오. 서버 프로세스가 노드 수준에서 컨테이너안에 실행으로 이동함에 따라 로깅도 이동한다.

journalctl 명령 사용 (kubernetes cluster가 systemd를 사용할때만 가능)

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# -------------------------------------
# journalctl 명령 사용 (kubernetes cluster가 systemd를 사용할때만 가능)
# -------------------------------------
# journalctl 명령 : systemd 로그가 journal로 관리되는데 이런 journal들을 조회할 때 쓰임.
# journalctl <tab x 2> 참조
# journalctl 옵션
# -u : 특정 unit 로그 확인
# -f : 실시간으로 로그 확인
# https://wiki.archlinux.org/index.php/Systemd/Journal

ps0107@k8smaster1:~$ sudo journalctl -u kubelet | tail -5                                                                            
Feb 05 02:10:19 k8smaster1 kubelet[5279]: I0205 02:10:19.268859    5279 reconciler.go:177] operationExecutor.UnmountVolume started for volume "default-token-76w5h" (UniqueName: "kubernetes.io/secret/ccc2b1fb-ffd4-4078-9bd2-b63e662e2ef4-default-token-76w5h") pod "ccc2b1fb-ffd4-4078-9bd2-b63e662e2ef4" (UID: "ccc2b1fb-ffd4-4078-9bd2-b63e662e2ef4")
Feb 05 02:10:19 k8smaster1 kubelet[5279]: I0205 02:10:19.282026    5279 operation_generator.go:860] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24257df3-fa70-47ba-b02f-8bbf5da263dd-default-token-76w5h" (OuterVolumeSpecName: "default-token-76w5h") pod "24257df3-fa70-47ba-b02f-8bbf5da263dd" (UID: "24257df3-fa70-47ba-b02f-8bbf5da263dd"). InnerVolumeSpecName "default-token-76w5h". PluginName "kubernetes.io/secret", VolumeGidValue ""
....

노드 pod/container 로그

  • /var/log/containers/ : 여러 container 로그들
  • /var/log/pods/ : 현재 pod들을 위한 로그 파일들
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62

# -------------------------------------
# 노드 pod/container 로그
# -------------------------------------
# /var/log/containers/ : 여러 container 로그들
# /var/log/pods/ : 현재 pod들을 위한 로그 파일들

# 주요 쿠버네티스 프로세스들은 컨테이너로 실행된다. pod 또는 container 관점으로 볼수 있다.
# 해당 로그가 어디에 위치하고 있는지 보면 /var/log/containers/ 의 컨테이너 로그는 /var/log/pods/ 안에 로그에 심볼릭링크가 걸려있다.
ps0107@k8smaster1:~$ sudo  find / -name "*apiserver*log"
/var/log/containers/kube-apiserver-k8smaster1_kube-system_kube-apiserver-20ee76df3202cd45c43568d04f2c6350c33cf02e082aff03331863c75b9c2e9f.log

ps0107@k8smaster1:~$ sudo tail /var/log/containers/kube-apiserver-k8smaster1_kube-system_kube-apiserver-20ee76df3202cd45c43568d04f2c6350c33cf02e082aff03331863c75b9c2e9f.log
{"log":"Trace[758496088]: [1.54222154s] [1.541983765s] Transaction committed\n","stream":"stderr","time":"2020-01-30T05:16:34.841260969Z"}
{"log":"I0130 05:16:34.824687       1 trace.go:81] Trace[1004905485]: \"Update /api/v1/namespaces/kube-system/endpoints/kube-controller-manager\" (started: 2020-01-30 05:16:33.282084781 +0000 UTC m=+161232.462995704) (total time: 1.542580238s):\n","stream":"stderr","time":"2020-01-30T05:16:34.841266335Z"}
{"log":"Trace[1004905485]: [1.542484521s] [1.542396457s] Object stored in database\n","stream":"stderr","time":"2020-01-30T05:16:34.841281926Z"}
{"log":"I0130 12:43:57.718791       1 log.go:172] http: TLS handshake error from 163.172.16.99:2332: tls: first record does not look like a TLS handshake\n","stream":"stderr","time":"2020-01-30T12:43:57.718994903Z"}
{"log":"I0202 14:09:21.314630       1 controller.go:606] quota admission added evaluator for: jobs.batch\n","stream":"stderr","time":"2020-02-02T14:09:21.314861538Z"}
{"log":"I0202 14:56:06.400851       1 controller.go:606] quota admission added evaluator for: cronjobs.batch\n","stream":"stderr","time":"2020-02-02T14:56:06.401042597Z"}
{"log":"I0203 01:17:35.531438       1 controller.go:606] quota admission added evaluator for: replicasets.extensions\n","stream":"stderr","time":"2020-02-03T01:17:35.531608401Z"}
{"log":"I0203 14:48:02.892069       1 log.go:172] http: TLS handshake error from 198.108.67.48:44142: read tcp 10.146.0.2:6443-\u003e198.108.67.48:44142: read: connection reset by peer\n","stream":"stderr","time":"2020-02-03T14:48:02.892238145Z"}
{"log":"I0204 02:03:55.599499       1 log.go:172] http: TLS handshake error from 198.108.67.48:19574: read tcp 10.146.0.2:6443-\u003e198.108.67.48:19574: read: connection reset by peer\n","stream":"stderr","time":"2020-02-04T02:03:55.599677119Z"}
{"log":"I0204 08:28:48.956703       1 controller.go:606] quota admission added evaluator for: ingresses.extensions\n","stream":"stderr","time":"2020-02-04T08:28:48.956873937Z"}

ps0107@k8smaster1:/var/log/containers$ ls
calico-node-wvxxm_kube-system_calico-node-d2879a40b389a1ef99359af7356ea1efcea497352d96f1e665729f783fdd92b7.log
calico-node-wvxxm_kube-system_install-cni-8e9c02311deebb50195a5f4333b2ba89646cc5799b5542d95f23612018468636.log
coredns-5c98db65d4-97gng_kube-system_coredns-6d822d735d3bb02dd4171294438f9bed5a8ed44821bbadb843cb9422a465c6b4.log
coredns-5c98db65d4-fvz2k_kube-system_coredns-5b255b61a93ed2fe9d3209e8135b3a54824af73690462af0c08a6f4ed55c7148.log
etcd-k8smaster1_kube-system_etcd-3147aa1dc7e6bcf0677e875000730db51b45c1f4e25458077fd921d333829122.log
kube-apiserver-k8smaster1_kube-system_kube-apiserver-20ee76df3202cd45c43568d04f2c6350c33cf02e082aff03331863c75b9c2e9f.log
kube-controller-manager-k8smaster1_kube-system_kube-controller-manager-156da4c1dc59e7144455a2014746da903421de5d7b2120874c069800cbfac717.log                                                                                                                               kube-proxy-t62q4_kube-system_kube-proxy-a6f991ba816b7927cb26cc6016fa45f98d87a0b630d7c897315edc4ed5038df9.log
kube-scheduler-k8smaster1_kube-system_kube-scheduler-7649252338f03b870c74e33731a2f8b162db7b981488d2ccadc1072903ede7d8.log
traefik-ingress-controller-jdvck_kube-system_traefik-ingress-lb-e757ed51250e22dc8977f9ed7b0d26352c57fc4961232e955e80bef14af85ab6.log

ps0107@k8smaster1:/var/log/pods$ ll
total 44
drwxr-xr-x 11 root root   4096 Feb  5 02:10 ./
drwxrwxr-x 10 root syslog 4096 Feb  4 06:25 ../
drwxr-xr-x  4 root root   4096 Jan 28 08:33 kube-system_calico-node-wvxxm_9cff8584-8fa3-493b-a01e-0567c6cd11f2/
drwxr-xr-x  3 root root   4096 Jan 28 08:33 kube-system_coredns-5c98db65d4-97gng_9e2cfd14-4a08-4703-a7e8-d4c85f7878a1/
drwxr-xr-x  3 root root   4096 Jan 28 08:33 kube-system_coredns-5c98db65d4-fvz2k_a4208d56-10d2-4958-b345-01603d5aa57b/
drwxr-xr-x  3 root root   4096 Dec  5 16:19 kube-system_etcd-k8smaster1_92c04c96b41833d6edb6666054fd741f/
drwxr-xr-x  3 root root   4096 Jan 28 08:29 kube-system_kube-apiserver-k8smaster1_214ef7c9b4c2389fde69afa9327c7a33/
drwxr-xr-x  3 root root   4096 Jan 28 08:29 kube-system_kube-controller-manager-k8smaster1_645e7a8519364c082c136bba3c26849b/
drwxr-xr-x  3 root root   4096 Jan 28 08:29 kube-system_kube-proxy-t62q4_3b2fe619-eca0-4356-8ff3-7ef781935d19/
drwxr-xr-x  3 root root   4096 Jan 28 08:29 kube-system_kube-scheduler-k8smaster1_ecae9d12d3610192347be3d1aa5aa552/
drwxr-xr-x  3 root root   4096 Feb  4 08:47 kube-system_traefik-ingress-controller-jdvck_c79ea85c-07b0-4277-939b-9e27d9422159/

# 위에서 조회한 /var/log/containers/ 안에 있는 로그와 같다. /var/log/pods/ 안에 있는 해당 로그가 symbolic link 걸려 있다.
ps0107@k8smaster1:/var/log/pods/kube-system_kube-apiserver-k8smaster1_214ef7c9b4c2389fde69afa9327c7a33/kube-apiserver$ sudo tail 0.log 
{"log":"Trace[758496088]: [1.54222154s] [1.541983765s] Transaction committed\n","stream":"stderr","time":"2020-01-30T05:16:34.841260969Z"}
{"log":"I0130 05:16:34.824687       1 trace.go:81] Trace[1004905485]: \"Update /api/v1/namespaces/kube-system/endpoints/kube-controller-manager\" (started: 2020-01-30 05:16:33.282084781 +0000 UTC m=+161232.462995704) (total time: 1.542580238s):\n","stream":"stderr","time":"2020-01-30T05:16:34.841266335Z"}
{"log":"Trace[1004905485]: [1.542484521s] [1.542396457s] Object stored in database\n","stream":"stderr","time":"2020-01-30T05:16:34.841281926Z"}
{"log":"I0130 12:43:57.718791       1 log.go:172] http: TLS handshake error from 163.172.16.99:2332: tls: first record does not look like a TLS handshake\n","stream":"stderr","time":"2020-01-30T12:43:57.718994903Z"}
{"log":"I0202 14:09:21.314630       1 controller.go:606] quota admission added evaluator for: jobs.batch\n","stream":"stderr","time":"2020-02-02T14:09:21.314861538Z"}
{"log":"I0202 14:56:06.400851       1 controller.go:606] quota admission added evaluator for: cronjobs.batch\n","stream":"stderr","time":"2020-02-02T14:56:06.401042597Z"}
{"log":"I0203 01:17:35.531438       1 controller.go:606] quota admission added evaluator for: replicasets.extensions\n","stream":"stderr","time":"2020-02-03T01:17:35.531608401Z"}
{"log":"I0203 14:48:02.892069       1 log.go:172] http: TLS handshake error from 198.108.67.48:44142: read tcp 10.146.0.2:6443-\u003e198.108.67.48:44142: read: connection reset by peer\n","stream":"stderr","time":"2020-02-03T14:48:02.892238145Z"}
{"log":"I0204 02:03:55.599499       1 log.go:172] http: TLS handshake error from 198.108.67.48:19574: read tcp 10.146.0.2:6443-\u003e198.108.67.48:19574: read: connection reset by peer\n","stream":"stderr","time":"2020-02-04T02:03:55.599677119Z"}
{"log":"I0204 08:28:48.956703       1 controller.go:606] quota admission added evaluator for: ingresses.extensions\n","stream":"stderr","time":"2020-02-04T08:28:48.956873937Z"}

# kube-dns, kube-proxy, kube-flannel 등도 이런 식으로 조회해 볼수 있다.

kubernetes cluster가 systemd를 사용하지 않을때

master node에서 확인

  • /var/log/kube-apiserver.log => API 받는 응답을 볼수 있다.
  • /var/log/kube-scheduler.log => 스케쥴링 결정을 만드는 응답을 볼수 있다.
  • /var/log/kube-controller-manager.log => controller 관련 로그를 볼수 있다.

worker node에서 확인

  • /var/log/kubelet.log => 노드에 실행 중인 컨테이너들의 응답을 볼수 있다
  • /var/log/kube-proxy.log => service load balancing 관련 로그들

그밖에 도움이 되는 웹페이지

  • https://kubernetes.io/docs/tasks/debug-application-cluster/\debug-service/
  • https://kubernetes.io/docs/tasks/debug-application-cluster/\determine-reason-pod-failure/

로그들 출력 보기

  • 컨테이너 standard out 로그는 kubectl logs 명령을 통해서 볼수 있다. 해당 컨테이너가 삭제되면 로그들도 삭제 된다.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

# 컨테이너 standard out 로그는 kubectl logs 명령을 통해서 볼수 있다. 해당 컨테이너가 삭제되면 로그들도 삭제 된다.
# 현재 pod들 목록 보기
ps0107@k8smaster1:~$ kubectl get po --all-namespaces
NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
kube-system   calico-node-wvxxm                    2/2     Running   0          7d22h
kube-system   calico-node-zxkxk                    2/2     Running   2          7d22h
kube-system   coredns-5c98db65d4-97gng             1/1     Running   0          7d22h
kube-system   coredns-5c98db65d4-fvz2k             1/1     Running   0          7d22h
kube-system   etcd-k8smaster1                      1/1     Running   0          7d22h
kube-system   kube-apiserver-k8smaster1            1/1     Running   0          7d22h
kube-system   kube-controller-manager-k8smaster1   1/1     Running   0          7d22h
kube-system   kube-proxy-t62q4                     1/1     Running   0          7d22h
kube-system   kube-proxy-tnmtr                     1/1     Running   1          7d22h
kube-system   kube-scheduler-k8smaster1            1/1     Running   0          7d22h
kube-system   traefik-ingress-controller-8884w     1/1     Running   0          5h11m
kube-system   traefik-ingress-controller-jdvck     1/1     Running   0          22h

# 탭 2번하면 목록을 볼수 있다.
ps0107@k8smaster1:~$ kubectl -n kube-system logs <tab x 2>
calico-node-wvxxm                   etcd-k8smaster1                     kube-proxy-tnmtr
calico-node-zxkxk                   kube-apiserver-k8smaster1           kube-scheduler-k8smaster1
coredns-5c98db65d4-97gng            kube-controller-manager-k8smaster1  traefik-ingress-controller-8884w
coredns-5c98db65d4-fvz2k            kube-proxy-t62q4                    traefik-ingress-controller-jdvck

ps0107@k8smaster1:~$ kubectl -n kube-system logs kube-apiserver-k8smaster1 
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
I0128 08:29:23.585970       1 server.go:560] external host was not specified, using 10.146.0.2
I0128 08:29:23.600341       1 server.go:147] Version: v1.15.1
I0128 08:29:24.066914       1 plugins.go:158] Loaded 10 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,MutatingAdmissionWebhook.

# -f 옵션을 통해 실시간으로 확인 가능하다.
ps0107@k8smaster1:~$ kubectl -n kube-system logs kube-apiserver-k8smaster1 -f
Kubernetes, Kubernetes Administration Course
kubernetes cka LFS458
This post is licensed under CC BY 4.0 by the author.
Recent Update
Contents

[kubernetes-실습] Scheduling - Taint를 이용한 pod 배포 관리

[kubernetes-실습] 로깅과 트러블슈팅 : Metrics와 DashBoard