In our previous article series on Basics on Kubernetes which is still going, we talked about different components like control plane, pods, etcd, kube-proxy, deployments, etc. You can read the article series on Learnsteps. In this article, we are going to see how we can do basic debugging in Kubernetes.
Before starting I am assuming that you are aware of kubectl and its usage. While debugging issues it is important to be able to do is look at the events of the Kubernetes components and to do that you can easily use the below command
kubectl describe resource -n namespace
resource is different kubernetes objects like pods, deployments, services, endpoint, replicaset etc.The above command will tell a lot of information about the object and at the end of the information, you have events that are generated by the resource.
Are Kubernetes resources not coming up?
If you created a new resource and there is some issue you can use the describe command and you will be able to see more information on why that resource has a problem. Like one of the cilium pods in kube-system was failing. We can try looking at the events and try to figure out what was wrong.
kubectl describe pods cilium-operator-669b896b78-7jgml -n kube-system
#removed other information as it was too long
Events:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 42d (x2 over 43d) kubelet, minikube Liveness probe failed: Get http://127.0.0.1:9234/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Normal SandboxChanged 4m32s kubelet, minikube Pod sandbox changed, it will be killed and re-created.
Warning BackOff 4m21s (x3 over 4m24s) kubelet, minikube Back-off restarting failed container
Normal Pulled 4m10s (x2 over 4m30s) kubelet, minikube Container image "quay.io/cilium/operator-generic-ci:f6956ca70491cdcf456a2174e5ff64c8c02d18fd" already present on machine
Normal Created 4m10s (x2 over 4m30s) kubelet, minikube Created container cilium-operator
Normal Started 4m9s (x2 over 4m28s) kubelet, minikube Started container cilium-operator
In the events, you can see that the liveness probe for cilium pod was failing. Now, in this case, the application itself is not able to come so the next step that you can take is to look at the application logs. This is very important you can always look at the pod’s logs to verify what is the issue. You can use the below command to look at the pod logs
kubectl logs -f podname -c container_name -n namespaceThis will show you the application logs and if there is something wrong with the application you will be able to see it here.
Not able to send traffic to the application?
Traffic reaches the pod using the service object in Kubernetes. Once your pods are up and you have created a service for the pods. You have to make sure that your service has your pods in your endpoint. You can describe the service to see the status of service, events, and if there are pods in the endpoint component. For this purpose, we will look at the kube-dns service itself.
kubectl describe svc kube-dns -n kube-system
Name: kube-dns
Namespace: kube-system
Labels: k8s-app=kube-dns
kubernetes.io/cluster-service=true
kubernetes.io/name=KubeDNS
Annotations: prometheus.io/port: 9153
prometheus.io/scrape: true
Selector: k8s-app=kube-dns
Type: ClusterIP
IP: 10.96.0.10
Port: dns 53/UDP
TargetPort: 53/UDP
Endpoints: 172.17.0.2:53,172.17.0.3:53 ## IMPORTANT
Port: dns-tcp 53/TCP
TargetPort: 53/TCP
Endpoints: 172.17.0.2:53,172.17.0.3:53 ## IMPORTANT
Port: metrics 9153/TCP
TargetPort: 9153/TCP
Endpoints: 172.17.0.2:9153,172.17.0.3:9153 ## IMPORTANT
Session Affinity: None
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedToUpdateEndpointSlices 42d (x597 over 51d) endpoint-slice-controller Error updating Endpoint Slices for Service kube-system/kube-dns: Error updating kube-dns-225jd EndpointSlice for Service kube-system/kube-dns: Operation cannot be fulfilled on endpointslices.discovery.k8s.io "kube-dns-225jd": the object has been modified; please apply your changes to the latest version and try again
Warning FailedToUpdateEndpointSlices 10m (x5 over 10m) endpoint-slice-controller Error updating Endpoint Slices for Service kube-system/kube-dns: node "minikube" not found
If you see above the endpoint are 172.17.0.2 and 172.17.0.3 these are our core DNS pods IPs. So here kube-dns service has a backend to send traffic to.
You can also look at all the Kubernetes events using the below command
kubectl get eventsThis will tell all the events from the Kubernetes cluster like below
LAST SEEN TYPE REASON OBJECT MESSAGE
2m30s Normal Starting node/minikube Starting kubelet.
2m28s Normal NodeHasSufficientMemory node/minikube Node minikube status is now: NodeHasSufficientMemory
2m28s Normal NodeHasNoDiskPressure node/minikube Node minikube status is now: NodeHasNoDiskPressure
2m28s Normal NodeHasSufficientPID node/minikube Node minikube status is now: NodeHasSufficientPID
2m29s Normal NodeAllocatableEnforced node/minikube Updated Node Allocatable limit across pods
110s Normal Starting node/minikube Starting kube-proxy.
103s Normal RegisteredNode node/minikube Node minikube event: Registered Node minikube in Controller
10s Normal RegisteredNode node/minikube Node minikube event: Registered Node minikube in ControllerConclusion
kubectl describe command and kubectl log are very powerful and most of the issues will be solved by these. If you know the resources that can be created you can just run describe command on it and the events will tell you if there is something wrong. Then there are advanced issues that were not the target of this article. Always use these commands to debug issues before trying out anything advanced.
If you like the article please share and subscribe.
