In our previous article series on Basics on Kubernetes which is still going, we talked about different components like control plane, pods, etcd, kube-proxy, deployments, etc. You can read the article series on Learnsteps. In this article, we are going to see how we can do basic debugging in Kubernetes.
Before starting I am assuming that you are aware of kubectl and its usage. While debugging issues it is important to be able to do is look at the events of the Kubernetes components and to do that you can easily use the below command
kubectl describe resource -n namespace resource is different kubernetes objects like pods, deployments, services, endpoint, replicaset etc.
The above command will tell a lot of information about the object and at the end of the information, you have events that are generated by the resource.
Are Kubernetes resources not coming up?
If you created a new resource and there is some issue you can use the describe command and you will be able to see more information on why that resource has a problem. Like one of the cilium pods in kube-system was failing. We can try looking at the events and try to figure out what was wrong.
kubectl describe pods cilium-operator-669b896b78-7jgml -n kube-system #removed other information as it was too long Events: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Unhealthy 42d (x2 over 43d) kubelet, minikube Liveness probe failed: Get http://127.0.0.1:9234/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers) Normal SandboxChanged 4m32s kubelet, minikube Pod sandbox changed, it will be killed and re-created. Warning BackOff 4m21s (x3 over 4m24s) kubelet, minikube Back-off restarting failed container Normal Pulled 4m10s (x2 over 4m30s) kubelet, minikube Container image "quay.io/cilium/operator-generic-ci:f6956ca70491cdcf456a2174e5ff64c8c02d18fd" already present on machine Normal Created 4m10s (x2 over 4m30s) kubelet, minikube Created container cilium-operator Normal Started 4m9s (x2 over 4m28s) kubelet, minikube Started container cilium-operator
In the events, you can see that the liveness probe for cilium pod was failing. Now, in this case, the application itself is not able to come so the next step that you can take is to look at the application logs. This is very important you can always look at the pod’s logs to verify what is the issue. You can use the below command to look at the pod logs
kubectl logs -f podname -c container_name -n namespace
This will show you the application logs and if there is something wrong with the application you will be able to see it here.
Not able to send traffic to the application?
Traffic reaches the pod using the service object in Kubernetes. Once your pods are up and you have created a service for the pods. You have to make sure that your service has your pods in your endpoint. You can describe the service to see the status of service, events, and if there are pods in the endpoint component. For this purpose, we will look at the kube-dns service itself.
kubectl describe svc kube-dns -n kube-system Name: kube-dns Namespace: kube-system Labels: k8s-app=kube-dns kubernetes.io/cluster-service=true kubernetes.io/name=KubeDNS Annotations: prometheus.io/port: 9153 prometheus.io/scrape: true Selector: k8s-app=kube-dns Type: ClusterIP IP: 10.96.0.10 Port: dns 53/UDP TargetPort: 53/UDP Endpoints: 172.17.0.2:53,172.17.0.3:53 ## IMPORTANT Port: dns-tcp 53/TCP TargetPort: 53/TCP Endpoints: 172.17.0.2:53,172.17.0.3:53 ## IMPORTANT Port: metrics 9153/TCP TargetPort: 9153/TCP Endpoints: 172.17.0.2:9153,172.17.0.3:9153 ## IMPORTANT Session Affinity: None Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedToUpdateEndpointSlices 42d (x597 over 51d) endpoint-slice-controller Error updating Endpoint Slices for Service kube-system/kube-dns: Error updating kube-dns-225jd EndpointSlice for Service kube-system/kube-dns: Operation cannot be fulfilled on endpointslices.discovery.k8s.io "kube-dns-225jd": the object has been modified; please apply your changes to the latest version and try again Warning FailedToUpdateEndpointSlices 10m (x5 over 10m) endpoint-slice-controller Error updating Endpoint Slices for Service kube-system/kube-dns: node "minikube" not found
If you see above the endpoint are 172.17.0.2 and 172.17.0.3 these are our core DNS pods IPs. So here kube-dns service has a backend to send traffic to.
You can also look at all the Kubernetes events using the below command
kubectl get events
This will tell all the events from the Kubernetes cluster like below
LAST SEEN TYPE REASON OBJECT MESSAGE 2m30s Normal Starting node/minikube Starting kubelet. 2m28s Normal NodeHasSufficientMemory node/minikube Node minikube status is now: NodeHasSufficientMemory 2m28s Normal NodeHasNoDiskPressure node/minikube Node minikube status is now: NodeHasNoDiskPressure 2m28s Normal NodeHasSufficientPID node/minikube Node minikube status is now: NodeHasSufficientPID 2m29s Normal NodeAllocatableEnforced node/minikube Updated Node Allocatable limit across pods 110s Normal Starting node/minikube Starting kube-proxy. 103s Normal RegisteredNode node/minikube Node minikube event: Registered Node minikube in Controller 10s Normal RegisteredNode node/minikube Node minikube event: Registered Node minikube in Controller
kubectl describe command and
kubectl log are very powerful and most of the issues will be solved by these. If you know the resources that can be created you can just run describe command on it and the events will tell you if there is something wrong. Then there are advanced issues that were not the target of this article. Always use these commands to debug issues before trying out anything advanced.
If you like the article please share and subscribe.