Learn Steps

Basics on Kubernetes: Basic debugging with kubectl

In our previous article series on Basics on Kubernetes which is still going, we talked about different components like control plane, pods, etcd, kube-proxy, deployments, etc. You can read the article series on Learnsteps. In this article, we are going to see how we can do basic debugging in Kubernetes.

Before starting I am assuming that you are aware of kubectl and its usage. While debugging issues it is important to be able to do is look at the events of the Kubernetes components and to do that you can easily use the below command

kubectl describe resource -n namespace
resource is different kubernetes objects like pods, deployments, services, endpoint, replicaset etc.

The above command will tell a lot of information about the object and at the end of the information, you have events that are generated by the resource.

Are Kubernetes resources not coming up?

If you created a new resource and there is some issue you can use the describe command and you will be able to see more information on why that resource has a problem. Like one of the cilium pods in kube-system was failing. We can try looking at the events and try to figure out what was wrong.

kubectl describe pods cilium-operator-669b896b78-7jgml -n kube-system


#removed other information as it was too long
Events: 
Events:
  Type     Reason          Age                    From               Message
  ----     ------          ----                   ----               -------
  Warning  Unhealthy       42d (x2 over 43d)      kubelet, minikube  Liveness probe failed: Get http://127.0.0.1:9234/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
  Normal   SandboxChanged  4m32s                  kubelet, minikube  Pod sandbox changed, it will be killed and re-created.
  Warning  BackOff         4m21s (x3 over 4m24s)  kubelet, minikube  Back-off restarting failed container
  Normal   Pulled          4m10s (x2 over 4m30s)  kubelet, minikube  Container image "quay.io/cilium/operator-generic-ci:f6956ca70491cdcf456a2174e5ff64c8c02d18fd" already present on machine
  Normal   Created         4m10s (x2 over 4m30s)  kubelet, minikube  Created container cilium-operator
  Normal   Started         4m9s (x2 over 4m28s)   kubelet, minikube  Started container cilium-operator

In the events, you can see that the liveness probe for cilium pod was failing. Now, in this case, the application itself is not able to come so the next step that you can take is to look at the application logs. This is very important you can always look at the pod’s logs to verify what is the issue. You can use the below command to look at the pod logs

kubectl logs -f podname -c container_name -n namespace

This will show you the application logs and if there is something wrong with the application you will be able to see it here.

Not able to send traffic to the application?

Traffic reaches the pod using the service object in Kubernetes. Once your pods are up and you have created a service for the pods. You have to make sure that your service has your pods in your endpoint. You can describe the service to see the status of service, events, and if there are pods in the endpoint component. For this purpose, we will look at the kube-dns service itself.

kubectl describe svc kube-dns -n kube-system 

Name:              kube-dns
Namespace:         kube-system
Labels:            k8s-app=kube-dns
                   kubernetes.io/cluster-service=true
                   kubernetes.io/name=KubeDNS
Annotations:       prometheus.io/port: 9153
                   prometheus.io/scrape: true
Selector:          k8s-app=kube-dns
Type:              ClusterIP
IP:                10.96.0.10
Port:              dns  53/UDP
TargetPort:        53/UDP
Endpoints:         172.17.0.2:53,172.17.0.3:53  ## IMPORTANT
Port:              dns-tcp  53/TCP
TargetPort:        53/TCP
Endpoints:         172.17.0.2:53,172.17.0.3:53  ## IMPORTANT
Port:              metrics  9153/TCP
TargetPort:        9153/TCP
Endpoints:         172.17.0.2:9153,172.17.0.3:9153 ## IMPORTANT
Session Affinity:  None
Events:
  Type     Reason                        Age                  From                       Message
  ----     ------                        ----                 ----                       -------
  Warning  FailedToUpdateEndpointSlices  42d (x597 over 51d)  endpoint-slice-controller  Error updating Endpoint Slices for Service kube-system/kube-dns: Error updating kube-dns-225jd EndpointSlice for Service kube-system/kube-dns: Operation cannot be fulfilled on endpointslices.discovery.k8s.io "kube-dns-225jd": the object has been modified; please apply your changes to the latest version and try again
  Warning  FailedToUpdateEndpointSlices  10m (x5 over 10m)    endpoint-slice-controller  Error updating Endpoint Slices for Service kube-system/kube-dns: node "minikube" not found

If you see above the endpoint are 172.17.0.2 and 172.17.0.3 these are our core DNS pods IPs. So here kube-dns service has a backend to send traffic to.

You can also look at all the Kubernetes events using the below command

kubectl get events

This will tell all the events from the Kubernetes cluster like below

LAST SEEN   TYPE     REASON                    OBJECT          MESSAGE
2m30s       Normal   Starting                  node/minikube   Starting kubelet.
2m28s       Normal   NodeHasSufficientMemory   node/minikube   Node minikube status is now: NodeHasSufficientMemory
2m28s       Normal   NodeHasNoDiskPressure     node/minikube   Node minikube status is now: NodeHasNoDiskPressure
2m28s       Normal   NodeHasSufficientPID      node/minikube   Node minikube status is now: NodeHasSufficientPID
2m29s       Normal   NodeAllocatableEnforced   node/minikube   Updated Node Allocatable limit across pods
110s        Normal   Starting                  node/minikube   Starting kube-proxy.
103s        Normal   RegisteredNode            node/minikube   Node minikube event: Registered Node minikube in Controller
10s         Normal   RegisteredNode            node/minikube   Node minikube event: Registered Node minikube in Controller

Conclusion

kubectl describe command and kubectl log are very powerful and most of the issues will be solved by these. If you know the resources that can be created you can just run describe command on it and the events will tell you if there is something wrong. Then there are advanced issues that were not the target of this article. Always use these commands to debug issues before trying out anything advanced.

If you like the article please share and subscribe.