What is Optimistic Concurrency? How does is it help to scale the Kubernetes cluster?


In this blog post, we will know how does optimistic concurrency helps Kubernetes to scale and accommodate more and more plugins into it. We will also try to give you certain examples in order for us to understand them better.

Brief about Kubernetes. 

Kubernetes is an orchestration tool which helps us to orchestrate different application for us. The base or heart of the Kubernetes cluster is a different type of controller which runs together in order to complete a task. As it is based on a declarative model. It means that we throw a configuration or a final state to Kubernetes and a different controller will act on it and try to reach the desired state. Different part of Kubernetes Cluster

Etcd

It provides persistence to a Kubernetes cluster. Kubernetes uses this for storing each and every state and also used to provide a way for service discovery.

API server

Abstraction layer to talk to etcd. It’s not a good idea to access ETCd directly. It applies different Validation like Authorization and Authentication to request which comes to Kubernetes

Controllers

It is a simple control loop that reads the state from etcd and reacts towards it. The basic loop is:-

Step 1: Watch or reads for any changes
Step 2: Change the state of the object (it may be a Kubernetes object like create a POD or any outer world objects like creating AWS s3 bucket or so on.) in order to achieve the desired state.
Step 3: Update the status
Step 4: Repeat with Step 1.

An example of controllers is Kubelet, Deployment, Custom Cloud Controller.

Kubernetes API server

It’s not a good thing to talk to etcd directly so we need some kind of middleman which can provide us with different more facilities. It is the only component which talks to etcd.

Brief about Optimistic Concurrency Control.

Let’s imagine a scenario where everyone trying to book a seat in a movie theatre in the month of February. As corner seats are in great demand in a valentine’s week (trust me I have heard this from somewhere). Preventing two couples from booking the same pair of seats at the same time can be done through concurrency control. There are two ways to control concurrency in this scenario:- 

    1. Pessimistic concurrency control

    2. Optimistic concurrency control. 

We are going to discuss Optimistic concurrency control as of now. Optimistic concurrency does not take precautions about collision beforehand and interrupts only and only when the collision is detected. It is done by version control. See every document has a version and if you are reading and writing to the same version then there is no collision. But if you reading a different version and while writing the version has changed then (someone else was first) then bam there is a collision. It will throw an error like this:- “Version Mismatched”. This is an indication that you need to retry the transaction again.

Time of BookingCouple 1Couple 2
Time 1Start BOOKINGStarting BOOKING
Time 2Read Booking Information (Version V0)Read Booking Information
Time3Changes information like V0.name and V0.ageChanges information like V0.name and V0.age
Time 4Write Information to DBModifies V0.paymentMethod
Time 5Commit
Time 6A version of Booking Row Changed to V1Write record to DB
Time 7
As the version changed to V1. Throws Version Mismatch Error.
Time 8
Restart Transaction

Benefit Of Optimistic Concurrency Includes

  1. It can scale well if the throughput of the write requests is on a higher side.

Disadvantages of Optimistic Concurrency

  1. Once the Transaction fails we need to restart the whole process again. 

How does a Pod Get Launched When you submit A POD definition to Kubernetes?

We consider a very basic Kubernetes cluster that contains only a few controllers like scheduler and kubelet for simplicity case. Optimistic concurrency in Kubernetes is handled by a key resource version which is present in each and every resource in the metadata section. Resource version changes whenever we try to update a resource. 

So this is what happens when we try to create a pod and how optimistic concurrency helps us to do thing in a scalable way 

Step 1: When we say kubectl apply -f pod_manifest.yaml. Kubectl which is a Kubernetes client talks to the Kubernetes API server and creates an entry to etcd. A pod resource is created in etcd which has resource version 1 (let’s say).

Step 2: As we have different controllers running. Their job is to listen to the changes ( How does it listen to events? We can have a different blog post for this) and react to it.

Step 3: So as soon as a pod manifest is created. The Kubelet and scheduler controller rises and starts to do their work. Now both try to read the state so that it can start working on it.

Step 4: Scheduler controller as it can see that in pod spec nodeName is not there so it will start its business logic and get the appropriate node for this pod. After getting the appropriate node name it updates the pod spec with the correct node name. (Note that after updates resource version gets increased)

Step 5: Meanwhile, Kubelet sees this, and after getting the correct node name from the spec checks whether the node name equals to kubelet name or not. If it does not equal to kubelet name then it goes back to sleep. If it equals the kubelet name it starts the container and updates the status ( And hence the resource version gets increases).

Seems easy right? I have not taken different controllers like Kube proxy and some low-level intricacies. Moreover, in a production environment, no one tries to deploy a single pod. It is managed by deployment resources or so. Hence increases the complexity manifold.

Role of Optimistic concurrency in these controllers:- 

As all the controllers run in a distributed independent manner. So we can think of Kubernetes as a very big distributed system. The concurrency problem can occur at any time. Let us take an example:- 

TimeScheduler (controller 1)Kubelet (Controller 2)
Time 1Submits Pod ManifestsSubmits Pod Manifests
Time 2Read Pod Information(ResourceVersion V0)Read Pod Information(ResourceVersion V0)
Time3Changes information like V0.spec.schedulerName and V0.spec.nodeNameChanges information like v0.status.conditions.lastProbeTime
Time 4Contacts API Server for persisting it to etcdModifies V0.status.phase to “Running”
Time 5Writes into etcd
Time 6A resource Version of Pod is  V1Contacts API Server for persisting it to etcd
Time 7
As the version changed to V1. Apis Throws Version Mismatch Error.
Time 8
Retries the whole controller logic again

As you can imagine controllers have distributed and independent this type of pattern where multiple controllers can read and write into a single resource this scenario is very frequent. As we scale our Kubernetes in terms of functionality like introducing multiple Custom Resources into a cluster this type of issue can come anytime. So introducing Optimistic concurrency not only makes this simple ( Just retries ) and also scalable too. 

If you like the article please share and subscribe.


Rahul Sinha

Rahul is a full stack engineer, loves FIFA and Itachi Uchiha and a go to guy for your javascript problems.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Are you preparing for DevOps and SRE Interview?

Here is a book for you.


Interview preparation and interview questions for DevOps and SRE






A good reads for last-minute preparations for handling DevOps and SRE interviews at companies like LinkedIn, Visa, Atlassian, etc.