Kubehandler: An event dispatcher for Kube controllers

By Ravi Chandra Padmala

For the past month, we’ve been working on a custom Kubernetes controller to help manage our resources. Since there isn’t much documentation around this, we started off with the Kubernetes sample-controller as a reference implementation.

We soon got to a point where we were subscribing to more than a handful of
informers.

The structure of the controller.go file in the sample-controller forced us to…

1. Have a lot of repetitive code that was not easy to test⁰.
2. Test many of our private functions rather than just the interface. Though mocking the informers and clientsets using fake clientsets is possible, it’s quite tedious.
3. Add a lot of boilerplate code for each resource we handled.

Some boilerplate code that we had to add:

1. Adding dependencies to the controller struct to store Listers and InformerSynced flags
2. Creating a new event handler and adding it to the corresponding Informer in the controller constructor
3. Creating an enqueue function to put events into the work queue
4. A handler func that handles messages of this resource type present in the workqueue
5. Adding dispatch logic to syncHandler for this resource[1]

Most of this boilerplate was untested because they were all just private methods on the controller. On top of this, we needed to handle many more types of resources. By doing this, we were in gross violation of DRY (Don’t Repeat Yourself) and also had a lot of untested code.

Kubehandler solves this problem.

By extracting the common event handling code into a separate file and building a registration mechanism into it, we were able to define an interface for the EventHandlers. We then define an EventLoop which invokes the EventHandler.

The EventHandler interface looks like:

type EventHandler interface {
 GetName() string
 GetSynced() cache.InformerSynced
 GetInformer() cache.SharedInformer
 AddFunc(namespace, name string) error
 UpdateFunc(namespace, name string) error
 DeleteFunc(namespace, name string) error
}

As you can see, Kubehandler abstracts out the boilerplate code mentioned earlier. It’s important to keep in mind that the current implementation ignores the state of the resource passed in to the handler. It‘s the responsibility of the handler to look up the resource when it is being processed via the clientset.

Kubehandler also makes it easier to test event handlers. It gets rid of a
lot of duplicate code and you end up having to test only the AddFunc, UpdateFunc and DeleteFunc behaviour.

Here is a sample test…

func TestShouldUpdateDeploymentStatus(t *testing.T) {
    deployName := "test-deploy"
    deploy := &appsv1.Deployment{
        ObjectMeta: metav1.ObjectMeta{
            Name:      deployName,
        },
        Spec: appsv1.DeploymentSpec{},
    }

    mockClient := &MockClient{}
    decoratedDeployment := operatorV1.DecoratedDeployment{}
    mockClient.On("GetDeployment", deployName).Return(deploy, nil)

    handler := DeploymentsHandler{
        Client:         mockClient,
        DefaultHandler: kubehandler.DefaultHandler{},
    }

    err := handler.UpdateFunc("default", deployName)
    mockClient.AssertExpectations(t)
}

Kubehandler has definitely made our code DRYer and maintainable, we would love to hear about your experiences with it!

It’s available on github at: https://github.com/gojektech/kubehandler.

Please leave a comment if you find this interesting and have any tips, suggestions. We’re expanding outside Indonesia! Please grab this chance to work with a Super App that has 18+ products and growing exponentially.

[0] We also looked at the Kubernetes code to see how controllers were tested. In some of the code we looked at (job_controller_test.go in particular) the controller test mutates the subject under test by redefining private functions and we definitely wanted to avoid doing that.

[1] In the early stages, we used a single shared workqueue with dispatching done in the syncHandler function.