By Kumar Abhijeet

At Gojek, a lot of our teams use Ruby on Rails to develop a plethora of applications. While building an application, we often need some jobs to be performed asynchronously. Stuff like processing a chunk of data, performing batch executions, or even sending an email.

Congruent to team management, where we delegate some of our jobs to increase the team’s efficiency, we figured we needed an asynchronous job framework for our application to process jobs in the background to sustain the desired efficiency.

But why do we need an async job system? 🤔

In a user-facing system, when the user runs a command or a service, the system should acknowledge the success of the operation, although some critical and time-intensive jobs are still executing in the background.

This use case is solved by an async job execution system, where we can enqueue the jobs to be executed at a later time, and get a heads up when they are executed, while the user-facing system keeps operating smoothly.

It should also allow a large batch of jobs to be scheduled and executed with retries to make the system more reliable and resilient.

Enter Sidekiq

Um, no… Not that sidekick.

Sidekiq is an open-source framework that provides efficient background processing for Ruby applications. It uses Redis as an in-memory data structure to store all of its job and operational data. It’s important to be aware that Sidekiq by default doesn’t do scheduling, it only executes jobs. We can use job scheduling frameworks like Resque, which is also based on Redis.

It provides features to execute any job asynchronously or to be scheduled for later execution, by enqueuing them in a background job queue. It also offers explicit retry strategies in case of a failed execution. Hence, it perfectly fulfils our use case. 🤠

How we use Sidekiq?

Being a part of the Lambda ecosystem, we provide a Heroku-like PAAS framework for our developers to quickly create and deploy units of logic, and therefore, the platform needs to perform a great profusion of jobs like provisioning infrastructure, creation and deletion of tags, DNS, etc.

Hence, for our platform to work seamlessly, we use Sidekiq to push these jobs to the background and perform asynchronously. You can read more about Lambda in this post. 👇

Okay, let’s get our hands dirty 💪

We first need to setup Redis before using Sidekiq.

brew install 
redisredis-server

Let’s add Sidekiq to the Gemfile

gem 'sidekiq'

Run bundle to install sidekiq

bundle

For Sidekiq to work with Rails’ jobs API, we need to tell Rails to use Sidekiq as the queue_adapter; add the below line to config/application.rb

config.active_job.queue_adapter = :sidekiq

Now, we are done with the setup. Let’s roll. 😎

via GIPHY

Let’s create a simple job which sends an email, and uses Active Job, which is a standard interface for interacting with job runners.

SendEmailJob.perform_async(body, email)

The job can be called wherever required and will be pushed to a background queue and performed asynchronously.

Let’s write some tests 🧪

Test-Driven Development is one of the Gojek’s core Tech principles, and we emphasise writing specs first.

We are going to use RSpec as the testing framework, and rspec-sidekiq gem for testing Sidekiq jobs as it provides a collection of matchers and helpers.

Here, we have written specs for ensuring whether the job has enqueued to the correct queue and it calls the perform method correctly.

Some tips to remember while testing Sidekiq jobs:

1. Sidekiq does not make any changes to Redis while testing, hence manipulating with the queue and jobs is not possible.
2. Make use of test doubles in RSpec to mock the queue and jobs, allow required behaviour, and then set expectations.
3. Jobs triggered with perform_later with a set wait time will be pushed to ScheduledSet. Hence upon enqueuing, will wait for other jobs to finish.
4. Make use of the Sidekiq dashboard, which gives monitoring information and is quite useful while debugging.

That’s all from us. Hopefully this post helped you understand how Sidekiq can be used to your advantage. Thanks for reading! 💚