Darkroom: Realtime Image Processing

By Deepesh Naini

The GO-FOOD Engineering team is responsible for looking after the Food vertical of GO-JEK. To give you a sense of proportion, we do more food delivery orders in Indonesia than India’s largest food-delivery service.

The most important part of our ecosystem is content (food items, description etc.) and images — together they form an integral part of our content system. Content can be created and updated by the in-house Content Management System (CMS) team. Merchants can also directly manage their content via the merchant facing app, GO-RESTO.

We were spending more than $20,000 every month on a third party application to do real-time image processing on our GO-FOOD app. We knew we could build this in-house and save money and time. This is the story of how we went about doing it.

IMPORTANT: I worked on this project along with three other interns. Suffice to say, working as an intern at GO-JEK gives you enough scope to solve actual challenging real-world problems. We don’t accept candidates if we don’t have problems to solve and if there’s one thing I’ve learnt about solving for GO-JEK: we have no shortage of complex technical problems when we have a Super App of 18 products.

Problem Statement

The image upload lifecycle consists of a watermarking process and image compression, post which it is stored in an S3 bucket.

When the images are to be displayed on the consumer app, we first calculate the container where the image needs to be rendered and then the image is requested as per the calculated size. The calculation of parameters such as height, width, crop, etc… depends on the resolution of the device on which the image is getting rendered. Considering the variety of devices available today and the scale at which we operate, the daily number of requests for images from the app are in the magnitude of millions.

Solution

Until about a month back we were using a third party application for rendering images on the GO-FOOD app. The third-party application exposes REST endpoints that accepts parameters and modifies images in realtime. This application as a service was costing us more than $20k each month.

The birth of Darkroom

Every year we get summer internship students and we throw open challenging problems for them. This year we got 3 interns to create a drop-in replacement of the third party application we were using. We called it Darkroom. (In case you were wondering, there was a time when photos were printed and this was done in a Darkroom. Didn’t happen in the Dark Ages, just FYI 😉)

Key Features of Darkroom:

1. Register an Application: As part of the registration process, each user gets an AppId which is linked to the S3 bucket, credentials, folder path and the Cloudfront URL.
2. Watermarking: As part of the watermarking, each user receives the watermarked image by specifying the AppId, watermark image and the image to be watermarked.
3. Realtime image processing: This is a REST endpoint which is exposed by Darkroom which can accept filters like height, width, crop parameters, grayscale filters, etc and returns the image.

How it works

As shown above, all requests coming to Darkroom are routed via an edge network CDN cache. This way repeated requests of the same image and same width, height parameters can simply be returned from the edge network, thereby controlling the load on app servers and quicker response times.

Darkroom is written in GoLang and is a compute intensive application. Therefore, any image manipulation it does consumes a lot of CPU power. The memory footprint of the application is not high.

Darkroom also makes sure it does not distort the image while cropping or resizing. It takes into account the aspect ratio of the original image and decides the target width or height.

Requests served by Darkroom

The traffic pattern of Darkroom is unpredictable and depends on a lot of external factors. For example, people trying to browse more food options on holidays, or people trying to order food because of promotions, etc.

At its peak, Darkroom has served 500k Requests per Minute. On a daily average it serves about 300k Requests per Minute.

Note the drop on July 19th? That’s when Darkroom was launched. We were not reliant on a third party app anymore. And, importantly, there was zero downtime when Darkroom went live — which means our customers didn’t have to update the app or notice any change in the app.

Next Steps

It took us two months to build this out and in the process we saved money, time and automated a vital component for GO-JEK. Lean engineering is about taking a bunch of these small tasks and automating them, and there are plenty more examples of this in our blog. Our next vision for Darkroom was to open source it into a SAAS product. Read more about how we did that in this post.