Golang Integration Testing Made Easy
How we at Gojek implement integration testing in our development processes.
By Arie Ardaya Lizuardi
At Gojek, integration tests play a big role in our software development practices. This ensures that multiple parts of our application work together as expected and helps us meticulously solve problems.
In this blog, I’ll share nitty-gritties about the integration testing approach we implement in our daily software development process — especially backend development.
While this approach could be implemented in different programming languages, the flow helps us solve issues such as:
- Absence of mocking library for that particular dependency. In my case, it was Apache Kafka or when I wanted to test Redis and there was no mocking library for the development language I used.
- Even when there is a mocking library, situations where we want to make sure the query is sent correctly to the DB or when we use caching and want to ensure the cache is created/deleted.
The diagram above describes the flow of our integration testing. We utilise Makefile and docker-compose as our tools. Makefile simplifies testing command execution and Docker compose helps us run multiple containers that we use in our application. The flow would look like this:
Step-1: Create Makefile commands. This sets up our basic testing commands.
Step-2: Create or edit docker-compose.yaml. This defines application dependencies (DB, cache, message broker, etc). You can edit your docker-compose.yaml if you have a new dependency.
Step-3: Create an integration testing code in your favourite language.
Step-4: Execute the test command.
If you have a new use case to test, just repeat Step 3 and 4.
Let’s jump into an example.
Say we have a blog backend API server that will save an article, get articles by id, and get all articles. We will have an end to end test for the whole application flow and also the database interaction test. This is how our folder structure would look like:
[Note: Some folders and files are omitted]
│ ├── handler
│ ├── repository
│ │ └── article_postgres.go
│ ├── usecase
│ ├── article_repository_test.go
│ ├── e2e_test.go
│ ├── it.go
│ └── postgres_suite_test.go
│ ├── server.go
The code inside the server package will have a dependency on the handler, use case, and repository. The end to end testing will spawn the server, and we’ll send some requests and assert the response. The database interaction test will assert the method inside the repository package. You can find the full source code below:
The server code:
The database interaction code inside the repository package:
Create Makefile commands
Originally, Makefile is not only used to list out a set of directions to compile C or C++files, but also for other purposes. In our case, we list out a set of directives for our integration testing commands.
If you’re not too familiar with Makefile, check out this and this to get a better understanding.
Open your command-line interface
# Change directory to the root of your project directory # Create a new file $ touch Makefile # Open your favorite editor to edit Makefile $ nano Makefile
The Makefile will look like this
# we will put our integration testing in this path INTEGRATION_TEST_PATH?=./it # set of env variables that you need for testing ENV_LOCAL_TEST=\ POSTGRES_PASSWORD=mysecretpassword \ POSTGRES_DB=myawesomeproject \ POSTGRES_HOST=postgres \ POSTGRES_USER=postgres # this command will start a docker components that we set in docker-compose.yml docker.start.components: docker-compose up -d --remove-orphans postgres; # shutting down docker components docker.stop: docker-compose down; # this command will trigger integration test # INTEGRATION_TEST_SUITE_PATH is used for run specific test in Golang, if it's not specified # it will run all tests under ./it directory test.integration: $(ENV_LOCAL_TEST) \ go test -tags=integration $(INTEGRATION_TEST_PATH) -count=1 -run=$(INTEGRATION_TEST_SUITE_PATH) # this command will trigger integration test with verbose mode test.integration.debug: $(ENV_LOCAL_TEST) \ go test -tags=integration $(INTEGRATION_TEST_PATH) -count=1 -v -run=$(INTEGRATION_TEST_SUITE_PATH)
Docker compose will help us run multiple Docker containers easily. We will configure our docker dependencies in the YAML file. If you have not installed docker-compose, you can install it from this link.
version: "3.1" services: postgres: image: postgres environment: POSTGRES_PASSWORD: mysecretpassword POSTGRES_DB: myawesomeproject POSTGRES_HOST: postgres POSTGRES_USER: postgres ports: - 5432:5432
This configuration tells docker-compose to run Postgres image. You can add other images depending on your use case.
Create integration test code
Now, we need to create integration testing for the above code. Let’s create e2e_test.go and article_repository_test.go file under it folder.
Execute test integration command from your CLI
// Run docker compose $ make docker.start.components // Run integration testing $ make test.integration
You will see something like this:
DB_HOST=localhost DB_PORT=5432 DB_NAME=blog DB_USER=postgres DB_PASS=mysecretpassword DB_MAX_IDLE_CONN=10 DB_MAX_OPEN_CONN=200 DB_CONN_MAX_LIFETIME=30m ENV=local PORT=8080 \ go test -tags=integration ./it -v -count=1 === RUN TestE2ETestSuite ____ __ / __/___/ / ___ / _// __/ _ \/ _ \ /___/\__/_//_/\___/ v4.1.10 High performance, minimalist Go web framework https://echo.labstack.com ____________________________________O/_______ O\ ⇨ http server started on [::]:8080 === RUN TestE2ETestSuite/Test_EndToEnd_CreateArticle === RUN TestE2ETestSuite/Test_EndToEnd_GetAllArticle === RUN TestE2ETestSuite/Test_EndToEnd_GetArticleByID --- PASS: TestE2ETestSuite (0.18s) --- PASS: TestE2ETestSuite/Test_EndToEnd_CreateArticle (0.03s) --- PASS: TestE2ETestSuite/Test_EndToEnd_GetAllArticle (0.05s) --- PASS: TestE2ETestSuite/Test_EndToEnd_GetArticleByID (0.05s) === RUN TestPostgresRepositoryTestSuite time="2020-08-17T15:44:42+07:00" level=error msg="http: Server closed" time="2020-08-17T15:44:42+07:00" level=info msg="shutting down the server" === RUN TestPostgresRepositoryTestSuite/TestPostgresArticleRepository_CreateArticle === RUN TestPostgresRepositoryTestSuite/TestPostgresArticleRepository_GetAllArticle === RUN TestPostgresRepositoryTestSuite/TestPostgresArticleRepository_GetArticleByID --- PASS: TestPostgresRepositoryTestSuite (0.23s) --- PASS: TestPostgresRepositoryTestSuite/TestPostgresArticleRepository_CreateArticle (0.07s) --- PASS: TestPostgresRepositoryTestSuite/TestPostgresArticleRepository_GetAllArticle (0.07s) --- PASS: TestPostgresRepositoryTestSuite/TestPostgresArticleRepository_GetArticleByID (0.08s) PASS ok github.com/arielizuardi/golang-backend-blog/it 1.810s
Voila! We just finish our integration testing. 🖖
Want to read more stories from the vault? Check out our blogs!