Aspecto blog

On microservices, OpenTelemetry, and anything in between

How To Write Integration Tests Easily Using Trace-Based Testing

OpenTelemetry, Malabi, MongoDB, Jest, HTTP in the lab

Share this post

This article is part of the Aspecto Hello World series, where we tackle microservices-related topics for you. Our team searches the web for common issues, then we solve them ourselves and bring you complete how-to guides. Aspecto is an OpenTelemetry-based distributed tracing platform for developers and teams of distributed applications.

Hands-on: What to expect


Integration tests are never fun to write.

Setting up an environment, finding a way to reproduce the needed state of it, populating it with relevant data, etc – It’s all work no play.

In this blog post, I will show you how to write an integration test and talk about why you want to do it.

The motivation for writing an integration test

Let’s begin by talking about why we even want to write one.

So you finished writing all of the code of your task and everything is working.

How do you make sure it stays this way and does not break when another developer changes just one tiny thing that happens to break your feature? (Not on purpose of course, or so we hope).

The solution: Instead of moving onto the next feature right away, you can write an integration test to make sure that your team gets notified if something breaks (by running it in a CI/CD pipeline that alerts you). 

The role of the integration test is to verify that not only does each function you wrote works as a standalone (that’s unit tests), but also to verify that all of the functions play out well together.

Spoiler alert: in this guide, I will also show you how to make sure that a database received the relevant params.

What we will build:

I like to keep everything as simple as possible, giving you only what you need to get on with your day.

Therefore, I will build a simple to-do app: a nodejs express service that authenticates the user, with endpoints for creating & viewing his tasks.

The integration test will verify the following:

  1. Login
  2. Create a new TODO item
  3. Verify the todo item was saved with the relevant email & todo text was saved to the database
  4. Fetch all todo items
  5. Verify that the relevant email was used to query the database

The tools I will be using:

  1. NodeJS + Express
  2. Jest – a framework for authoring and executing nodejs tests
  3. Malabi – a Trace-Based Testing library that lets me make assertions on real data passed to moving parts like mongodb / elasticsearch and more (I will further elaborate on this below).

Part 1 – Setting up the tested microservice:

Note: If you already have a service you wish to test, feel free to skip to part 2 where I show how to write the actual integration test.

Step 1 – Set up the project

Let’s use express-generator to generate the initial code for our express app, with the pug view engine.

Also, let’s install the packages we will be using:

npx express-generator -v pug
npm i --save jsonwebtoken mongoose passport passport-jwt axios

We will be storing the data in a MongoDB database, which you can run locally using docker like this:

docker pull mongo
docker run -d -p 27017:27017 mongo

Step 2 – create the views:

We’ll add a very simple UI for our imaginary users to add TODO items.


extends layout
block content
   form(action='/auth/login?redirectToTodos=true', method='POST')
       | username:
       input(type='text', name='username', value='')
       |          <br/>password:
       input(type='password', name='password', value='password')
     input(type='submit', value='Submit')


extends layout
block content
   p Welcome to the app screen <b>#{user}</b>
   form(action='/todos/', method='POST')
           | new todo text:
           input(type='text', name='username', value='hi')
       input(type='submit', value='Submit')
     each val in todos

Step 3 – Adding routes

In the routes folder, let’s delete the users’ file, we won’t be using it.

Now place the following code in the index.js file.


const var express = require('express');
const router = express.Router();
const jwt = require('jsonwebtoken');
router.get('/login', function(req, res, next) {
 res.render('login', { title: 'Express' });
});'/auth/login', function(req, res, next) {
 const email = req.body.username;
 const token = jwt.sign({ email }, TOKEN_SECRET, {
   expiresIn: 60 * 60,
 res.cookie('auth', token, { httpOnly: true });
 req.query.redirectToTodos ? res.redirect('/todos') : res.json({ success: true });

module.exports = router;

Here we have two endpoints – one for rendering the login view for our users, and the other one for performing the login with a username & password. We’re not verifying passwords here, as authentication is not the focus of the blog.

The way it works in general: login, sign a JWT token with email, send to client with the response.

I added a small modification here – redirectToTodos param, to avoid writing client-side code.

In real life, I would not add it and have the client handle the JSON response. For the purpose of this post, it’s good enough.

Let’s add another routes file called todos.js:

const express = require('express');
const router = express.Router();
const passport = require('passport');
const mongoose = require('mongoose');
async function connectToMongoose() {
 await mongoose.connect('mongodb://localhost:27017/test');
connectToMongoose().catch(err => console.log(err));
const todoSchema = new mongoose.Schema({
 text: String,
 email: String
const Todo = mongoose.model('Todo', todoSchema);
 (req, res, next) => {
   passport.authenticate('jwt', { session: false },  async (err, user, info) => {
     if (err) {
       console.log('error is', err);
       res.status(500).send('An error has occurred, we cannot greet you at the moment.');
     else {
       console.log('user is', user);
       console.log('info is', info);
       const { email } = user;
       const todos = await Todo.find({ email });
       res.render('app',{ success: true, user: email, todos });
   })(req, res, next);
 (req, res, next) => {
   passport.authenticate('jwt', { session: false },  async (err, user, info) => {
     if (err) {
       console.log('error is', err);
       res.status(500).send('An error has occurred, we cannot greet you at the moment.');
     else {
       console.log('user is', user);
       console.log('info is', info);
       const todo = new Todo({ text: 'Hey that is my new todo', email: });
       res.json({ success: true });
   })(req, res, next);

module.exports = router;

In this file I define the mongoose Todo model and add two endpoints: GET for retrieving all todos for the current user, and POST for creating one.

I use the passport-jwt strategy to decode the user’s email (which is sent with the request as a cookie)

Step 4 – Adding app.js

The app.js file contains different setups for authentication & routing. I won’t dive into this deeply. If you’re curious feel free to check out my guide to microservices authentication strategies.

var createError = require('http-errors');
var express = require('express');
var path = require('path');
var cookieParser = require('cookie-parser');
var logger = require('morgan');
var indexRouter = require('./routes/index');
var todoRouter = require('./routes/todos');
var app = express();
// view engine setup
app.set('views', path.join(__dirname, 'views'));
app.set('view engine', 'pug');
app.use(express.urlencoded({ extended: false }));
app.use(express.static(path.join(__dirname, 'public')));
const passport = require('passport');
const JwtStrategy = require('passport-jwt').Strategy,
 ExtractJwt = require('passport-jwt').ExtractJwt;
app.use('/', indexRouter);
app.use('/todos', todoRouter);
// catch 404 and forward to error handler
app.use(function(req, res, next) {
// error handler
app.use(function(err, req, res, next) {
 // set locals, only providing error in development
 res.locals.message = err.message;
 res.locals.error ='env') === 'development' ? err : {};
 // render the error page
 res.status(err.status || 500);
const cookieExtractor = function(req) {
 let token = null;
 if (req && req.cookies)
   token = req.cookies['auth'];
 return token;
const opts = {
 jwtFromRequest: ExtractJwt.fromExtractors([cookieExtractor]),
 secretOrKey: TOKEN_SECRET,
 new JwtStrategy(opts, (jwt_payload, done) => {
   try {
     console.log('jwt_payload', jwt_payload);
     done(null, jwt_payload);
   } catch (err) {
module.exports = app;

Step 5 – Running the app

Now you can simply run the app using and go to localhost:3000/login, login and submit the form to create a todo item and see that it is working.

npm start

Part 2: Writing an integration test

In this part, we will be using jest & Malabi to write our integration test.

Jest is a library that lets you author and execute tests for nodejs. 

As for Malabi – an explanation follows.

Malabi is an open-source Javascript framework based on OpenTelemetry that allows you to leverage trace data and improve assertion capabilities. This library introduces a new way of testing services: Trace-based testing (TBT).

Here’s a quick tutorial on Malabi and how it works

Trace based testing

So what is Trace Based Testing anyway? Good thing you ask.

It is a new approach that utilizes OpenTelemetry to our advantage in testing.

Opentelemetry gives us SDKs that let us instrument our application, meaning to create data of what calls are being made to endpoints, databases and essentially is aimed at giving us visibility on our microservices.

You can read more about Malabi and Trace-based testing in this blog post.

By instrumenting our tested microservice, we’re able to make assertions on what API calls were made, with what parameters. 

We can also assert how a database was queried – in our case, which exact parameter did Mongo receive when it was asked for all todos by the current user. We want to assert the current user and know we don’t have a bug causing us to log in with one user but query another user’s data. 

Malabi wraps the opentelemetry SDK, creates spans (actions performed in the service), and exposes an HTTP endpoint for you to query & assert on.

Enough theory for now – Let’s begin.

Step 1 – perform installs

npm install --save-dev jest malabi

Add the following lines at the top of our app.js file

const malabi = require('malabi');

This tells Malabi to create spans for us (actions we can assert on, for example – a MongoDB query and an HTTP request are 2 examples of possible spans).

It also tells malabi to expose an API endpoint at port 18393 that lets the test runner get these spans and then assert on them.

Step 2- add a new folder & test file: tests/integ.spec.js

const axios = require('axios').default;
const { fetchRemoteTelemetry, clearRemoteTelemetry } = require('malabi');
const getMalabiTelemetryRepository = async () => await fetchRemoteTelemetry({ portOrBaseUrl: 18393 });
describe('testing service-under-test remotely', () => {
 beforeEach(async () => {
   // We must reset all collected spans between tests to make sure span aren't leaking between tests.
   await clearRemoteTelemetry({ portOrBaseUrl: 18393 });
 it('successful /todo request', async () => {
   // simulate login
   const loginRes = await`http://localhost:3000/auth/login`, { username: '', password: 'password' });
   const authCookie = loginRes.headers['set-cookie'];
   // Create a new todo item
   const newTodoRes = await`http://localhost:3000/todos`, {}, { headers: {
       Cookie: authCookie
   } });
   // console.log('newTodoRes', newTodoRes);
   // call to the service under test - internally it will call another API to fetch the todo items.
   await axios(`http://localhost:3000/todos/`, { headers: {
       Cookie: authCookie
   } });
   // Get instrumented spans
   const repo = await getMalabiTelemetryRepository({ portOrBaseUrl: 13893 });
   // This is the span that holds data from the creation of the new todo item
   const postTodoSpan = repo.spans.mongo().first;
   const emailUsedForCreatingNewTodo = JSON.parse(postTodoSpan.dbStatement);
   console.log('emailUsedForCreatingNewTodo', emailUsedForCreatingNewTodo);
   // Assert the email was saved equals the email we logged in with
   // This is the span that holds data from the fetching of todos from mongo for current user
   const getTodosSpan = repo.spans.mongo().second;
   const emailUsedInMongoQuery = JSON.parse(getTodosSpan.dbStatement);
   // Assert the email was used for querying equals the email we logged in with

A few things to note about the above file:

getMalabiTelemetryRepository – fetches spans created in the current run, from the malabi endpoint.

We can then use the received object to query for specific span types, like all Mongo spans, all http requests, etc.

We also have clearRemoteTelemetry that cleans up the in-memory cache of spans between test runs. This helps maintain our test clean & not polluted between test runs.

Notice we have made 2 assertions: one that the post request saved the todo item with the relevant email (the one we logged in with), and the second one verifies that we fetched only todos that belong to that user.

You can now run the test using the “jest” command, and see that everything is working as expected.

As you can see, this is a very simple and clean way we can make sure the connections between our different functions and moving parts are working as expected.

Of course, in production, you would most likely set this up as part of a CI/CD workflow, but this is not the focus of this guide.

That’s basically it, I hope this guide was useful for you, feel free to reach out to me @thetomzach for any questions you may have!

Tom Zach is a Software Engineer at Aspecto. Feel free to follow him on Twitter for more great articles like this one: How to Deploy Jaeger on AWS: a Comprehensive Step-by-Step Guide.

Spread the word

Subscribe for more distributed applications tutorials and insights that will help you boost microservices troubleshooting.