Aspecto blog

On microservices, OpenTelemetry, and anything in between

OpenTelemetry KafkaJS Instrumentation for Node.js

kafka instrumentation

Share this post

OpenTelemetry is a CNCF project, which, among other things, enables the collection of distributed traces. At Aspecto, we use OpenTelemetry at the core of our product.

While implementing it in our backend, we found a few plugins that were missing, especially when dealing with asynchronous communication.

One of them was KafkaJS.

We took this opportunity to give back to the community and developed it ourselves. This plugin allows you to track all Kafka interactions in your collected traces, which means, you get a more comprehensive view of your application behavior when using Kafka as a message broker.

This article describes this plugin for the kafkajs package.

The kafkajs plugin captures producer and consumer operations and creates spans according to the semantic conventions for Messaging Systems.

  • Each message being produced and consumed is represented by a span with attributes such as messaging.destination(topic name).
  • Context is propagated from producers to consumers. When a message is sent to Kafka, the trace will reveal which services consume it and what other cascading operations happen down the pipe.
  • Batch operations can aggregate multiple messages into a single batch and receive\process them together. This is handled in the plugin according to the specification.
  • The plugin can be extended with hooks, which enable users to run custom logic for adding span attributes depending on the Kafka message.
Simple trace example in Jaeger viewer

The above screenshot shows an example of a producer application named kafka-producer, that exposes an HTTP endpoint (first line), route it in express (second line), and produce two messages to a Kafka topic namedtest, which are then consumed by another application calledkafka-consumer .

As mentioned above, kafkajs was one of the missing plugins we found and as you are reading this we are working to add more plugins.

You can find the OpenTelemetry kafkajs Instrumentation for Node.js here.

Feel free to reach out to us with any questions as we are very much invested in OpenTelemetry and the OpenTelementry community.

Community contributions and involvement are more than welcome ?

As always, this extension (and many others) was developed by Aspecto with ❤️

Spread the word

Subscribe for more distributed applications tutorials and insights that will help you boost microservices troubleshooting.