Aspecto blog

On microservices, OpenTelemetry, and anything in between

OpenTelemetry Collector as an Agent on Kubernetes – Part 2

Jack Sparrow OpenTelemetry Collector Kubernetes Agent

Share this post

In this article, you will learn how to deploy and configure the OpenTelemetry Collector as an agent on Kubernetes.

In part 1, we described how to set up a local Kubernetes environment with Minikube. We configured an OpenTelemetry collector and deployed it to a local Kubernetes cluster. We outlined and explained each of the Kubernetes resources we need to deploy the Opentelemetry Collector as a Gateway. 

You can read this article as a stand-alone guide or as part 2 of our OpenTelemetry Collector on Kubernetes series.

What to Expect

  1. Why deploy the collector as an agent
  2. Configuring Kubernetes resources for Collector as agent deployment
  3. Handling common issues
    1. Issue #1 – Use locally built docker inside a Kubernetes cluster
    2. Issue #2 – Cross Namespace Communication in Kubernetes

Related Guides

Why Deploy the OpenTelemetry Collector as an Agent

We use the collector gateway as a central point for all our telemetry data from our distributed architecture. It means the collector usually receives a lot of traffic and therefore is responsible for handling this traffic. 

Deploying specific collector instances for certain services can help us offload some of those responsibilities. For example, the agent can handle actions like batching, compression, retries, and more. 

In addition, it improves the overall performance of the system. Since the agent is running in the same host as our services, there is no DNS resolving (it’s just localhost). Thus it receives the data much faster.

Lastly, having a specific collector instance per service means we can add specific features and configurations to our telemetry data. We can add specific attributes and use any collector process configuration specific to this service.

Here is an illustration of running a collector gateway + agent as sidecar architecture:

An illustration of running an OpenTelemetry collector gateway + agent as sidecar architecture

There are a couple of strategies to deploy the collector as an agent. 

We can deploy it as a DeamonSet, Sidecar, or StatefullSet. Each strategy has its use cases. To read more about deployment strategies, I recommend reading this article.

In this tutorial, I will deploy the OpenTelemetry collector agent as a sidecar.

Kubernetes Resource Configuration for Collector as an Agent

Let’s create a new manifest file named business-application.yml. We will start by creating the agent’s configuration. 

One exporter and one receiver on OTLP protocols:

apiVersion: v1
kind: ConfigMap
 name: otel-agent-conf
 otel-agent-config: |
       endpoint: otel-collector:4317
         insecure: true
         receivers: [otlp]
         processors: []
         exporters: [otlp]

Secondly, we will create a deployment with two containers. The application and the sidecar. 

For the application service, I will use a test service that sends an example span every 5 seconds to the agent’s endpoint. For all code examples, please check out the source code.

I built it using:

docker build -t test-service ./test-service

Here is the deployment:

apiVersion: apps/v1
kind: Deployment
 name: test-service
     app: test-service
       app: test-service
       - name: test-service
         image: test-service
           - name: SERVICE_NAME
             value: test-service
       - name: otel-agent
         image: otel/opentelemetry-collector:latest
           - '/otelcol'
           - '--config=/conf/otel-agent-config.yaml'
           - name: otel-agent-config-vol
             mountPath: /conf
       - configMap:
           name: otel-agent-conf
             - key: otel-agent-config
               path: otel-agent-config.yaml
         name: otel-agent-config-vol

Deploying the OpenTelemetry Collector Gateway + Agent locally

Let’s apply the configuration we have so far to check the new service is working and deploy the agent and gateway to verify the new service is working.

You can find all gateway configurations in this gateway.yml file.

Deploy it using

kubectl apply -f gateway.yml

Next deploy the manifests we just wrote using:

kubectl apply -f business-application.yml

The application crashed.

  • Issue #1 – Use locally built docker inside a Kubernetes cluster. Printing the log from the test-service pod we get the following message:
Defaulted container "test-service" out of: test-service, otel-agent
Error from server (BadRequest): container "test-service" in pod "test-service-5646dd8d77-fp27c" is waiting to start: trying and failing to pull image

The cluster can’t build the image because it can’t pull it. It searches the cluster’s docker daemon for the image and can’t find it. The solution, as described here, is to push the image to the local docker environment of minikube.

Following the documentation, we enter the following, change the imagePullPolicy to Never, and run it again:

eval $(minikube docker-env)

Let’s verify that everything runs smoothly. The logs state that the exporter could not be found. It makes sense because we didn’t create a service to enable this communication.

Add another service resource in the gateway manifest. It needs to be of type ClusterIP and have the gateways labels in its selector block:

apiVersion: v1
kind: Service
 name: otel-collector-in
 namespace: opentelemetry
   app: opentelemetry
   component: otel-collector
 type: ClusterIP
   component: otel-collector
   - name: otlp-grpc # Default endpoint for OpenTelemetry gRPC receiver.
     port: 4317
     protocol: TCP
     targetPort: 4317
   - name: otlp-http # Default endpoint for OpenTelemetry HTTP receiver.
     port: 4318
     protocol: TCP
     targetPort: 4318

Let’s run the application once more. Still no connection with the exporter.

  • Issue #2 – Cross Namespace Communication in Kubernetes. When running pods in different namespaces the pod’s DNS is suffixed with the name of the namespace. In our config map, we need to specify the full DNS in the exporter’s endpoint:
endpoint: otel-collector.opentelemetry.svc.cluster.local:4317

Read the Kubernetes documentation regarding DNS to learn more.

After fixing the last issue and running the cluster once more, we can see the traces reaching the exports we configured in the Gateway. Success!

Final Thoughts

In this article, we configured a simple Collector agent that runs as a sidecar for an example span-generating application. 

We outlined how to deploy and configure the OpenTelemetry Collector as an agent on Kubernetes.

Please feel free to check out the source code for this article. Thanks for reading and please let me know if you have any questions or comments.

The next part of this series will explain how to save us a lot of time and effort by using Helm.

Spread the word

Subscribe for more distributed applications tutorials and insights that will help you boost microservices troubleshooting.