Efficient Kubernetes Event Tracking: A Hands-On Guide to Seamless OpenTelemetry Integration

One of the most important aspects of Kubernetes monitoring is event tracking. It allows you to understand the behavior of your Kubernetes cluster and the applications running on it. They are a great way to understand what is happening in your cluster and how it is performing.

Last week in the #tag-observability channel for the CNCF workspace on slack, we had a great discussion about Kubernetes event tracking. There are many different ways to collect events from Kubernetes, but I noticed that not everyone is aware of the OpenTelemetry solution, so I decided to write this blog post to share how you can collect Kubernetes events using OpenTelemetry.

Well known Solutions for Kubernetes Event Collection

There are many different ways to collect Kubernetes events. The most common ones are:

• Event Router by VMWare
• Event Exporter by CaiCloud
• Kubernetes Event Exporter by Resmo

These solutions are pretty good, but they have some limitations. For example, the Event Router by VMWare is not actively maintained anymore, and Event Exporter by Caicloud doesn’t provide an event filter feature. Regarding the Kubernetes Event Exporter by Resmo, it is a great solution, but I couldn’t find any limitation to it.

Why OpenTelemetry Collector for Kubernetes Event Collection?

For me the Opentelemetry collector is an amazing solution for Kubernetes event collection because it’s a vendor-agnostic solution, it provides a lot of features out of the box, to build an observability pipeline. It’s a CNCF project, so it’s an open-source project, and it’s actively maintained by the community.

How to collect Kubernetes Events using OpenTelemetry Collector?

To collect Kubernetes events using the OpenTelemetry collector, we’re going to leverage an existing OpenTelemetry receiver named Kubernetes Object Receiver , this receiver is responsible for collecting Kubernetes objects, such as events, nodes, and pods.

There are two features that I want to highlight about this receiver:

• It provides a filter feature, so you can filter the events that you want to collect.
• It provides a watch mode so you can avoid pulling the Kubernetes API every X seconds.

So the Opentelemetry configuration to collect Kubernetes events is pretty simple, it should looks like this:

receivers:
  k8sobjects:
    objects:
      - name: event
        mode: watch
        group: events.k8s.io

processors:
  batch:

exporters:
  otlp:
    endpoint: otelcol:4317

service:
  pipelines:
    logs:
      receivers: [k8sobjects]
      processors: [batch]
      exporters: [otlp]

An avalanche of events

Don’t underestimate the number of events that you can collect from your Kubernetes cluster, if you have a big cluster, you can easily collect thousands of events per second, so you need to be careful with the configuration of the OpenTelemetry collector because it can easily overload your cluster.

Once you have the receiver running for a reasonable amount of time, I recommend you look into the data collected understand what matters for you, and drop the events that you don’t need, you can find an example of a filter configuration that you can use to filter events out.

receivers:
  k8sobjects:
    objects:
      - name: event
        mode: watch
        group: events.k8s.io

processors:
  batch:
  filter:
    logs:
      log_record:
        - 'IsMatch(body["reason"], "(Pulling|Pulled)") == true'

exporters:
  otlp:
    endpoint: otelcol:4317

service:
  pipelines:
    logs:
      receivers: [k8sobjects]
      processors: [filter,batch]
      exporters: [otlp]

Conclusion

Using the OpenTelemetry for Kubernetes event collection, you have a great solution to ensure you will not miss any event from your Kubernetes cluster, and you can easily integrate it with any observability pipeline that you have.

I hope you enjoy it, and if you have any questions, feel free to reach out on Twitter . Don’t forget to share this post with your friends and colleagues. 🚀