Thanos Meets OpenTelemetry: Your Guide to Scaling OTLP Metrics.

If you’re familiar with the Prometheus ecosystem, you’ve probably heard of Thanos, a scalable metrics system that can handle the high volume of data generated by modern cloud-native applications.

Thanos is a powerful tool that can help you store, query, and visualize your metrics data efficiently, providing many different features to help you scale your metrics infrastructure, such as:

• Global query view: Query data from multiple Prometheus servers as if they were a single server.
• Long-term storage: Store metrics data for extended periods, even when the original Prometheus server is down.
• High availability: Ensure your metrics data is always available, even in the event of a failure.

Thanos is a great tool for scaling your metrics infrastructure, but what if you’re using OpenTelemetry to collect your metrics keeping the OTLP format? How can you leverage Thanos to scale your OTLP metrics seamlessly?

This was a limitation until the latest release of Thanos v0.38, which introduced the OTLP ingestion support. This new feature allows you to use the Thanos Receiver to ingest OTLP metrics directly, making it easier to scale your metrics infrastructure without having to change your existing setup.

If you’re curious about the implementation details, you can check the PR that introduced the OTLP ingestion support in Thanos.

In this article, we’ll explore how you can leverage the Thanos Receiver to scale your OTLP metrics seamlessly. We’ll cover the following topics:

• Thanos Receiver setup: How to configure the Thanos Receiver to ingest OTLP metrics.
• OpenTelemetry Setup: How to configure OpenTelemetry to send metrics in the OTLP format.
• OpenTelemetry multi-tenancy setup: How to configure OpenTelemetry to send metrics with tenant information.

By the end of this article, you’ll have a step-by-step guide to setting up a modern observability pipeline using Thanos and OpenTelemetry, allowing you to scale your metrics infrastructure seamlessly.

Prerequisites

Before we dive into the setup, let’s make sure you have the necessary tools installed on your machine. You’ll need the following tools:

• kubernetes cluster: You can use kind to create a local Kubernetes cluster or any other Kubernetes cluster.
• kubectl: The Kubernetes command-line tool to interact with your Kubernetes cluster.
• Helm: The package manager for Kubernetes to install the Thanos Receiver.

Thanos Setup

This setup is straightforward, leveraging the Thanos Helm chart to deploy both Thanos Receiver and Thanos Query in your Kubernetes cluster. The Thanos Receiver handles OTLP metric ingestion, while Thanos Query enables querying across stored metric data.

For OTLP ingestion, Thanos Receiver is pre-configured to listen on port 19291, with the OTLP endpoint enabled by default. No additional configuration is required—just run the following command to deploy Thanos Receiver in your Kubernetes cluster.

helm upgrade -i thanos oci://registry-1.docker.io/bitnamicharts/thanos \
  --set global.security.allowInsecureImages=true \
  --set receive.enabled=true \
  --set image.registry=quay.io \
  --set image.repository=thanos/thanos \
  --set image.tag=main-2025-02-19-151ae74

The default configuration of the Thanos Receiver is sufficient to ingest OTLP metrics, but might want to define the following flags to customize the behavior of the Thanos Receiver:

• –receive.otlp-enable-target-info: Enables target information in OTLP metrics ingested by Thanos Receive. When enabled, it converts resource attributes into target metadata, making them available as a target info metric.
• –receive.otlp-promote-resource-attributes: Specifies the resource attributes to be included in OTLP metrics ingested by Thanos Receive. These attributes are mapped to target metadata, enhancing observability and query capabilities.

OpenTelemetry Setup

In this step, we configure OpenTelemetry to send metrics in the OTLP format to the Thanos Receiver. The OpenTelemetry Collector will be used to collect, process, and export your metrics data. We will deploy the Collector using the official Helm chart provided by the OpenTelemetry project.

Configuration

Prepare your values file with the following configuration.

mode: deployment
image:
  repository: otel/opentelemetry-collector-contrib
config:
  exporters:
    otlphttp:
      metrics_endpoint: "http://thanos-receive.default.svc.cluster.local:19291/api/v1/otlp"
      tls:
        insecure: true
  service:
    pipelines:
      metrics:
        exporters:
          - otlphttp

Deployment

Once your configuration is set, install or upgrade the OpenTelemetry Collector with the following Helm command:

helm upgrade -i opentelemetry-collector open-telemetry/opentelemetry-collector --values values.yaml

Checking Metrics Ingestion

To check if your metrics are being ingested correctly, let’s run a query against the Thanos Query through the Thanos Query UI.

kubectl port-forward svc/thanos-query 9090:9090

Open your browser and navigate to http://localhost:9090 to access the Thanos Query UI. Run the following query to check if your metrics are being ingested correctly:

sum(rate(otelcol_receiver_accepted_metric_points_total{job="opentelemetry-collector"}[5m]))

If you see a non-zero value, your metrics are being ingested correctly, and you’re all set!

Conclusion

In this article, we explored how to leverage Thanos Receiver to seamlessly scale OTLP metric ingestion. We covered the setup of Thanos Receiver for ingesting OTLP metrics and the configuration of OpenTelemetry to send metrics in the OTLP format.

This approach aligns with modern observability pipelines, enabling scalable metric infrastructure while integrating with the Prometheus ecosystem, where OTLP ingestion is a stable feature in Prometheus 3.0.

I hope this article has provided a clear, step-by-step guide to setting up a modern observability pipeline using Thanos and OpenTelemetry. If you have any questions or feedback, feel free to reach out on my social media channels.