OpenTelemetry: Revolutionizing Cloud Observability

Introduction

Observability has undergone a tectonic shift over the past several years. What was once a fragmented landscape of proprietary agents, vendor-locked SDKs, and incompatible data formats has coalesced around a single open standard: OpenTelemetry. By early 2026, OpenTelemetry has not only achieved general availability across all three original telemetry signals (traces, metrics, and logs) but has also introduced profiling as its fourth signal, firmly establishing itself as the universal telemetry framework for cloud-native applications.

The project, governed by the Cloud Native Computing Foundation (CNCF), now ranks as the second most active CNCF project by contributor count, trailing only Kubernetes itself. Its adoption has reached a critical inflection point where the question for most engineering organizations is no longer whether to adopt OpenTelemetry, but how to implement it effectively at scale.

This article provides a comprehensive exploration of OpenTelemetry as it stands in 2026. We will dissect its architecture from the ground up, walk through each of its telemetry signals, examine the Collector in granular detail, explore auto-instrumentation capabilities across every major language ecosystem, and address the practical concerns that engineering teams face when operating OpenTelemetry in production. Whether you are evaluating OpenTelemetry for the first time or looking to optimize an existing deployment, this guide covers the full spectrum of what you need to know.

OpenTelemetry Architecture: A Complete Overview

Understanding OpenTelemetry begins with understanding its layered architecture. The project is deliberately designed as a separation of concerns, splitting the instrumentation surface from the data pipeline and from the backend analysis. This separation is what makes vendor neutrality possible and what distinguishes OpenTelemetry from every proprietary observability agent that preceded it.

The API Layer

The API layer is the outermost contract that application code interacts with. It defines the interfaces for creating spans, recording metrics, and emitting log records. Critically, the API layer is designed to be a no-op by default. If an application includes OpenTelemetry API calls but no SDK is configured, those calls simply do nothing. This design choice is essential for library authors: they can instrument their code with OpenTelemetry API calls without forcing any particular telemetry implementation on consumers of their library.

The API layer defines the core abstractions: TracerProvider, MeterProvider, and LoggerProvider. Each provider serves as a factory for creating signal-specific instruments. A TracerProvider produces Tracers, which in turn create Spans. A MeterProvider produces Meters, which create instruments like Counters, Histograms, and Gauges. A LoggerProvider produces Loggers that emit LogRecords.

Every API call is designed around a context-propagation model. Spans carry trace context, and that context flows through the application via explicit or implicit context propagation mechanisms. This ensures that a trace initiated in one service can be correlated with spans created in downstream services, even when those services communicate over HTTP, gRPC, or message queues.

The SDK Layer

The SDK layer is where the actual implementation lives. It provides the concrete implementations of the API interfaces and adds configuration hooks for sampling, resource detection, span processing, and export. When you configure an OpenTelemetry SDK in your application, you are wiring together three things: how telemetry data is created (via the API), how it is processed (via processors and samplers), and where it is sent (via exporters).

The SDK is language-specific. Each language ecosystem (Java, Python, Go, Node.js, .NET, Rust, C++, Ruby, PHP, Swift, and others) has its own SDK implementation that follows the OpenTelemetry specification. The specification is the authoritative document that ensures behavioral consistency across all language implementations. When the specification says that a BatchSpanProcessor must flush spans in batches at a configurable interval, every language SDK must implement that behavior identically.

Resource detection is a critical SDK concern. A Resource in OpenTelemetry terms is a set of attributes that describe the entity producing telemetry. This includes information like service name, service version, deployment environment, cloud provider, region, container ID, Kubernetes pod name, and host information. The SDK provides resource detectors that automatically populate these attributes from the runtime environment. On AWS, for example, the EC2 resource detector pulls instance ID, region, and availability zone. On Kubernetes, the detector pulls pod name, namespace, node name, and deployment name.

The OTLP Protocol

The OpenTelemetry Protocol (OTLP) is the native wire format for transmitting telemetry data. OTLP supports transport over gRPC and HTTP/1.1 with both Protocol Buffers and JSON encodings. By 2026, OTLP has become the de facto standard for telemetry transport, supported natively by virtually every major observability backend.

OTLP defines a unified schema for all four telemetry signals. Trace data is transmitted as ResourceSpans, which contain ScopeSpans, which contain individual Spans. Metric data is transmitted as ResourceMetrics containing ScopeMetrics containing individual metric data points. Log data follows the same nesting pattern with ResourceLogs, ScopeLogs, and LogRecords. This consistent hierarchical structure makes it straightforward to implement receivers, processors, and exporters that handle all signal types uniformly.

The protocol includes built-in support for partial success responses, allowing a backend to acknowledge receipt of some data while signaling failure for other portions. This is essential for high-volume production deployments where occasional partial failures are inevitable and dropping entire batches would be unacceptable.

The OpenTelemetry Collector

The Collector is arguably the most important component in the OpenTelemetry ecosystem. It is a standalone binary that sits between your applications and your observability backends, acting as a telemetry pipeline that can receive data in multiple formats, process it through a chain of transformations, and export it to one or more destinations. We will explore the Collector in extensive detail in its own section below, but it is worth noting here that the Collector is what transforms OpenTelemetry from a mere instrumentation library into a complete telemetry platform.

The Four Pillars of Telemetry

OpenTelemetry originally set out to unify the "three pillars" of observability: traces, metrics, and logs. By 2026, the project has added a fourth signal, profiling, which reached feature-freeze status in late 2025 and has been rapidly adopted in production environments. Understanding each signal, its data model, and its correlation capabilities is fundamental to effective OpenTelemetry usage.

Distributed Traces

Traces remain the flagship signal of OpenTelemetry and the most mature part of the project. A distributed trace represents the complete journey of a request through a system, from the moment it enters through an API gateway to every downstream service call, database query, cache lookup, and message queue interaction that the request triggers.

The data model for traces is built around Spans. A Span represents a single unit of work within a trace. Each Span has a trace ID (shared by all spans in the same trace), a span ID (unique to this span), a parent span ID (linking it to the calling span), a name, a start and end timestamp, a status, a set of attributes (key-value pairs), a list of events (timestamped annotations), and a list of links (references to spans in other traces).

Span attributes are where the real diagnostic power lives. By attaching attributes like http.method, http.status_code, db.system, db.statement, rpc.service, and custom business attributes, each span becomes a rich context window into what happened during that unit of work. The OpenTelemetry semantic conventions define standardized attribute names across dozens of domains including HTTP, database, messaging, RPC, cloud resources, and more. In 2026, the semantic conventions have reached stability for most major domains, meaning attribute names are guaranteed not to change in backward-incompatible ways.

Span events are particularly useful for recording exceptions. When an error occurs within a span, the SDK automatically records an exception event that includes the exception type, message, and stack trace. This eliminates the need for separate error-tracking systems in many cases, as the trace itself carries full exception context.

Span links allow connecting spans across trace boundaries. A common use case is message queue processing: the span that consumes a message can link back to the span that produced it, even though they belong to different traces. This enables end-to-end visibility through asynchronous processing pipelines.

Metrics

The metrics signal in OpenTelemetry went through a long and sometimes contentious design process, but the result is a sophisticated system that supports both the "push" model (where applications emit metrics to a Collector) and compatibility with the "pull" model (where a system like Prometheus scrapes metric endpoints).

OpenTelemetry defines several metric instrument types. Counters are monotonically increasing values, perfect for counting requests, errors, or bytes transferred. UpDownCounters can increase or decrease, suitable for tracking active connections or queue depth. Histograms capture the distribution of values, essential for latency measurements. Gauges record instantaneous values like CPU usage or memory consumption.

Each metric data point carries attributes (sometimes called labels or dimensions), a timestamp, and the metric value itself. Metrics can be aggregated temporally as either delta or cumulative. Delta temporality reports the change since the last data point, while cumulative temporality reports the running total since the process started. The choice between delta and cumulative has significant implications for backend compatibility. Prometheus traditionally expects cumulative counters, while systems like Datadog prefer delta. The OpenTelemetry Collector can perform temporality conversion, smoothing over these incompatibilities.

A key architectural decision in OpenTelemetry metrics is the separation between the metric instrument (what the application code interacts with) and the metric reader (what aggregates and exports the data). This separation allows a single instrument to be read by multiple readers with different aggregation settings, enabling the same metric data to be exported to different backends with different requirements.

Histogram boundaries are a notable concern. OpenTelemetry provides default histogram bucket boundaries, but in practice, you often need to customize these for your specific latency distributions. A service that typically responds in 1-5 milliseconds needs very different histogram buckets than one that responds in 100-500 milliseconds. The ExponentialHistogram instrument type, stable since mid-2025, addresses this by dynamically adjusting bucket boundaries based on observed values, providing high-resolution distributions without manual configuration.

Logs

Logs were the last of the original three signals to reach stability in OpenTelemetry, and for good reason. The logging landscape is vastly more diverse than tracing or metrics. Every language has its own logging frameworks (Log4j, SLF4J, Python logging, Winston, Serilog, and dozens more), and organizations have decades of investment in existing log pipelines. OpenTelemetry could not simply replace all of this; instead, it had to integrate with it.

The OpenTelemetry approach to logs is built around the concept of a Log Bridge. Rather than replacing your existing logging framework, you install a bridge that intercepts log records from your existing logger and enriches them with OpenTelemetry context before forwarding them to the OpenTelemetry pipeline. This means your application code continues to use logger.info("Processing order") as it always has, but behind the scenes, each log record is annotated with the active trace ID and span ID, enabling direct correlation between logs and traces.

The LogRecord data model includes a timestamp, a severity level, a body (the log message), attributes, the trace ID and span ID from the active context, and resource information. This is a superset of what most existing log formats provide, and the additional context is what makes logs genuinely useful in a distributed tracing context. When you find an interesting span in your trace view, you can immediately jump to the exact logs that were emitted during that span's execution.

The OpenTelemetry Collector includes a rich set of log processors. The transform processor can parse unstructured log messages into structured attributes using regex or JSON parsing. The filter processor can drop logs below a certain severity level. The attributes processor can add, rename, or remove attributes. Combined, these processors allow the Collector to serve as a full log processing pipeline, replacing dedicated log shippers like Fluentd or Logstash in many deployments.

Profiling: The Fourth Signal

The addition of profiling as a fourth OpenTelemetry signal represents a significant expansion of the project's scope. Profiling captures where an application spends its CPU time, how memory is allocated, what locks are being contended, and which code paths are hot. Unlike traces, which capture request-level flow, profiling captures resource-level consumption at the function or line level.

The profiling signal in OpenTelemetry uses a data model based on pprof, the profile format originally developed for Go but now widely used across the industry. OpenTelemetry profiles are correlated with traces via the same context propagation mechanism: a profile sample captured during the execution of a particular span can be linked back to that span. This enables a powerful debugging workflow where you identify a slow span in your trace, then drill down into the profile data to see exactly which function calls consumed the most time within that span.

The profiling signal is particularly powerful when combined with continuous profiling. Rather than capturing profiles only during manual debugging sessions, continuous profiling captures low-overhead samples constantly in production. When a performance issue arises, the historical profile data is already available for analysis without needing to reproduce the problem. Services like Grafana Pyroscope, Datadog Continuous Profiler, and Polar Signals have all announced or shipped OpenTelemetry profiling support.

The OpenTelemetry Collector Deep Dive

The Collector deserves a thorough examination because it is the component where most of the operational complexity and optimization opportunity lives. In production, your Collector configuration determines your telemetry fidelity, your pipeline reliability, and your observability costs.

Collector Architecture

The Collector is built around a pipeline model. A pipeline consists of one or more receivers, zero or more processors, and one or more exporters. You can define multiple pipelines in a single Collector instance, and each pipeline handles a specific signal type (traces, metrics, logs, or profiles). Pipelines can share receivers and exporters, but each pipeline has its own processor chain.

The Collector binary comes in two official distributions. The Core distribution includes only the most essential, stable components. The Contrib distribution includes everything in Core plus hundreds of additional receivers, processors, and exporters contributed by the community and vendor partners. In practice, many organizations build custom Collector distributions using the OpenTelemetry Collector Builder (ocb), which generates a Collector binary containing only the components you actually need. This reduces binary size, attack surface, and startup time.

Receivers

Receivers are the entry points for data into the Collector. The OTLP receiver is the most common, accepting data over gRPC (default port 4317) and HTTP (default port 4318). But the Collector supports dozens of other receivers for compatibility with existing telemetry sources:

The Jaeger receiver accepts data in Jaeger format, enabling migration from Jaeger to OpenTelemetry without changing application instrumentation. The Zipkin receiver serves the same purpose for Zipkin users. The Prometheus receiver can scrape Prometheus metric endpoints, allowing the Collector to replace a Prometheus server for metric collection. The hostmetrics receiver collects system-level metrics (CPU, memory, disk, network) from the host where the Collector runs. The filelog receiver tails log files, replacing Fluentd or Filebeat for log collection. The kafka receiver consumes telemetry data from Kafka topics, enabling buffered ingestion patterns. The statsd receiver accepts StatsD metrics, providing a migration path from StatsD-based monitoring.

In 2026, several newer receivers have gained traction. The sqlquery receiver executes SQL queries against databases at configurable intervals and converts the results into metrics, useful for monitoring database-specific KPIs. The kubeletstats receiver pulls container and pod metrics directly from the Kubelet API. The k8s_cluster receiver monitors Kubernetes cluster-level resources like deployments, replica sets, and jobs. The awsxray receiver accepts AWS X-Ray trace data, facilitating migration from X-Ray to OpenTelemetry.

Processors

Processors transform data as it flows through the pipeline. They are applied in order, and the sequence matters. Common processors include:

The batch processor is nearly universal in production deployments. It accumulates data into batches before forwarding them to exporters, reducing the number of outgoing network requests and improving throughput. You configure batch size limits (number of items or bytes) and timeout intervals. A typical configuration batches up to 8192 spans or 200 milliseconds, whichever comes first.

The memory_limiter processor prevents the Collector from consuming unbounded memory. It monitors the Collector's heap usage and starts dropping data when usage exceeds a soft limit, aggressively dropping when it approaches a hard limit. In production, this processor is essential for preventing Collector crashes during traffic spikes.

The filter processor drops data that matches specified conditions. You can filter by attribute values, resource attributes, span names, metric names, log severity, and more. This is a primary tool for cost control: by filtering out verbose debug logs, low-value metrics, or health-check traces before they reach your backend, you reduce storage costs without losing important data.

The attributes processor adds, updates, or deletes attributes on spans, metrics, and logs. A common use case is adding environment information (like deployment region or cluster name) to all telemetry data passing through the Collector.

The transform processor provides a full expression language (OTTL, the OpenTelemetry Transformation Language) for complex data manipulation. OTTL can modify attribute values, convert data types, extract substrings, perform math operations, and conditionally apply transformations. In 2026, OTTL has matured significantly with support for functions like replace_pattern, merge_maps, limit, and truncate_all.

The tail_sampling processor is one of the most powerful and complex. Unlike head-based sampling (which makes the sampling decision at trace creation time), tail-based sampling waits until a trace is complete and then decides whether to keep it based on the full trace data. This allows policies like "keep all traces with errors," "keep all traces longer than 2 seconds," or "keep all traces that hit the payment service." We will explore sampling strategies in detail later.

The groupbyattrs processor reorganizes telemetry data by regrouping spans or metrics based on specified attributes. This is useful when data arrives from multiple sources with different resource attribute sets and you need to normalize them before export.

Exporters

Exporters send data to backends. The OTLP exporter sends data in OTLP format and is the default choice for backends that support it. The debug exporter (formerly the logging exporter) writes data to the Collector's stdout, useful during development and troubleshooting. The file exporter writes data to local files, useful for archiving or debugging.

Vendor-specific exporters include the datadog exporter, the splunk_hec exporter, the awsxray exporter, the azuremonitor exporter, the googlecloud exporter, the dynatrace exporter, and the elasticsearch exporter. Each translates OpenTelemetry data into the vendor's native format and transmits it using the vendor's ingestion API.

The loadbalancing exporter is noteworthy for scaled deployments. When running tail-based sampling, all spans belonging to the same trace must be processed by the same Collector instance to make an informed sampling decision. The loadbalancing exporter distributes traces across a pool of Collector instances using consistent hashing on the trace ID, ensuring this colocation.

Pipeline Configuration Example

A production Collector configuration typically looks something like this in YAML. You define receivers that accept OTLP data, processors that batch, limit memory, filter, and transform, and exporters that send data to your chosen backend. The service section wires these components into named pipelines. You might have a traces pipeline that passes through the batch and tail_sampling processors, a metrics pipeline with batch and filter processors, and a logs pipeline with transform and filter processors, each routing to different exporters.

Auto-Instrumentation: Zero-Code Observability

One of OpenTelemetry's most compelling features is auto-instrumentation, the ability to add tracing, metrics, and logging to an application without modifying its source code. Auto-instrumentation works by hooking into the language runtime or popular libraries to automatically create spans for inbound and outbound requests, record metrics for request duration and error rates, and propagate trace context across service boundaries.

Java Auto-Instrumentation

Java's auto-instrumentation is the most mature in the OpenTelemetry ecosystem. The Java agent (opentelemetry-javaagent) attaches to the JVM at startup via the -javaagent flag and uses bytecode manipulation to instrument over 150 libraries and frameworks. This includes HTTP clients (Apache HttpClient, OkHttp, Java 11+ HttpClient), web frameworks (Spring MVC, Spring WebFlux, JAX-RS, Servlet API), database clients (JDBC, Hibernate, R2DBC), messaging systems (Kafka, RabbitMQ, AWS SQS), RPC frameworks (gRPC, Apache Dubbo), and many more.

The Java agent is highly configurable via environment variables or system properties. You can disable specific instrumentations, add custom resource attributes, configure sampling, set the export endpoint, and adjust batch processor settings, all without touching application code. This makes it practical to roll out OpenTelemetry across large Java microservice fleets by simply adding the agent to the deployment configuration.

In 2026, the Java agent has added support for virtual threads (Project Loom), ensuring that context propagation works correctly across virtual thread boundaries. It also includes experimental support for the profiling signal, capturing CPU profiles correlated with trace spans.

Python Auto-Instrumentation

Python auto-instrumentation works through the opentelemetry-instrument command, which wraps your Python application at startup and monkey-patches popular libraries. Supported libraries include Flask, Django, FastAPI, requests, urllib3, aiohttp, SQLAlchemy, psycopg2, redis-py, celery, boto3, and many others.

Python's dynamic nature makes auto-instrumentation somewhat simpler than Java's bytecode manipulation, but it also introduces some challenges. Monkey-patching can occasionally conflict with other libraries that also modify the same functions. The Python auto-instrumentation project has been proactive about detecting and resolving these conflicts, and by 2026, the most common library combinations work reliably out of the box.

For async Python applications (which are increasingly common in 2026), the auto-instrumentation correctly handles asyncio context propagation, ensuring that spans created in one coroutine are properly parented to spans in the calling coroutine.

Node.js Auto-Instrumentation

Node.js auto-instrumentation leverages the --require flag or the --loader flag (for ESM modules) to load instrumentation packages before the application code runs. The @opentelemetry/auto-instrumentations-node metapackage bundles instrumentations for Express, Koa, Fastify, Hapi, the built-in http and https modules, pg (PostgreSQL), mysql2, mongodb, redis, ioredis, aws-sdk, graphql, and many others.

A significant development in the Node.js ecosystem during 2025-2026 has been improved ESM (ECMAScript Modules) support. Early versions of OpenTelemetry Node.js instrumentation worked only with CommonJS modules. With Node.js increasingly adopting ESM as the default module system, the instrumentation libraries have been updated to use the Node.js loader hooks API, enabling auto-instrumentation for ESM-based applications.

.NET Auto-Instrumentation

The .NET auto-instrumentation agent hooks into the CLR profiler API to intercept method calls in ASP.NET Core, HttpClient, SqlClient, Entity Framework Core, gRPC, and other common libraries. It supports both .NET 6 and later versions and the older .NET Framework 4.6.2 and above.

The .NET instrumentation has benefited from close collaboration between the OpenTelemetry project and Microsoft. The System.Diagnostics.Activity API in .NET, which predates OpenTelemetry, has been aligned with OpenTelemetry semantics so that spans created via Activity are automatically captured by the OpenTelemetry SDK. This means many .NET libraries that already use Activity for diagnostics get OpenTelemetry support "for free."

Go Instrumentation

Go does not support the kind of runtime bytecode manipulation that enables true auto-instrumentation in Java or .NET. Instead, Go instrumentation relies on explicit library wrappers. OpenTelemetry provides instrumentation packages for net/http, database/sql, gRPC, Gin, Echo, Fiber, AWS SDK, and other popular libraries. You wrap your HTTP handlers, database connections, or gRPC clients with the instrumentation packages to get automatic span creation.

While this requires code changes, the changes are typically minimal and mechanical. Wrapping an http.Handler with otelhttp.NewHandler or a database/sql driver with otelsql.Open adds one or two lines of code per integration point. In 2026, the Go ecosystem has also seen the emergence of compile-time instrumentation tools that use Go's AST manipulation capabilities to automatically add OpenTelemetry wrappers during the build process, approaching the convenience of true auto-instrumentation.

OpenTelemetry for Kubernetes

Kubernetes is the deployment platform for the majority of OpenTelemetry users, and the project has developed extensive Kubernetes-specific tooling.

The OpenTelemetry Operator

The OpenTelemetry Operator is a Kubernetes operator that manages Collector instances and auto-instrumentation injection. You deploy it once to your cluster, and it watches for custom resources that define Collector configurations and instrumentation rules.

For Collector management, the Operator supports three deployment modes. Deployment mode runs the Collector as a standard Kubernetes Deployment, suitable for centralized collection patterns where a pool of Collector pods receives data from all applications. DaemonSet mode runs a Collector pod on every node, suitable for collecting node-level metrics and logs. Sidecar mode injects a Collector container into every application pod, suitable for per-application processing before centralized collection. StatefulSet mode, added in 2025, supports the tail-sampling use case where Collector instances need stable identities for consistent hashing.

For auto-instrumentation, the Operator watches for an Instrumentation custom resource that specifies which language runtimes to instrument and how. When a pod is annotated with instrumentation.opentelemetry.io/inject-java: "true", the Operator's webhook automatically modifies the pod spec to include the Java agent, set the necessary environment variables, and configure the agent to export to the Collector. This works for Java, Python, Node.js, .NET, and Go (with the eBPF-based approach described below), enabling cluster-wide auto-instrumentation without modifying any application deployment manifests beyond adding a single annotation.

Resource Detection and Enrichment

When running on Kubernetes, the Collector and application SDKs can automatically detect and attach Kubernetes-specific resource attributes. The k8sattributes processor in the Collector queries the Kubernetes API to resolve pod IP addresses to pod names, namespaces, node names, deployment names, replica set names, and labels. This enrichment happens at the Collector level, meaning applications do not need any Kubernetes awareness in their code.

The resourcedetection processor in the Collector detects the underlying cloud environment (AWS, GCP, Azure) and attaches cloud-specific attributes like region, availability zone, account ID, and instance type. Combined with Kubernetes attributes, this provides a complete picture of where every piece of telemetry originated: from the specific function in the code, through the pod, node, deployment, namespace, cluster, and cloud region.

Cluster-Level Metrics

The k8s_cluster receiver monitors Kubernetes API resources and generates metrics about cluster state: the number of desired vs. available replicas for each deployment, the status of each pod (Running, Pending, Failed), node conditions (Ready, MemoryPressure, DiskPressure), resource requests and limits, and more. Combined with the kubeletstats receiver (which pulls per-container CPU, memory, filesystem, and network metrics from the Kubelet), you get complete Kubernetes observability through the OpenTelemetry pipeline without needing separate tools like kube-state-metrics or cAdvisor.

Vendor Ecosystem and Backend Support

One of the strongest arguments for adopting OpenTelemetry is the breadth of vendor support. By 2026, every major observability vendor not only accepts OpenTelemetry data but actively recommends it as the preferred instrumentation path.

Datadog

Datadog has invested heavily in OpenTelemetry support. Their backend natively ingests OTLP traces, metrics, and logs. The Datadog Agent itself now includes an embedded OpenTelemetry Collector, allowing customers to use OTLP as their instrumentation protocol while still benefiting from Datadog-specific features like Live Processes, Network Performance Monitoring, and Application Security Management. Datadog's OTLP ingestion preserves the full fidelity of OpenTelemetry semantic conventions and automatically maps them to Datadog's native tag system.

For organizations migrating from the Datadog Agent's proprietary tracing libraries (dd-trace) to OpenTelemetry, Datadog provides a migration guide and compatibility layer that translates dd-trace context headers to W3C Trace Context, enabling gradual migration where some services use dd-trace and others use OpenTelemetry.

Grafana Labs

Grafana Labs has positioned itself as the most OpenTelemetry-native commercial vendor. Their entire observability stack (Grafana Tempo for traces, Grafana Mimir for metrics, Grafana Loki for logs, Grafana Pyroscope for profiles) natively ingests OTLP. The Grafana Alloy agent (formerly Grafana Agent) is built on the OpenTelemetry Collector framework, sharing the same receiver, processor, and exporter architecture while adding Grafana-specific components.

Grafana Cloud's free tier includes generous OTLP ingestion limits, making it an accessible option for getting started with OpenTelemetry without any upfront cost. The integration between Grafana dashboards and OpenTelemetry data is seamless: you can jump from a trace to its correlated logs to the relevant metric dashboards without leaving the Grafana UI.

Honeycomb

Honeycomb was an early and vocal advocate for OpenTelemetry and has built its entire product around high-cardinality trace analysis. Honeycomb's backend is designed specifically for the rich, wide events that OpenTelemetry produces, and their query engine excels at ad-hoc exploration of trace data with arbitrary attribute combinations. Honeycomb encourages users to instrument exclusively with OpenTelemetry and has deprecated its older proprietary SDKs.

New Relic, Splunk, and Dynatrace

New Relic accepts OTLP data natively on a dedicated endpoint and maps OpenTelemetry semantic conventions to its entity model. Splunk Observability Cloud (formerly SignalFx) is built on OpenTelemetry, with the Splunk Distribution of the OpenTelemetry Collector as its primary data collection agent. Dynatrace's OneAgent can coexist with OpenTelemetry instrumentation and ingests OTLP data alongside its proprietary telemetry, providing a migration path for organizations that want to gradually transition to open-source instrumentation.

The convergence around OTLP has created a genuinely competitive market for observability backends. Organizations can evaluate backends based on query capabilities, alerting features, cost, and user experience rather than on SDK compatibility or data format support. Switching backends is no longer a multi-month instrumentation project; it is a configuration change in the Collector.

Sampling Strategies

In production systems that handle millions of requests per second, collecting and storing a trace for every single request is neither practical nor necessary. Sampling is the mechanism by which you decide which traces to keep and which to discard. OpenTelemetry supports multiple sampling strategies, each with different trade-offs between fidelity, cost, and complexity.

Head-Based Sampling

Head-based sampling makes the sampling decision at the very beginning of a trace, before any spans have been created. The decision is typically probabilistic: a TraceIdRatioBased sampler configured at 10% will keep roughly one out of every ten traces, determined by hashing the trace ID. The decision is propagated to all downstream services via the W3C Trace Context sampled flag, so all services agree on whether to record spans for a given trace.

Head-based sampling is simple, predictable, and low-overhead. It requires no coordination between services and no buffering of span data. However, it is blind to the content of the trace. A sampler that keeps 10% of traces will discard 90% of error traces along with 90% of success traces. For systems where errors are rare, this means you might miss most of the interesting traces.

Tail-Based Sampling

Tail-based sampling defers the sampling decision until the trace is complete (or nearly complete). The Collector buffers incoming spans, groups them by trace ID, waits for the trace to finish (determined by a configurable timeout), and then evaluates sampling policies against the complete trace. Policies can include: keep all traces with error spans, keep all traces with latency above a threshold, keep all traces that include specific services, keep a random percentage of remaining traces.

The tail_sampling processor in the OpenTelemetry Collector implements this strategy. In a typical configuration, you define a composite policy that says "always keep error traces, always keep slow traces (over 2 seconds), and probabilistically sample 5% of everything else." This ensures that you never miss important traces while keeping costs manageable.

Tail-based sampling introduces operational complexity. All spans for a given trace must be routed to the same Collector instance, requiring consistent-hash-based load balancing. The Collector must buffer spans in memory while waiting for traces to complete, which requires careful memory management. Incomplete traces (where some spans are lost or delayed) can cause incorrect sampling decisions. Despite these challenges, tail-based sampling has become the standard approach for production OpenTelemetry deployments in 2026 because its ability to prioritize important traces is so valuable.

Adaptive Sampling

Adaptive sampling dynamically adjusts sampling rates based on current system conditions. During normal operation, a low sampling rate (say 1%) is sufficient. When error rates spike or latency increases, the sampling rate automatically increases to capture more diagnostic data. When the system returns to normal, the rate decreases again.

Several approaches to adaptive sampling have emerged. The probabilistic_sampler processor in the Collector can be combined with external rate-adjustment logic. Some vendors provide proprietary adaptive sampling that adjusts rates based on backend ingestion capacity. The OpenTelemetry community has been working on a standardized adaptive sampling specification, though it has not yet reached stability.

Combining Sampling Strategies

In practice, most organizations use a combination of sampling strategies. The application SDK might apply a coarse head-based sampler (keeping 50% of traces) to reduce the data volume before it even reaches the Collector. The Collector then applies tail-based sampling to make intelligent decisions about which of those traces to keep. This layered approach reduces Collector memory pressure while still enabling policy-based sampling.

Context Propagation

Context propagation is the mechanism by which trace context flows across service boundaries. Without it, you would have isolated spans in each service with no way to stitch them together into distributed traces. OpenTelemetry supports multiple propagation formats, with W3C Trace Context as the default.

W3C Trace Context

The W3C Trace Context specification defines two HTTP headers: traceparent and tracestate. The traceparent header carries the trace ID, parent span ID, and trace flags (including the sampled flag) in a compact binary-encoded format. The tracestate header carries vendor-specific context as a list of key-value pairs. When Service A calls Service B over HTTP, the OpenTelemetry SDK automatically injects these headers into the outgoing request. Service B's SDK extracts the headers and creates a child span linked to the parent span in Service A.

W3C Trace Context is now the default propagation format in OpenTelemetry and is supported by all major vendors and frameworks. It has largely replaced the earlier propagation formats (B3, Jaeger, and X-Ray) that were specific to individual tracing systems, though OpenTelemetry continues to support those formats for backward compatibility.

Baggage

Baggage is a separate propagation mechanism that allows you to attach arbitrary key-value pairs to the context that are propagated across service boundaries. Unlike trace context (which carries trace-specific identifiers), baggage carries application-level data. Common use cases include propagating a user ID, a tenant ID, a feature flag set, or a request priority level across all services in a request path.

Baggage is propagated via the baggage HTTP header. Each service can read baggage values and use them for logging, metric attribution, or routing decisions. However, baggage should be used judiciously because every baggage item adds overhead to every cross-service call. Large baggage payloads can noticeably increase request latency.

Cross-Service Correlation

The combination of trace context propagation and consistent resource attribution enables powerful cross-service correlation. When investigating a production issue, you can start from a user-facing error, follow the trace through every service it touched, examine the logs and metrics associated with each span, and identify the root cause in a downstream dependency. This end-to-end visibility is what makes distributed tracing genuinely transformative for operational efficiency.

Context propagation also works across messaging systems. When a producer publishes a message to Kafka, RabbitMQ, or AWS SQS, the OpenTelemetry instrumentation injects trace context into the message headers. When the consumer processes the message, the instrumentation extracts the context and creates a span linked to the producer's trace. This works seamlessly with the auto-instrumentation for most popular messaging libraries.

OpenTelemetry and eBPF Integration

One of the most exciting developments in the OpenTelemetry ecosystem in 2025-2026 has been the integration with eBPF (Extended Berkeley Packet Filter) technology. eBPF allows running sandboxed programs in the Linux kernel, enabling deep system observation without modifying application code or even attaching SDK agents.

Grafana Beyla

Grafana Beyla is the most prominent eBPF-based OpenTelemetry instrumentation tool. It runs as a standalone process (or Kubernetes DaemonSet) that uses eBPF probes to observe network traffic and function calls in running applications. Beyla can automatically detect HTTP and gRPC requests, extract request metadata (method, path, status code, duration), and generate OpenTelemetry traces and metrics without any application modification.

Beyla's approach is fundamentally different from traditional auto-instrumentation. Rather than hooking into library code via bytecode manipulation or monkey-patching, it observes system calls and network packets at the kernel level. This means it works with any programming language and any framework, including compiled languages like Go and Rust where traditional auto-instrumentation is difficult or impossible.

The trade-off is fidelity. eBPF-based instrumentation can capture network-level events but cannot see application-internal details like database query parameters, custom business attributes, or internal function call hierarchies. It provides "outside-in" observability, excellent for service-to-service communication patterns but limited for deep application-level debugging.

Use Cases for eBPF-Based Instrumentation

eBPF instrumentation excels in several scenarios. For legacy applications where modifying code or attaching agents is impractical, eBPF provides immediate observability. For polyglot environments with dozens of languages and frameworks, eBPF provides consistent instrumentation without maintaining language-specific configurations. For security-sensitive environments where adding agents to application processes is prohibited, eBPF runs in a separate process with kernel-level access. For service mesh environments, eBPF can capture the same service-to-service metrics that a sidecar proxy would, but with lower resource overhead.

In 2026, the OpenTelemetry project has begun integrating eBPF-based instrumentation into the Operator. An annotation like instrumentation.opentelemetry.io/inject-go: "true" on a Go application pod now triggers eBPF-based instrumentation via Beyla rather than SDK-based instrumentation, acknowledging that eBPF is the most practical approach for languages without runtime manipulation capabilities.

Migrating from Proprietary Agents to OpenTelemetry

Many organizations considering OpenTelemetry are not starting from scratch. They have existing observability infrastructure built on proprietary agents, and the migration path needs to be incremental to avoid disruption.

A Step-by-Step Migration Approach

The recommended migration pattern has five phases.

In the first phase, you deploy the OpenTelemetry Collector alongside your existing agents. The Collector is configured to receive data from your current agents using compatibility receivers (Jaeger receiver, Zipkin receiver, StatsD receiver, Prometheus receiver) and export to your existing backend. This validates that the Collector works in your environment without changing any application instrumentation.

In the second phase, you configure the Collector to dual-export: sending data to both your existing backend and a new OTLP-compatible backend. This lets you compare data fidelity between the two backends and validate that the Collector's processing pipeline preserves the data you care about.

In the third phase, you begin migrating application instrumentation. Start with new services, which can use OpenTelemetry from the start. For existing services, begin with auto-instrumentation, which requires minimal code changes. Add the OpenTelemetry agent to your deployment configuration and configure it to export via OTLP to the Collector.

In the fourth phase, you migrate the remaining services and remove the compatibility receivers from the Collector. At this point, all telemetry flows through the OTLP path.

In the fifth phase, you decommission the old agents and finalize the transition. This is also when you can optimize Collector pipelines, implement tail-based sampling, and fine-tune attribute processing now that all data flows through OpenTelemetry.

Specific Migration: Datadog Agent to OpenTelemetry Collector

For organizations on Datadog, the migration is smoother than most because the Datadog Agent itself includes an OTLP receiver. You can configure applications to export OTLP to the Datadog Agent, which handles the translation to Datadog's native format. This is useful as an intermediate step, but for full flexibility, you will eventually want to replace the Datadog Agent with the OpenTelemetry Collector using the Datadog exporter.

The key steps are: first, enable the OTLP receiver in the Datadog Agent's configuration. Second, instrument applications with OpenTelemetry and point them at the Datadog Agent's OTLP endpoint. Third, validate that traces, metrics, and logs appear correctly in Datadog. Fourth, deploy the OpenTelemetry Collector with the Datadog exporter and gradually shift traffic from the Datadog Agent to the Collector. Fifth, remove the Datadog Agent once all telemetry flows through the Collector.

The Datadog exporter in the Collector maps OpenTelemetry semantic conventions to Datadog tags, converts OTLP metrics to Datadog metric format (handling temporality conversion), and supports Datadog-specific features like resource naming and service inference. The mapping is well-documented and handles most common scenarios automatically.

Cost Management with OpenTelemetry

Observability costs are a growing concern for engineering organizations. As systems become more complex and instrumentation becomes more pervasive, the volume of telemetry data can grow explosively. OpenTelemetry's architecture provides multiple levers for controlling telemetry volume and cost.

Controlling Volume at the Source

The first opportunity for cost control is at the SDK level. By configuring appropriate sampling rates, you reduce the volume of data before it leaves the application process. Attribute limits (maximum number of attributes per span, maximum attribute value length) prevent individual spans from becoming excessively large. Span limits (maximum number of events per span, maximum number of links per span) similarly cap span size.

Filtering in the Collector

The Collector's filter processor is the primary tool for cost management. Common filtering patterns include dropping health-check and readiness-probe traces (which are high-volume and low-value), dropping debug-level logs in production, dropping metrics with known-uninteresting label combinations, and dropping spans for internal service-to-service calls that are well-understood.

The transform processor can reduce data size by truncating long attribute values, removing redundant attributes, and condensing verbose span names. The attributes processor can hash or redact sensitive data before it reaches the backend, serving both cost and compliance purposes.

Aggregation

For metrics, the Collector can aggregate data before export. The metricstransform processor can combine multiple metrics into summary metrics, reducing the number of distinct time series. The cumulativetodelta and deltatocumulative processors convert between temporality modes to match backend expectations, preventing unnecessary data duplication.

For logs, the Collector can aggregate similar log records into counts. Instead of storing thousands of identical "Connection pool exhausted" log messages, the Collector can store a single record with a count attribute. This pattern dramatically reduces log storage costs for repetitive messages.

Multi-Destination Routing

The Collector's ability to export to multiple backends enables cost-optimized routing. You might send all traces to a high-performance, high-cost backend for real-time analysis while simultaneously sending sampled traces to a low-cost object storage backend for long-term retention. You might send critical metrics to a managed Prometheus service while sending verbose debug metrics to a self-hosted Mimir instance. This flexibility is one of the Collector's strongest selling points for cost-conscious organizations.

OpenTelemetry for Frontend

While OpenTelemetry originated in backend and infrastructure observability, its reach has extended to frontend applications. The OpenTelemetry JavaScript SDK includes a browser distribution that enables tracing and metric collection in web browsers, providing end-to-end visibility from user interaction to backend processing.

Browser Instrumentation

The @opentelemetry/sdk-trace-web package provides the core tracing SDK for browser environments. Combined with auto-instrumentations for XMLHttpRequest, the Fetch API, and document load events, you can automatically trace every API call made by a web application and correlate it with the backend trace that handles the request.

Browser instrumentation captures several categories of data. Document load traces include spans for DNS lookup, TCP connection, TLS handshake, request, response, and DOM processing. This provides visibility into page load performance broken down by phase. User interaction traces capture the time from a user click or form submission to the completion of the resulting API call and DOM update. Resource timing traces capture the loading of individual assets (scripts, stylesheets, images) using the Resource Timing API.

User Session Correlation

A particularly valuable capability is correlating frontend traces with user sessions. By attaching a session ID attribute to all browser-side spans, you can query your trace backend for all traces associated with a specific user session. This enables powerful debugging workflows: a user reports a problem, you look up their session ID, and you see every API call they made, how long each took, whether any failed, and what happened in the backend for each call.

The session ID can be propagated to the backend as a baggage item, ensuring that all backend spans associated with a user session carry the same session identifier. This enables cross-cutting queries like "show me all database queries executed during session X" or "show me the error rate for all API calls from session Y."

Challenges

Frontend instrumentation faces unique challenges. Browser environments are resource-constrained, so the SDK must be lightweight and avoid blocking the main thread. Cross-origin requests require CORS headers to permit the trace context headers. Single-page applications need special handling for route changes and view transitions. Privacy regulations may restrict what data can be collected in the browser. Despite these challenges, frontend OpenTelemetry instrumentation has matured significantly and is increasingly deployed alongside real user monitoring (RUM) solutions.

Production Best Practices

Running OpenTelemetry in production at scale requires attention to several operational concerns. The difference between a proof-of-concept deployment and a production-grade deployment lies in how well you handle failure modes, resource constraints, and operational visibility.

Collector Scaling

A single Collector instance can typically handle tens of thousands of spans per second, but large-scale deployments require multiple instances. The scaling pattern depends on your deployment model.

For gateway deployments (where Collectors receive data from applications over the network), you scale horizontally behind a load balancer. If you are using tail-based sampling, the load balancer must use consistent hashing on the trace ID to ensure all spans for a trace reach the same Collector instance. The loadbalancing exporter handles this when you have a two-tier Collector architecture: front-tier Collectors receive data from applications and use the loadbalancing exporter to distribute to back-tier Collectors that perform tail sampling.

For sidecar deployments (one Collector per application pod), scaling is automatic since each pod has its own Collector. However, sidecar Collectors should be lightweight, performing only batching and export, with heavier processing offloaded to a centralized Collector tier.

High Availability

For the Collector to be highly available, you need to handle both planned (upgrades, configuration changes) and unplanned (crashes, node failures) disruptions. Running multiple Collector replicas behind a load balancer provides resilience against individual instance failures. The Collector's internal buffering (via the batch processor) provides a small window of tolerance for brief downstream outages.

For longer outages, some deployments add a Kafka topic between the application and the Collector. Applications export to Kafka, and the Collector consumes from Kafka. This decouples the application from the Collector's availability and provides backpressure handling through Kafka's consumer lag mechanism. The Collector's kafka receiver and kafka exporter make this pattern straightforward to implement.

Pipeline Optimization

Processor ordering matters for performance. Place the memory_limiter processor first in every pipeline so it can apply backpressure before other processors consume memory. Place the filter processor early to drop unwanted data before it flows through more expensive processors. Place the batch processor last (before the exporter) to ensure that export operations are efficiently batched.

Monitor the Collector itself. The Collector exposes Prometheus metrics about its own performance: spans received, spans dropped, export latency, queue depth, memory usage, and processor-specific metrics. Set up alerts for high drop rates, export failures, and memory approaching the limit. The Collector's zpages extension provides a web UI with real-time pipeline statistics, useful for debugging during initial setup.

Resource Allocation

Collector CPU usage scales primarily with the number of processors and the complexity of processing rules. Tail-based sampling is the most CPU-intensive processor due to trace reassembly and policy evaluation. Collector memory usage depends on the batch processor's buffer size and the tail sampling processor's trace buffer. A rule of thumb is to allocate 2x the expected peak buffer size to account for traffic spikes.

For Kubernetes deployments, set resource requests and limits on Collector pods. The memory_limiter processor should be configured with limits below the pod's memory limit to allow the Collector to gracefully shed load rather than being OOM-killed. A common pattern is to set the pod memory limit to 2 GiB, the memory_limiter soft limit to 1.5 GiB, and the hard limit to 1.7 GiB.

Security Considerations

The Collector handles sensitive telemetry data and should be secured accordingly. Enable TLS on all receiver endpoints. Use mTLS for Collector-to-Collector communication in multi-tier deployments. Configure authentication on the Collector's receivers to prevent unauthorized data injection. Use the attributes processor to redact or hash sensitive data (credit card numbers, social security numbers, passwords) before it leaves the Collector.

The Collector should run with minimal privileges. In Kubernetes, configure the Collector's pod security context to run as a non-root user, drop all capabilities, and use a read-only root filesystem. The only exception is the filelog receiver, which needs read access to log file directories.

The Road Ahead

OpenTelemetry's trajectory shows no signs of slowing. The profiling signal is moving toward general availability, which will complete the four-signal vision. The project is investing in improved semantic conventions coverage, aiming to standardize attribute names across every major technology domain. The Collector is gaining support for more sophisticated pipeline topologies, including conditional routing and fan-in patterns.

The integration of AI and machine learning workloads into OpenTelemetry is an emerging focus area. LLM inference requests, model training runs, and AI agent orchestration all generate telemetry that benefits from standardized instrumentation. The semantic conventions working group has begun defining attributes for AI/ML operations, including model name, model version, token counts, inference latency, and prompt/completion metadata.

eBPF-based instrumentation is likely to become a first-class citizen in the OpenTelemetry project rather than a complementary tool. The ability to instrument applications at the kernel level without any application modification is too compelling for the project to treat as an external concern. We can expect deeper integration between eBPF-based discovery and SDK-based instrumentation, with eBPF providing baseline coverage and SDKs adding application-specific detail.

The consolidation of the observability vendor landscape around OTLP is accelerating the commoditization of telemetry ingestion. Vendor differentiation is increasingly happening at the query, analysis, and visualization layers rather than at the data collection layer. This is a healthy evolution that benefits users by making it easier to switch between backends and by focusing vendor innovation on the areas that directly improve operational efficiency.

Conclusion

OpenTelemetry has achieved what many thought impossible: a single, open-source, vendor-neutral standard for all telemetry data in cloud-native systems. Its architecture, a clean separation of API, SDK, Collector, and protocol, has proven flexible enough to accommodate the needs of everything from single-service applications to globally distributed microservice fleets handling millions of requests per second.

The project's success is not merely technical. It represents a philosophical shift in how the industry thinks about observability. Telemetry data is no longer a proprietary asset locked inside vendor-specific formats. It is an open resource that flows through standardized pipelines to whatever analysis tools best serve the organization's needs. This shift has empowered engineering teams to focus on understanding their systems rather than managing their monitoring infrastructure.

For organizations that have not yet adopted OpenTelemetry, the path is clear and well-trodden. Start with the Collector to centralize your existing telemetry. Add auto-instrumentation to get immediate trace and metric coverage. Gradually enrich your instrumentation with custom spans and attributes as you identify the observability gaps that matter most for your specific systems. The investment pays dividends in reduced incident response times, improved deployment confidence, and a deeper understanding of system behavior under real-world conditions.

OpenTelemetry is not the future of observability. It is the present. And in 2026, that present has never looked more capable or more accessible.