eBPF Performance Tuning: Methodologies, Workflows, and Production Playbooks

Introduction: From Observability to Performance Engineering

Most discussions about eBPF center on observability tooling and security enforcement. Those are solved problems in 2026. What remains underexplored is the discipline that matters most to engineers with production systems under pressure: performance tuning. Not just collecting metrics, but systematically diagnosing why a service is slow, where memory is leaking, which queries are dragging down your database, and how to prove that your optimization actually worked.

This article is a practitioner's guide. It covers the methodologies, the specific bpftrace programs, the flame graph workflows, and the production playbooks that turn eBPF from a kernel curiosity into the most powerful performance engineering tool available on Linux today. If you have ever stared at a dashboard full of green metrics while users complain about latency, this is for you.

The shift from traditional profiling to eBPF-based performance analysis is not incremental. Traditional tools like strace, perf, and application-level profilers each give you one slice of the picture. They impose overhead that distorts measurements. They require restarts, recompilations, or agent deployments. eBPF eliminates these constraints by running programs directly in the kernel, attaching to any function or tracepoint, and aggregating data in-kernel before surfacing results to userspace. The overhead is measured in single-digit nanoseconds per probe, making it safe for continuous use in production.

By 2026, the eBPF performance tuning ecosystem has matured considerably. The BCC toolkit provides over 100 ready-made tools. bpftrace offers a high-level scripting language for ad-hoc analysis. libbpf and CO-RE (Compile Once, Run Everywhere) have solved the portability problem that plagued earlier eBPF adoption. And the kernel itself continues to expand the surface area available to eBPF programs, with newer kernels supporting more probe types, larger program sizes, and more sophisticated map structures.

This guide assumes you are running Linux kernel 5.15 or later (the minimum for most CO-RE features) and have bpftrace 0.20+ installed. All examples are tested against production-grade workloads on both bare metal and Kubernetes environments.

---

The USE Method with eBPF: A Systematic Resource Checklist

Brendan Gregg's USE method is the most reliable framework for systematic performance analysis. For every resource in the system, you check three things: Utilization (how busy is it?), Saturation (is work queuing?), and Errors (are operations failing?). The method is simple, exhaustive, and prevents the common mistake of chasing symptoms rather than root causes.

What makes eBPF transformative for the USE method is that it can measure all three dimensions for every resource with a single toolset. Before eBPF, checking USE for CPUs required mpstat, for disks required iostat, for network required sar, and for saturation you often had to infer from indirect signals. With eBPF, you can write precise tracepoint programs that give you exact utilization, exact queue depths, and exact error counts, all from the same framework.

CPU Resources

Utilization is the percentage of time CPUs spend executing work rather than sitting idle. The traditional approach uses /proc/stat counters, but these are system-wide averages that hide per-core hotspots and short bursts. With eBPF, you can trace scheduler events to measure exact per-core utilization at arbitrary granularity:

# Per-CPU utilization at 1-second intervals using scheduler tracepoints
bpftrace -e 'tracepoint:sched:sched_switch {
  @idle[cpu] = count();
}
interval:s:1 {
  print(@idle);
  clear(@idle);
}'

Saturation for CPUs means runnable threads waiting for a CPU. The run queue length is the key metric. A sustained run queue depth greater than the CPU count indicates saturation:

# Run queue length sampling
bpftrace -e 'profile:hz:99 {
  @runqlen = lhist(curtask->se.statistics.nr_switches, 0, 100, 1);
}'

Errors for CPUs are rare but include machine check exceptions and thermal throttling events. eBPF can attach to the relevant tracepoints to count these automatically.

Memory Resources

Utilization is tracked through page allocation and free events. Beyond simple free memory percentages, eBPF lets you understand allocation patterns by tracking mm_page_alloc and mm_page_free tracepoints, giving you the allocation rate, the dominant allocation sizes, and which processes are consuming the most memory.

Saturation manifests as page scanning, swapping, and OOM kills. You can trace the vmscan tracepoints to measure exactly how much time the kernel spends reclaiming memory:

# Page reclaim activity tracing
bpftrace -e 'tracepoint:vmscan:mm_vmscan_direct_reclaim_begin {
  @start[tid] = nsecs;
}
tracepoint:vmscan:mm_vmscan_direct_reclaim_end /@start[tid]/ {
  @reclaim_ns = hist(nsecs - @start[tid]);
  delete(@start[tid]);
}'

Errors include allocation failures and OOM events. Tracing oom_score_adj_update and mark_victim gives you early warning before the OOM killer fires.

Disk I/O Resources

Utilization is measured by tracing block I/O completions and calculating the fraction of time the device is busy. The block:block_rq_issue and block:block_rq_complete tracepoints provide exact I/O timing:

# Block device utilization and latency histogram
bpftrace -e 'tracepoint:block:block_rq_issue {
  @start[args->dev, args->sector] = nsecs;
}
tracepoint:block:block_rq_complete /@start[args->dev, args->sector]/ {
  @usecs = hist((nsecs - @start[args->dev, args->sector]) / 1000);
  delete(@start[args->dev, args->sector]);
}'

Saturation is the I/O queue depth. When requests start queuing beyond the device's optimal depth, latency increases nonlinearly. eBPF can track queue depths at the block layer to identify when this threshold is crossed.

Errors include I/O errors, timeouts, and retries. These are directly available via the block:block_rq_complete tracepoint's error field.

Network Resources

Utilization at the network level means bandwidth consumption relative to link capacity. eBPF socket-level tracing gives you per-connection, per-process bandwidth accounting without relying on interface-level counters that aggregate everything:

# Per-process network throughput
bpftrace -e 'kretprobe:tcp_sendmsg {
  @bytes[comm] = sum(retval);
}
interval:s:5 {
  print(@bytes);
  clear(@bytes);
}'

Saturation appears as socket buffer overflows, TCP window zero events, and transmit queue drops. These are invisible to traditional monitoring but clearly observable through eBPF socket tracepoints.

Errors include TCP retransmissions, connection resets, and ICMP errors. The tcp:tcp_retransmit_skb tracepoint is one of the most valuable probes in all of eBPF for network performance analysis.

---

Thread State Analysis (TSA) with eBPF

The USE method tells you which resource is the bottleneck. The TSA method tells you what your threads are actually doing when they should be making progress. Thread State Analysis examines where threads spend their time across six states: executing on-CPU, runnable but waiting for a CPU, sleeping in an interruptible state, sleeping in an uninterruptible state (usually I/O), stopped, and zombie.

TSA is particularly powerful for diagnosing latency problems in multi-threaded applications where traditional CPU profiling shows low utilization but users experience high latency. The threads are not CPU-bound; they are waiting on locks, I/O, or other resources. eBPF is uniquely suited to TSA because it can trace scheduler events, block I/O events, lock acquisition events, and voluntary sleep events simultaneously.

Implementing TSA with bpftrace

The core of TSA is tracking state transitions via the sched_switch tracepoint. When a thread is switched off a CPU, the previous state field tells you why:

# Thread state analysis - where are threads spending time?
bpftrace -e 'tracepoint:sched:sched_switch {
  if (args->prev_comm == "myapp") {
    @state[args->prev_state] = count();
    @offcpu_start[args->prev_pid] = nsecs;
  }
  if (args->next_comm == "myapp") {
    if (@offcpu_start[args->next_pid]) {
      @offcpu_us = hist((nsecs - @offcpu_start[args->next_pid]) / 1000);
      delete(@offcpu_start[args->next_pid]);
    }
  }
}'

The prev_state values map directly to thread states: 0 means the thread was runnable (involuntary preemption or yield), 1 means interruptible sleep (waiting for I/O, locks, or condition variables), 2 means uninterruptible sleep (usually disk I/O or NFS), and 4 or higher indicates stopped or traced states.

Off-CPU Analysis Deep Dive

Off-CPU analysis is the complement to CPU profiling. CPU profiling tells you which code paths burn the most CPU time. Off-CPU analysis tells you which code paths cause the most waiting time. For latency-sensitive services, off-CPU time is often the dominant contributor to request latency.

The key insight is capturing the kernel stack trace when a thread goes off-CPU. This stack trace shows the exact code path that led to the blocking event:

# Off-CPU flame graph data collection
bpftrace -e 'tracepoint:sched:sched_switch /args->prev_comm == "myapp"/ {
  @offcpu[kstack, ustack, args->prev_pid] = sum(nsecs - @start[args->prev_pid]);
  @start[args->next_pid] = nsecs;
}
tracepoint:sched:sched_switch /args->next_comm == "myapp"/ {
  @start[args->next_pid] = nsecs;
}'

This output can be piped directly into Brendan Gregg's FlameGraph tools to produce an off-CPU flame graph, which is one of the most powerful diagnostic visualizations available.

Wake-up Analysis

When a thread is sleeping, something must wake it up. Tracing wakeup events via sched:sched_wakeup and correlating them with the sleeping thread's stack reveals the dependency chain. For example, if thread A is sleeping on a mutex and thread B releases the mutex, the wakeup trace will show thread B waking thread A, and the stack traces will reveal the exact lock and code location involved.

# Wakeup chain analysis
bpftrace -e 'tracepoint:sched:sched_wakeup /args->comm == "myapp"/ {
  printf("waker: %s (pid %d) -> wakee: %s (pid %d)\n",
    comm, pid, args->comm, args->pid);
  print(kstack);
}'

This technique is invaluable for debugging lock contention, producer-consumer bottlenecks, and inter-service communication latency.

---

The bpftrace Cookbook: Production One-Liners

The power of bpftrace lies in its ability to answer specific performance questions with single commands. This section provides a cookbook of production-tested one-liners organized by the question they answer.

Latency Questions

"What is the latency distribution of syscalls made by my application?"

bpftrace -e 'tracepoint:raw_syscalls:sys_enter /comm == "myapp"/ {
  @start[tid] = nsecs;
}
tracepoint:raw_syscalls:sys_exit /comm == "myapp" && @start[tid]/ {
  @syscall_us = hist((nsecs - @start[tid]) / 1000);
  delete(@start[tid]);
}'

"Which specific syscalls are the slowest?"

bpftrace -e 'tracepoint:raw_syscalls:sys_enter /comm == "myapp"/ {
  @start[tid] = nsecs;
  @sc[tid] = args->id;
}
tracepoint:raw_syscalls:sys_exit /comm == "myapp" && @start[tid]/ {
  @slow_syscalls[@sc[tid]] = hist((nsecs - @start[tid]) / 1000);
  delete(@start[tid]);
  delete(@sc[tid]);
}'

"What is the read latency for my application's file I/O?"

bpftrace -e 'tracepoint:syscalls:sys_enter_read /comm == "myapp"/ {
  @start[tid] = nsecs;
  @fd[tid] = args->fd;
}
tracepoint:syscalls:sys_exit_read /comm == "myapp" && @start[tid]/ {
  @read_us[@fd[tid]] = hist((nsecs - @start[tid]) / 1000);
  delete(@start[tid]);
  delete(@fd[tid]);
}'

Resource Questions

"How much memory is my application allocating per second?"

bpftrace -e 'uprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc /comm == "myapp"/ {
  @alloc_bytes = sum(arg0);
  @alloc_sizes = hist(arg0);
}
interval:s:1 {
  print(@alloc_bytes);
  clear(@alloc_bytes);
}'

"Which kernel functions are consuming the most CPU in my workload?"

bpftrace -e 'profile:hz:99 /comm == "myapp"/ {
  @cpu_funcs[kstack(5)] = count();
}'

"What is the TCP connection establishment latency?"

bpftrace -e 'kprobe:tcp_v4_connect {
  @start[tid] = nsecs;
}
kretprobe:tcp_v4_connect /@start[tid]/ {
  @connect_us = hist((nsecs - @start[tid]) / 1000);
  delete(@start[tid]);
}'

Throughput Questions

"What is the distribution of network packet sizes?"

bpftrace -e 'tracepoint:net:net_dev_xmit {
  @pkt_size = hist(args->len);
  @pkt_by_dev[args->name] = count();
}'

"How many context switches per second is my application experiencing?"

bpftrace -e 'tracepoint:sched:sched_switch /args->prev_comm == "myapp"/ {
  @voluntary[args->prev_state == 0 ? "involuntary" : "voluntary"] = count();
}
interval:s:1 {
  print(@voluntary);
  clear(@voluntary);
}'

These one-liners form the building blocks of more complex analysis. In practice, you chain them together based on initial findings: a high context switch rate leads you to lock contention analysis, which leads you to off-CPU flame graphs, which pinpoints the exact code path to optimize.

---

Flame Graphs: The Visual Language of Performance

Flame graphs, invented by Brendan Gregg in 2011, have become the standard visualization for performance data. By 2026, they have evolved far beyond simple CPU profiling into a family of visualizations that cover every dimension of system performance. eBPF is the ideal data source for flame graphs because it can capture stack traces at any point in the system without the sampling bias of traditional profilers.

CPU Flame Graphs

CPU flame graphs answer the question: "Where is my application spending CPU time?" The x-axis represents the population of stack traces (not time), and the y-axis shows the call stack depth. The width of each frame is proportional to how often that function appears in the sampled stacks.

Generating a CPU flame graph with eBPF:

# Collect CPU profile data for 30 seconds
bpftrace -e 'profile:hz:99 /comm == "myapp"/ {
  @[ustack] = count();
}' -d 30 > cpu_stacks.txt

# Convert to flame graph
stackcollapse-bpftrace.pl cpu_stacks.txt | flamegraph.pl > cpu_flamegraph.svg

The 99Hz sampling rate is deliberate. It avoids aliasing with common timer frequencies (100Hz, 250Hz, 1000Hz) that could bias results toward timer-related code paths.

Reading CPU flame graphs effectively:

Look for plateaus: wide, flat areas at the top of the graph indicate functions where significant CPU time is spent without calling other functions. These are your optimization targets. Look for towers: narrow, deep stacks suggest deep call chains that might benefit from inlining or refactoring. Look for unexpected width: if a function you expect to be cheap appears wide, investigate whether it is being called too frequently rather than being too slow per call.

Off-CPU Flame Graphs

Off-CPU flame graphs answer: "Where is my application spending time waiting?" These are often more valuable than CPU flame graphs for latency analysis because modern applications spend most of their time waiting on I/O, locks, and network responses.

# Off-CPU data collection
bpftrace -e 'tracepoint:sched:sched_switch /args->prev_comm == "myapp"/ {
  @start[args->prev_pid] = nsecs;
}
tracepoint:sched:sched_switch /args->next_comm == "myapp" && @start[args->next_pid]/ {
  @blocked[ustack, kstack] = sum(nsecs - @start[args->next_pid]);
  delete(@start[args->next_pid]);
}' > offcpu_stacks.txt

An off-CPU flame graph from a production service might reveal that 40% of waiting time is in epoll_wait (normal for an event-driven server), 30% is in futex (lock contention), 20% is in nanosleep (timer waits), and 10% is in read (file I/O). The lock contention and file I/O portions are immediate optimization opportunities.

Differential Flame Graphs

Differential flame graphs compare two profiles to highlight what changed between them. They are invaluable for performance regression analysis. The output uses color to indicate change: red frames are hotter (more samples) in the second profile, and blue frames are cooler (fewer samples).

# Capture baseline profile
bpftrace -e 'profile:hz:99 /comm == "myapp"/ { @[ustack] = count(); }' \
  -d 60 > baseline.txt

# Deploy new version, then capture comparison profile
bpftrace -e 'profile:hz:99 /comm == "myapp"/ { @[ustack] = count(); }' \
  -d 60 > comparison.txt

# Generate differential flame graph
difffolded.pl baseline.folded comparison.folded | flamegraph.pl > diff.svg

Memory Flame Graphs

Memory flame graphs visualize allocation patterns. Instead of sampling CPU stacks, they capture the stack trace at every allocation event and weight by allocation size. This reveals which code paths are responsible for the most memory pressure:

# Memory allocation flame graph data
bpftrace -e 'uprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc /comm == "myapp"/ {
  @bytes[ustack] = sum(arg0);
}'

Memory flame graphs are essential for diagnosing memory leaks, excessive garbage collection pressure, and inefficient data structure choices.

---

Application-Level Performance Analysis

While kernel-level tracing gives you the system perspective, eBPF can also trace application-level functions to measure specific code paths. This is done through uprobes (user-space probes) that attach to function entry and exit points in application binaries.

Function Latency Histograms

You can measure the latency distribution of any function in your application without modifying its code:

# Measure latency of a specific application function
bpftrace -e 'uprobe:./myapp:processRequest {
  @start[tid] = nsecs;
}
uretprobe:./myapp:processRequest /@start[tid]/ {
  @latency_us = hist((nsecs - @start[tid]) / 1000);
  delete(@start[tid]);
}'

The histogram output immediately reveals whether you have a unimodal distribution (consistent performance), a bimodal distribution (two distinct code paths or cache hit vs. miss), or a long tail (occasional slow paths).

Slow Path Identification

One of the most common performance patterns is the "slow path": a code path that is rarely taken but dramatically slower when it is. eBPF can conditionally trace only the slow executions:

# Only capture stacks for requests slower than 10ms
bpftrace -e 'uprobe:./myapp:processRequest {
  @start[tid] = nsecs;
}
uretprobe:./myapp:processRequest /@start[tid]/ {
  $duration = nsecs - @start[tid];
  if ($duration > 10000000) {
    @slow_stacks[ustack] = count();
    @slow_duration = hist($duration / 1000000);
  }
  delete(@start[tid]);
}'

This technique captures stack traces only for requests exceeding 10ms, filtering out the noise of normal-path executions and focusing your attention on the exact code paths responsible for tail latency.

Request Flow Tracing

For microservices, understanding the latency breakdown within a single request is critical. By probing multiple functions along the request path, you can build a latency waterfall:

# Request flow breakdown
bpftrace -e '
uprobe:./myapp:parseRequest { @parse_start[tid] = nsecs; }
uretprobe:./myapp:parseRequest /@parse_start[tid]/ {
  @parse_us = hist((nsecs - @parse_start[tid]) / 1000);
  delete(@parse_start[tid]);
}
uprobe:./myapp:queryDatabase { @db_start[tid] = nsecs; }
uretprobe:./myapp:queryDatabase /@db_start[tid]/ {
  @db_us = hist((nsecs - @db_start[tid]) / 1000);
  delete(@db_start[tid]);
}
uprobe:./myapp:serializeResponse { @ser_start[tid] = nsecs; }
uretprobe:./myapp:serializeResponse /@ser_start[tid]/ {
  @ser_us = hist((nsecs - @ser_start[tid]) / 1000);
  delete(@ser_start[tid]);
}'

This produces separate latency histograms for parsing, database queries, and serialization, showing you exactly which phase dominates your request latency.

---

Database Performance Tracing with eBPF

Database performance is one of the most impactful areas for eBPF-based analysis. Traditional database monitoring relies on slow query logs and internal performance schemas, which miss queries that are individually fast but collectively expensive, and which cannot correlate database behavior with application code paths.

MySQL Query Tracing

eBPF can trace MySQL query execution by attaching to the dispatch_command function in the MySQL server binary. This captures every query, its execution time, and the full stack trace of the calling code:

# MySQL query latency tracing
bpftrace -e 'uprobe:/usr/sbin/mysqld:dispatch_command {
  @start[tid] = nsecs;
  @query[tid] = str(arg2);
}
uretprobe:/usr/sbin/mysqld:dispatch_command /@start[tid]/ {
  $duration_ms = (nsecs - @start[tid]) / 1000000;
  if ($duration_ms > 5) {
    printf("slow query (%d ms): %s\n", $duration_ms, @query[tid]);
  }
  @query_latency = hist($duration_ms);
  delete(@start[tid]);
  delete(@query[tid]);
}'

This approach has several advantages over MySQL's built-in slow query log. It captures queries below the slow query threshold (which defaults to 1 second). It measures wall-clock time including lock wait time, not just execution time. And it works without restarting or reconfiguring the database server.

PostgreSQL Query Analysis

PostgreSQL query tracing follows a similar pattern but targets the exec_simple_query function:

# PostgreSQL query tracing
bpftrace -e 'uprobe:/usr/lib/postgresql/16/bin/postgres:exec_simple_query {
  @start[tid] = nsecs;
  @query[tid] = str(arg0);
}
uretprobe:/usr/lib/postgresql/16/bin/postgres:exec_simple_query /@start[tid]/ {
  $duration_ms = (nsecs - @start[tid]) / 1000000;
  @pg_latency = hist($duration_ms);
  if ($duration_ms > 10) {
    printf("slow: %d ms | %s\n", $duration_ms, @query[tid]);
  }
  delete(@start[tid]);
  delete(@query[tid]);
}'

Connection Pool Analysis

Connection pool exhaustion is a common cause of application latency spikes. eBPF can trace connection acquisition and release to measure pool utilization and identify connection leaks:

# Connection pool utilization
bpftrace -e 'uprobe:./myapp:getConnection {
  @conn_wait_start[tid] = nsecs;
}
uretprobe:./myapp:getConnection /@conn_wait_start[tid]/ {
  @conn_wait_us = hist((nsecs - @conn_wait_start[tid]) / 1000);
  @active_conns = count();
  delete(@conn_wait_start[tid]);
}
uprobe:./myapp:releaseConnection {
  @active_conns = count();
}'

When connection wait times spike, the histogram shifts from microseconds to milliseconds or even seconds, providing a clear signal that the pool is saturated.

Query Pattern Analysis

Beyond individual query latency, eBPF can aggregate query patterns to identify which query types dominate database load. By hashing the query structure (stripping parameter values), you can group queries by their template and measure the aggregate impact:

# Query frequency and total time by connection
bpftrace -e 'uprobe:/usr/sbin/mysqld:dispatch_command {
  @start[tid] = nsecs;
}
uretprobe:/usr/sbin/mysqld:dispatch_command /@start[tid]/ {
  @total_time[comm, tid] = sum(nsecs - @start[tid]);
  @query_count[comm, tid] = count();
  delete(@start[tid]);
}'

This reveals the classic database performance insight: the slowest individual queries are rarely the biggest performance problem. Instead, it is usually a moderately fast query executed thousands of times per second that dominates total database time.

---

JIT-Compiled Language Profiling

Profiling applications written in JIT-compiled languages like Java, Go, and Node.js presents unique challenges. These languages generate machine code at runtime, which means the binary on disk does not contain the functions actually executing. Traditional profilers see only the JIT compiler's internal functions rather than the application code. eBPF solves this through symbol resolution mechanisms specific to each runtime.

Java Profiling with eBPF

Java applications run on the JVM, which JIT-compiles bytecode to native code at runtime. For eBPF to produce meaningful stack traces, it needs a mapping from JIT-compiled code addresses to Java method names. The JVM provides this through perf-map-agent (or the newer -XX:+PreserveFramePointer flag combined with -XX:+DumpPerfMapAtExit):

# Enable Java frame pointers for eBPF profiling
java -XX:+PreserveFramePointer -XX:+UnlockDiagnosticVMOptions \
  -XX:+DumpPerfMapAtExit -jar myapp.jar

# Profile with bpftrace (after perf map is available)
bpftrace -e 'profile:hz:99 /comm == "java"/ {
  @[ustack] = count();
}'

The PreserveFramePointer flag is essential. Without it, the JVM omits frame pointers as an optimization, and eBPF cannot walk the user-space stack. The performance cost of preserving frame pointers is typically 1-3%, which is acceptable for production profiling.

In 2026, the JVM ecosystem has improved eBPF support significantly. OpenJDK 21+ includes native support for perf map files, and tools like async-profiler can work alongside eBPF-based profiling to provide comprehensive Java performance data.

Go Profiling with eBPF

Go presents different challenges. The Go runtime uses a non-standard calling convention and stack layout that confused early eBPF stack walkers. However, since Go 1.17, the ABI switched to register-based calling conventions, and Go 1.21+ includes frame pointer support by default on x86-64:

# Go application profiling - frame pointers enabled by default since Go 1.21
bpftrace -e 'profile:hz:99 /comm == "myapp"/ {
  @[ustack] = count();
}'

For Go applications running on older versions, you can force frame pointer generation:

# Build with frame pointers for older Go versions
GOFLAGS="-buildvcs=false" go build -gcflags="-l" -o myapp .

Go's goroutine model also benefits from eBPF analysis. You can trace goroutine creation, blocking, and scheduling to understand concurrency bottlenecks:

# Goroutine creation rate
bpftrace -e 'uprobe:./myapp:runtime.newproc1 {
  @goroutine_creates = count();
}
interval:s:1 {
  print(@goroutine_creates);
  clear(@goroutine_creates);
}'

Node.js Profiling with eBPF

Node.js runs on the V8 JavaScript engine, which JIT-compiles JavaScript to machine code. V8 can generate perf map files for eBPF symbol resolution:

# Start Node.js with perf map generation
node --perf-basic-prof --perf-prof myapp.js

# Profile with bpftrace
bpftrace -e 'profile:hz:99 /comm == "node"/ {
  @[ustack] = count();
}'

The --perf-basic-prof flag generates a /tmp/perf-<pid>.map file that maps JIT code addresses to JavaScript function names. This file is automatically picked up by bpftrace for symbol resolution.

For Node.js applications, off-CPU analysis is often more valuable than CPU profiling because the event loop model means the single JavaScript thread spends much of its time waiting on I/O callbacks. Off-CPU flame graphs reveal which I/O operations are blocking the event loop.

---

Memory Allocation Tracing and Leak Detection

Memory leaks are among the most insidious production issues. They manifest slowly, often over days or weeks, and traditional profiling captures a snapshot rather than the cumulative pattern. eBPF excels at memory analysis because it can trace every allocation and free, maintain running counts in kernel-space maps, and report only the outstanding (leaked) allocations.

malloc/free Tracking

The fundamental approach traces malloc and free in libc, recording each allocation's address, size, and stack trace:

# Track outstanding allocations (potential leaks)
bpftrace -e '
uprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc /comm == "myapp"/ {
  @alloc_start[tid] = nsecs;
  @alloc_size[tid] = arg0;
}
uretprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc /comm == "myapp" && @alloc_start[tid]/ {
  @outstanding[retval] = @alloc_size[tid];
  @alloc_stacks[ustack] = sum(@alloc_size[tid]);
  delete(@alloc_start[tid]);
  delete(@alloc_size[tid]);
}
uprobe:/lib/x86_64-linux-gnu/libc.so.6:free /comm == "myapp"/ {
  delete(@outstanding[arg0]);
}'

After running for a period, the @outstanding map contains all allocations that have not been freed. The @alloc_stacks map shows which code paths are responsible for the most total allocated memory. Code paths that appear in @alloc_stacks but whose allocations never appear as free targets are leak candidates.

Memory Leak Detection Workflow

A systematic memory leak investigation follows these steps:

1. Baseline: Run the malloc/free tracker for a fixed period under steady-state load. Record the total outstanding allocation count and size.

2. Growth detection: Repeat the measurement after a longer period. If outstanding allocations grow linearly with time (not with load), you have a leak.

3. Stack attribution: Examine the @alloc_stacks output to identify which code paths are responsible for growing allocations.

4. Confirmation: Instrument the specific function identified in step 3 with a more focused probe to confirm the leak path.

5. Fix verification: After deploying the fix, repeat the baseline measurement to confirm that outstanding allocations stabilize.

Slab Allocator Analysis

For kernel memory issues, eBPF can trace the slab allocator to identify kernel memory leaks or excessive slab usage:

# Kernel slab allocation tracing
bpftrace -e 'tracepoint:kmem:kmalloc {
  @slab_allocs[kstack(5)] = sum(args->bytes_alloc);
}
tracepoint:kmem:kfree {
  @slab_frees = count();
}
interval:s:10 {
  print(@slab_allocs);
  clear(@slab_allocs);
}'

This is particularly useful for debugging kernel module memory leaks, which are invisible to user-space memory analysis tools.

Page Fault Analysis

Page faults are another dimension of memory performance. Minor page faults (mapping a page from the page cache) are fast but not free. Major page faults (reading from disk) can add milliseconds of latency. eBPF can distinguish between the two and attribute them to specific code paths:

# Page fault analysis by type and code path
bpftrace -e 'software:major-faults:1 /comm == "myapp"/ {
  @major_faults[ustack(5)] = count();
}
software:minor-faults:100 /comm == "myapp"/ {
  @minor_faults[ustack(5)] = count();
}'

Applications with high major fault rates are often suffering from insufficient memory or suboptimal memory access patterns that defeat the page cache.

---

I/O Performance Analysis

I/O performance is a perennial bottleneck in production systems. Whether it is disk I/O, file system operations, or block device behavior, eBPF provides unprecedented visibility into the I/O stack.

Disk I/O Latency Distribution

The most fundamental I/O measurement is latency distribution. Unlike averages, which hide the shape of performance, histograms reveal whether I/O latency is consistent, bimodal, or has a long tail:

# Disk I/O latency histogram by device
bpftrace -e 'tracepoint:block:block_rq_issue {
  @start[args->dev, args->sector] = nsecs;
}
tracepoint:block:block_rq_complete /@start[args->dev, args->sector]/ {
  $lat = (nsecs - @start[args->dev, args->sector]) / 1000;
  @io_lat_us[args->dev] = hist($lat);
  delete(@start[args->dev, args->sector]);
}'

A bimodal distribution (peaks at both 100us and 10ms) typically indicates a mix of cache hits and cache misses at the storage layer. A long tail (99th percentile 10x the median) suggests occasional queue depth spikes or storage controller congestion.

File System Tracing

eBPF can trace at the file system layer, above the block device layer, to capture higher-level operations like file open, read, write, and fsync:

# File system operation latency
bpftrace -e 'kprobe:vfs_read {
  @start[tid] = nsecs;
}
kretprobe:vfs_read /@start[tid]/ {
  @vfs_read_us = hist((nsecs - @start[tid]) / 1000);
  delete(@start[tid]);
}
kprobe:vfs_write {
  @wstart[tid] = nsecs;
}
kretprobe:vfs_write /@wstart[tid]/ {
  @vfs_write_us = hist((nsecs - @wstart[tid]) / 1000);
  delete(@wstart[tid]);
}'

The difference between file system and block I/O latency reveals the overhead of the file system itself, including journaling, metadata updates, and page cache management.

Block I/O Patterns

Understanding I/O patterns is critical for storage capacity planning. eBPF can characterize the I/O workload by size, direction (read vs. write), and sequentiality:

# I/O size distribution and read/write ratio
bpftrace -e 'tracepoint:block:block_rq_issue {
  @io_size = hist(args->bytes);
  @io_type[args->rwbs] = count();
}'

Sequential I/O (large, contiguous requests) performs dramatically better than random I/O on both spinning disks and SSDs. If your workload shows predominantly small, random I/O, optimization strategies include request coalescing, read-ahead tuning, and write buffering.

fsync and Write Barriers

For databases and other applications that require durable writes, fsync latency is often the dominant performance factor. eBPF can trace fsync calls and their interaction with the block layer:

# fsync latency by process
bpftrace -e 'tracepoint:syscalls:sys_enter_fsync /comm == "postgres"/ {
  @start[tid] = nsecs;
}
tracepoint:syscalls:sys_exit_fsync /@start[tid]/ {
  @fsync_ms = hist((nsecs - @start[tid]) / 1000000);
  delete(@start[tid]);
}'

High fsync latency is a common finding in database performance investigations. Solutions include battery-backed write caches, NVMe drives, and database-level WAL tuning.

---

Network Performance Debugging

Network performance issues are notoriously difficult to diagnose because they span multiple layers (application, socket, TCP, IP, driver) and multiple hosts. eBPF provides hooks at every layer, enabling systematic network performance analysis.

TCP Retransmission Analysis

TCP retransmissions are the single most important network performance metric. Each retransmission indicates packet loss and triggers TCP's congestion control algorithm, reducing throughput. eBPF traces retransmissions with full context:

# TCP retransmission tracing with connection details
bpftrace -e 'tracepoint:tcp:tcp_retransmit_skb {
  printf("retransmit: %s:%d -> %s:%d state=%d\n",
    ntop(args->saddr), args->sport,
    ntop(args->daddr), args->dport, args->state);
  @retrans[ntop(args->daddr), args->dport] = count();
}'

This reveals which destination hosts and ports are experiencing the most retransmissions. High retransmission rates to a specific service indicate either network path congestion, receiver-side buffer exhaustion, or application-level backpressure.

Connection Lifecycle Analysis

Understanding how long connections live, how long they take to establish, and why they are closed provides essential context for connection management optimization:

# TCP connection lifecycle
bpftrace -e 'kprobe:tcp_v4_connect {
  @connect_start[tid] = nsecs;
}
kretprobe:tcp_v4_connect /@connect_start[tid]/ {
  @connect_latency = hist((nsecs - @connect_start[tid]) / 1000);
  delete(@connect_start[tid]);
}
tracepoint:tcp:tcp_set_state {
  if (args->newstate == 7) {
    @close_reasons[comm] = count();
  }
}'

Socket Buffer Tuning

Socket buffer sizes directly affect network throughput for bulk data transfers. Undersized buffers limit the bandwidth-delay product that the connection can fill. eBPF can measure actual buffer utilization to guide tuning:

# Socket buffer utilization
bpftrace -e 'kprobe:tcp_sendmsg {
  $sk = (struct sock *)arg0;
  @sndbuf = hist($sk->sk_sndbuf);
  @wmem = hist($sk->sk_wmem_queued);
}'

If sk_wmem_queued frequently approaches sk_sndbuf, the send buffer is the bottleneck and should be increased. The optimal size is at least the bandwidth-delay product of the network path.

DNS Resolution Latency

DNS resolution latency is often overlooked but can add tens of milliseconds to every new connection. eBPF can trace DNS queries at the socket level:

# DNS resolution latency
bpftrace -e 'kprobe:udp_sendmsg /comm == "myapp"/ {
  $sk = (struct sock *)arg0;
  if ($sk->sk_dport == 13568) {
    @dns_start[tid] = nsecs;
  }
}
kprobe:udp_recvmsg /comm == "myapp" && @dns_start[tid]/ {
  @dns_us = hist((nsecs - @dns_start[tid]) / 1000);
  delete(@dns_start[tid]);
}'

Port 13568 is port 53 in network byte order. This probe captures the round-trip time for DNS queries, revealing whether DNS is contributing to connection establishment latency.

---

Production Performance Playbooks

Isolated tools are useful, but systematic methodologies are what separate effective performance engineers from those who guess. This section provides three production playbooks for the most common performance investigation types.

Playbook 1: Latency Investigation

When users report that a service is slow, follow this systematic approach:

Step 1 -- Quantify the problem. Attach to the service's request handling function and capture a latency histogram. Determine the median, p95, p99, and max latencies. If the median is high, the problem is systemic. If only p99 is high, you are looking for an intermittent slow path.

Step 2 -- Apply the USE method. Check CPU, memory, disk, and network utilization, saturation, and errors using the one-liners from earlier sections. This takes under 5 minutes and immediately rules out resource-level bottlenecks.

Step 3 -- Generate flame graphs. If the USE method did not identify a resource bottleneck, generate both on-CPU and off-CPU flame graphs. The on-CPU graph reveals computational bottlenecks. The off-CPU graph reveals waiting bottlenecks (locks, I/O, network).

Step 4 -- Trace the slow path. Using the conditional tracing technique, capture stack traces only for requests exceeding your latency SLO. This filters out the noise of fast-path requests.

Step 5 -- Drill into the bottleneck. Based on the flame graph findings, attach more specific probes. If the bottleneck is database queries, trace query latency. If it is lock contention, trace futex operations. If it is network, trace TCP states and retransmissions.

Step 6 -- Verify the fix. After deploying an optimization, generate a differential flame graph comparing pre-fix and post-fix profiles. The optimization should appear as blue (reduced) frames in the differential graph.

Playbook 2: Throughput Investigation

When a service cannot handle the expected request rate:

Step 1 -- Identify the saturation point. Increase load until throughput plateaus. The resource that saturates first is the bottleneck.

Step 2 -- Check for CPU saturation. Use run queue length tracing. If the run queue is consistently deeper than the CPU count, the workload is CPU-bound. Generate a CPU flame graph to find optimization opportunities.

Step 3 -- Check for I/O saturation. Use block I/O queue depth tracing. If I/O queues are deep and latency is increasing, the workload is I/O-bound. Analyze the I/O pattern (random vs. sequential, read vs. write, size distribution) to guide optimization.

Step 4 -- Check for lock contention. Trace futex operations and mutex acquisition latency. High contention on a single lock indicates a serialization bottleneck that limits parallelism.

Step 5 -- Check for connection limits. Trace connection pool utilization and socket states. Exhausted connection pools or file descriptor limits can throttle throughput.

Step 6 -- Profile memory allocation. High allocation rates cause garbage collection pressure in managed languages and cache thrashing in native applications. Use the malloc tracing one-liner to quantify allocation rate and identify hot allocation paths.

Playbook 3: Resource Usage Investigation

When a service consumes more CPU, memory, or I/O than expected:

Step 1 -- Establish the baseline. Measure current resource consumption with eBPF probes. Compare against historical data if available.

Step 2 -- Attribute to code paths. For CPU, use a flame graph. For memory, use allocation tracing. For I/O, use block I/O tracing with process attribution.

Step 3 -- Identify waste. Look for: duplicate work (same computation performed multiple times), unnecessary I/O (reading data that is discarded), excessive allocation (allocating and immediately freeing objects), and busy-waiting (spinning in loops instead of blocking).

Step 4 -- Quantify the potential savings. Use eBPF to measure the exact contribution of each wasteful pattern to total resource usage. This provides the data to prioritize optimization efforts.

---

Performance Regression Detection

In continuous deployment environments, performance regressions can be introduced multiple times per day. Detecting them before they reach users requires automated performance testing with eBPF baselines.

Automated Baseline Collection

The first step is establishing performance baselines. eBPF probes collect latency distributions, resource utilization, and allocation rates during a representative load test:

# Baseline collection script
#!/bin/bash
DURATION=300  # 5 minutes

# CPU profile
bpftrace -e 'profile:hz:99 /comm == "myapp"/ { @[ustack] = count(); }' \
  -d $DURATION > /baselines/cpu_$(date +%Y%m%d).txt &

# Latency histogram
bpftrace -e 'uprobe:./myapp:handleRequest { @s[tid]=nsecs; }
uretprobe:./myapp:handleRequest /@s[tid]/ {
  @lat=hist((nsecs-@s[tid])/1000); delete(@s[tid]);
}' -d $DURATION > /baselines/lat_$(date +%Y%m%d).txt &

# Memory allocation rate
bpftrace -e 'uprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc /comm == "myapp"/ {
  @bytes=sum(arg0);
} interval:s:1 { print(@bytes); clear(@bytes); }' \
  -d $DURATION > /baselines/mem_$(date +%Y%m%d).txt &

wait

Statistical Comparison

Comparing the current run against the baseline requires statistical rigor. A simple approach is comparing the p99 latency with a tolerance band:

# Compare current p99 against baseline with 10% tolerance
BASELINE_P99=$(extract_p99 /baselines/lat_baseline.txt)
CURRENT_P99=$(extract_p99 /baselines/lat_current.txt)
THRESHOLD=$(echo "$BASELINE_P99 * 1.10" | bc)

if [ $(echo "$CURRENT_P99 > $THRESHOLD" | bc) -eq 1 ]; then
  echo "REGRESSION: p99 latency increased from ${BASELINE_P99}us to ${CURRENT_P99}us"
  # Generate differential flame graph for root cause
  difffolded.pl baseline.folded current.folded | flamegraph.pl > regression.svg
fi

CI/CD Integration

Modern teams integrate eBPF-based performance testing into their CI/CD pipelines. The workflow is:

1. Build the new version.
2. Deploy to a staging environment with eBPF probes pre-configured.
3. Run a standardized load test while collecting eBPF data.
4. Compare against the baseline using statistical methods.
5. Gate the deployment: if any metric exceeds the regression threshold, the pipeline fails with a differential flame graph attached to the build report.

This approach catches regressions that traditional unit and integration tests miss: a new log statement that adds 500us of I/O to every request, a library upgrade that changes the allocation pattern, or a refactoring that introduces lock contention.

---

Advanced Techniques: Combining Methods

The most effective performance analysis combines multiple methods. Here are three advanced workflows that demonstrate how the techniques in this article work together.

Workflow 1: End-to-End Request Latency Decomposition

Goal: Understand exactly where time is spent for every request across the full stack.

1. Trace the application's request handler with uprobes to get total latency.
2. Simultaneously trace database queries, network calls, and file I/O within the same process.
3. Use TSA to account for time spent off-CPU (waiting on I/O, locks, scheduling).
4. Generate a combined on-CPU and off-CPU flame graph.
5. Correlate application-level latency breakdown with kernel-level I/O and network tracing.

The result is a complete latency decomposition: 30% CPU computation, 25% database queries, 20% network calls to downstream services, 15% lock contention, 10% garbage collection. Each portion is backed by stack traces pointing to exact code locations.

Workflow 2: Memory Leak Investigation in Production

Goal: Find and fix a slow memory leak without restarting the production service.

1. Attach malloc/free trackers to the running process.
2. Collect allocation data for 10 minutes under normal load.
3. Wait 1 hour, then collect again.
4. Compare the outstanding allocation maps. Growing entries are leak candidates.
5. Examine the stack traces for growing allocations.
6. Cross-reference with code review to identify the missing free/close/release.
7. Deploy the fix and verify that outstanding allocations stabilize.

Workflow 3: Database Query Optimization

Goal: Reduce database-related latency for a service making thousands of queries per second.

1. Trace all database queries with eBPF to capture the full query text and latency.
2. Aggregate by query template to find the most expensive query types (by total time, not individual latency).
3. Generate a flame graph of the application code that triggers each expensive query type.
4. For the top queries, trace connection pool wait time to determine if connection exhaustion is a factor.
5. Trace block I/O during query execution to understand whether queries are hitting disk or cache.
6. Optimize the top queries (add indexes, rewrite queries, add caching) and verify with differential measurements.

---

Tooling Ecosystem in 2026

The eBPF performance tooling ecosystem has consolidated around several key projects that make the techniques in this article accessible to a broader audience.

bpftrace remains the go-to tool for ad-hoc analysis. Its AWK-like syntax makes it easy to write one-liners and short programs for specific performance questions. Version 0.21 (released late 2025) added improved map operations, better JIT symbol resolution, and reduced startup time.

BCC (BPF Compiler Collection) provides over 100 production-ready tools covering everything from disk I/O analysis (biolatency, biosnoop) to memory analysis (memleak, slabratetop) to network analysis (tcpretrans, tcpdrop). These tools are the Swiss Army knife of eBPF performance analysis.

libbpf and CO-RE are the foundation for building portable eBPF programs that work across kernel versions. CO-RE (Compile Once, Run Everywhere) uses BTF (BPF Type Format) to adapt programs to different kernel data structures at load time, eliminating the need to compile eBPF programs on the target system.

Parca and Polar Signals provide continuous profiling platforms built on eBPF. They collect CPU profiles from every process in a cluster continuously, store them in a time-series database, and provide a query interface for investigating performance over time.

Grafana Pyroscope integrates eBPF-based continuous profiling into the Grafana observability stack, enabling engineers to correlate flame graphs with metrics and traces in a single interface.

kubectl-bpftrace brings bpftrace into Kubernetes environments, allowing you to run bpftrace programs on specific nodes in a cluster without SSH access. This is essential for performance analysis in managed Kubernetes environments where node access is restricted.

---

Security Considerations for Production eBPF

Running eBPF programs in production requires careful attention to security and operational concerns.

Privilege requirements: eBPF programs require CAP_BPF and CAP_PERFMON capabilities (or root access). In Kubernetes environments, this is typically managed through privileged DaemonSets with appropriate RBAC controls.

Verifier safety: The eBPF verifier ensures that programs cannot crash the kernel, access unauthorized memory, or run for unbounded time. However, the verifier does not prevent programs from consuming excessive CPU through frequent probe firing. Always test probe frequency on non-production systems first.

Data sensitivity: eBPF programs can read process memory, network packets, and file contents. Ensure that the data collected by performance tracing does not include sensitive information (passwords, encryption keys, PII). Filter or redact sensitive fields in the eBPF program itself.

Overhead management: While individual eBPF probes have negligible overhead, deploying dozens of probes simultaneously can accumulate measurable impact. Monitor the overhead of your eBPF programs using bpftool prog profile and remove probes after investigation is complete.

---

Conclusion: Performance Engineering as a Practice

eBPF has transformed performance analysis from an art practiced by a few kernel experts into a systematic engineering discipline accessible to any team willing to learn the methodologies. The USE method provides a checklist that ensures no resource is overlooked. Thread State Analysis reveals where time is actually spent. Flame graphs make complex performance data visually intuitive. And bpftrace puts the power of kernel-level tracing into a scripting language that any developer can learn.

The key insight is that performance tuning is not about tools; it is about methodology. The tools in this article are powerful, but their power comes from applying them systematically: start with the USE method to identify the bottleneck resource, use TSA to understand thread behavior, generate flame graphs to pinpoint code paths, and verify optimizations with differential analysis.

As cloud-native applications grow more complex, with deeper service meshes, more distributed state, and higher throughput requirements, the need for precise performance analysis will only increase. eBPF is the foundation of this analysis, providing the ability to ask any question about system performance and get an immediate, precise answer from production systems. The engineers and teams that master these techniques will build the fastest, most efficient, and most reliable systems in production.

The bpftrace one-liners, flame graph workflows, and production playbooks in this article are starting points. Adapt them to your specific applications, build internal runbooks, and integrate eBPF-based performance testing into your deployment pipelines. Performance engineering is not a one-time activity; it is a continuous practice, and eBPF is the instrument that makes it possible.