OpenTelemetry .NET Tutorial: Distributed Tracing Guide

Learn OpenTelemetry .NET step by step: set up distributed tracing, metrics, and logs in ASP.NET Core with runnable C# examples. Start tracing today!

A single request in a modern .NET app can pass through an API gateway, three microservices, a message queue, and a database before it returns. When that request becomes slow or fails, log files alone rarely tell you why. OpenTelemetry .NET fixes this. It gives you one vendor-neutral way to collect distributed traces, metrics, and logs from your C# applications. It works with any backend, including Jaeger, Grafana, Prometheus, Azure Monitor, Datadog, Honeycomb, and the .NET Aspire dashboard.

This guide covers OpenTelemetry in ASP.NET Core from scratch. You'll learn how distributed tracing works, how to add custom spans and metrics, how to carry trace context through message queues, and which best practices matter in production. It also explains why each piece exists, so you can make good decisions in your own systems.

What Is OpenTelemetry and Why Should .NET Developers Care?

OpenTelemetry (often shortened to OTel) is a Cloud Native Computing Foundation (CNCF) project. It defines a standard API, SDK, and wire protocol (OTLP) for telemetry data. Before OTel, each monitoring vendor shipped its own agent and SDK. Changing vendors meant rewriting instrumentation code across every service.

OpenTelemetry separates how you instrument from where the data goes. You write the instrumentation once. Switching backends then only means changing the exporter configuration.

.NET has an advantage here: the runtime already contains the OpenTelemetry concepts:

• System.Diagnostics.Activity is the .NET version of an OpenTelemetry span.
• ActivitySource is the .NET version of an OpenTelemetry tracer.
• System.Diagnostics.Metrics.Meter is the .NET version of an OpenTelemetry meter.
• ILogger works with the OpenTelemetry logging pipeline through a provider.

So your libraries don't need to reference the OpenTelemetry SDK at all. They use built-in .NET APIs, and the host application decides whether to collect and export the data. ASP.NET Core, HttpClient, SqlClient, and many other libraries already emit these signals.

The Three Pillars of Observability

• Traces show the path of a single request across services, as a tree of timed spans.
• Metrics are cheap numeric aggregates over time, such as request rate, error rate, latency percentiles, and queue depth.
• Logs are detailed, discrete events. With OTel they include the trace ID, so you can jump from a log line straight to the trace it belongs to.

How to Set Up OpenTelemetry in ASP.NET Core (Step by Step)

We'll instrument a minimal API called OrderApi. Start by installing the core packages:

dotnet new web -n OrderApi
cd OrderApi

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Instrumentation.Runtime
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol

Next, configure traces, metrics, and logs in Program.cs:

using OpenTelemetry;
using OpenTelemetry.Logs;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

const string ServiceName = "OrderApi";

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(serviceName: ServiceName, serviceVersion: "1.0.0")
        .AddAttributes(new Dictionary<string, object>
        {
            ["deployment.environment"] = builder.Environment.EnvironmentName
        }))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation(options =>
        {
            // Don't trace health-check noise
            options.Filter = ctx => !ctx.Request.Path.StartsWithSegments("/health");
        })
        .AddHttpClientInstrumentation()
        .AddSource(ServiceName))          // our custom ActivitySource
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()
        .AddMeter(ServiceName))           // our custom Meter
    .WithLogging(logging => { }, options =>
    {
        options.IncludeFormattedMessage = true;
        options.IncludeScopes = true;
    })
    .UseOtlpExporter();                   // one exporter for all three signals

builder.Services.AddHttpClient();

var app = builder.Build();

app.MapGet("/health", () => Results.Ok());
app.MapGet("/", () => "OrderApi is running");

app.Run();

Why it's set up this way:

• ConfigureResource tags every span, metric, and log with service.name. Without it, your backend shows unknown_service, and in a microservices system you can't tell the services apart.
• AddSource and AddMeter are opt-in allow-lists. The SDK ignores any ActivitySource or Meter that isn't registered. This is intentional, because it keeps unwanted library telemetry out. It's also the most common reason people say "my custom spans don't show up."
• UseOtlpExporter() sends everything over OTLP, the native OpenTelemetry protocol. By default it targets http://localhost:4317 (gRPC). You can override that with the standard OTEL_EXPORTER_OTLP_ENDPOINT environment variable, so you don't need code changes between environments.

View Your Traces Locally with the Aspire Dashboard

The quickest way to see OpenTelemetry output locally is the standalone .NET Aspire dashboard. It accepts OTLP and shows traces, metrics, and structured logs:

docker run --rm -it -p 18888:18888 -p 4317:18889 \
  --name aspire-dashboard mcr.microsoft.com/dotnet/aspire-dashboard:latest

Run your app and send a few requests. Then open http://localhost:18888 (the login token is printed in the container logs). Every incoming request now shows up as a trace, and every outgoing HttpClient call appears as a child span.

Creating Custom Spans with ActivitySource in C#

Automatic instrumentation covers the edges of your service, meaning incoming HTTP requests and outgoing calls. The interesting parts, such as pricing, validation, and inventory checks, happen inside your code. Use an ActivitySource to trace them:

using System.Diagnostics;

public static class Telemetry
{
    // One static instance per library/service; never create per request.
    public static readonly ActivitySource Source = new("OrderApi", "1.0.0");
}

public record Order(Guid Id, string CustomerTier, decimal Total, int ItemCount);

public class OrderService(ILogger<OrderService> logger, IHttpClientFactory httpFactory)
{
    public async Task<Order> PlaceOrderAsync(Order order, CancellationToken ct)
    {
        using var activity = Telemetry.Source.StartActivity("PlaceOrder");

        // StartActivity returns null when nobody is listening - always use ?.
        activity?.SetTag("order.id", order.Id);
        activity?.SetTag("order.item_count", order.ItemCount);
        activity?.SetTag("customer.tier", order.CustomerTier);

        try
        {
            await ValidateAsync(order, ct);

            var client = httpFactory.CreateClient();
            // This HTTP call becomes a child span automatically
            await client.GetAsync("https://httpbin.org/delay/1", ct);

            activity?.AddEvent(new ActivityEvent("inventory.reserved"));
            logger.LogInformation("Order {OrderId} placed for {Total}", order.Id, order.Total);

            activity?.SetStatus(ActivityStatusCode.Ok);
            return order;
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.AddException(ex);   // .NET 9+; records exception.type, message, stacktrace
            throw;
        }
    }

    private static async Task ValidateAsync(Order order, CancellationToken ct)
    {
        using var activity = Telemetry.Source.StartActivity("ValidateOrder");
        await Task.Delay(50, ct); // simulate work
        if (order.Total <= 0)
            throw new InvalidOperationException("Order total must be positive.");
    }
}

Register the service and map an endpoint:

builder.Services.AddScoped<OrderService>();

app.MapPost("/orders", async (Order order, OrderService svc, CancellationToken ct) =>
    Results.Ok(await svc.PlaceOrderAsync(order, ct)));

Why activity?. everywhere? When no listener is subscribed to the source, StartActivity returns null. This is a performance feature: uninstrumented code pays almost nothing for tracing. Treat the null case as normal and don't throw on it.

Why the using statement? Disposing an Activity stops it, records its duration, and restores the previous Activity.Current. If you forget to dispose it, the span never ends and later spans get the wrong parent.

ValidateOrder is automatically a child of PlaceOrder because Activity.Current flows through AsyncLocal across await calls. You don't need to pass parent references by hand.

Custom Metrics in .NET with Meter, Counter, and Histogram

Traces answer "what happened to this request?" Metrics answer "how is the system doing overall?" Metrics are pre-aggregated in memory, so they cost the same whether you handle ten requests or ten million. That makes them the right basis for dashboards and alerts.

using System.Diagnostics.Metrics;

public class OrderMetrics
{
    private readonly Counter<long> _ordersPlaced;
    private readonly Histogram<double> _orderValue;

    public OrderMetrics(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create("OrderApi");

        _ordersPlaced = meter.CreateCounter<long>(
            "orders.placed", unit: "{order}", description: "Number of orders placed");

        _orderValue = meter.CreateHistogram<double>(
            "orders.value", unit: "USD", description: "Order total value");
    }

    public void RecordOrder(Order order)
    {
        var tier = new KeyValuePair<string, object?>("customer.tier", order.CustomerTier);
        _ordersPlaced.Add(1, tier);
        _orderValue.Record((double)order.Total, tier);
    }
}

// Program.cs
builder.Services.AddSingleton<OrderMetrics>();

Why IMeterFactory? Since .NET 8, creating meters through dependency injection keeps them scoped to the service provider. This makes unit testing much easier, because you can check recorded values with MetricCollector<T> from Microsoft.Extensions.Diagnostics.Testing without global static state leaking between tests.

Log Correlation: Connecting ILogger to Traces

Because we called WithLogging, every ILogger call made while an Activity is active automatically includes TraceId and SpanId. In your observability tool, you can open a failing trace and see the exact log lines from that request across every service. This removes most of the manual grep work from debugging a distributed system.

Keep using structured logging with message templates ("Order {OrderId} placed") instead of string interpolation. The OTLP exporter sends OrderId as a separate attribute that you can search and filter on.

Advanced: Propagating Trace Context Across Message Queues

HTTP propagation is automatic: the HttpClient instrumentation adds the W3C traceparent header, and ASP.NET Core reads it on the receiving side. Message brokers such as RabbitMQ, Azure Service Bus, Kafka, or a custom queue don't always do this for you. If the context isn't passed along, the trace breaks into disconnected pieces. Here's how to inject and extract the context yourself:

using System.Diagnostics;
using OpenTelemetry;
using OpenTelemetry.Context.Propagation;

public record QueueMessage(string Body, Dictionary<string, string> Headers);

public static class MessagingTelemetry
{
    private static readonly TextMapPropagator Propagator = Propagators.DefaultTextMapPropagator;

    // Producer side
    public static QueueMessage CreateMessage(string body)
    {
        using var activity = Telemetry.Source.StartActivity("orders publish", ActivityKind.Producer);

        var headers = new Dictionary<string, string>();
        var context = activity?.Context ?? Activity.Current?.Context ?? default;

        Propagator.Inject(new PropagationContext(context, Baggage.Current), headers,
            static (carrier, key, value) => carrier[key] = value);

        return new QueueMessage(body, headers);
    }

    // Consumer side
    public static void Process(QueueMessage message)
    {
        var parent = Propagator.Extract(default, message.Headers,
            static (carrier, key) => carrier.TryGetValue(key, out var v) ? new[] { v } : Array.Empty<string>());

        Baggage.Current = parent.Baggage;

        using var activity = Telemetry.Source.StartActivity(
            "orders process", ActivityKind.Consumer, parent.ActivityContext);

        activity?.SetTag("messaging.system", "custom-queue");
        // ... handle the message
    }
}

Setting ActivityKind.Producer and ActivityKind.Consumer matters. Backends use the span kind to draw service maps and to calculate queue latency correctly. Before writing this yourself, check whether your broker's SDK already supports OpenTelemetry. Recent Azure Service Bus, MassTransit, and NServiceBus versions emit their own activities, and you only need to register their source names with AddSource.

OpenTelemetry .NET Best Practices for Production

1. Sample Traces Deliberately

Tracing every request in a high-traffic system is expensive to store and send. Use a parent-based ratio sampler so each trace is either fully kept or fully dropped across all services:

.WithTracing(tracing => tracing
    .SetSampler(new ParentBasedSampler(new TraceIdRatioBasedSampler(0.10))) // keep 10%
    // ...
)

If you need to always keep errors and slow requests, use tail-based sampling in the OpenTelemetry Collector. Your app can't know at the start of a request whether it will fail, but the Collector can decide after the trace is complete.

2. Send Data Through the OpenTelemetry Collector

In production, export to a local Collector (sidecar or agent) instead of directly to a vendor. The Collector handles batching, retries, sampling, PII scrubbing, and sending to multiple backends. Your app stays simple, and you can change vendors without redeploying code.

3. Follow Semantic Conventions

Use the standard attribute names (http.request.method, db.system, messaging.system) where they apply, and add a consistent prefix to custom ones (order.id). Backends build dashboards and alerts on top of these conventions.

4. Configure with Environment Variables

OpenTelemetry .NET supports the standard OTEL_* environment variables, such as OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, and OTEL_RESOURCE_ATTRIBUTES. Keep API keys and endpoints out of source code, and let Kubernetes or your cloud platform supply them.

Common Pitfalls (and How to Avoid Them)

• Custom spans are missing. The ActivitySource name passed to AddSource() doesn't match exactly. The match is case-sensitive, so use a shared constant.
• Creating an ActivitySource or Meter per request. Both are meant to live for the whole app. Creating them repeatedly leaks listeners and hurts performance. Make them static readonly or get them from DI.
• High-cardinality metric tags. Never tag metrics with user IDs, order IDs, or full URLs. Each unique tag combination creates a new time series, which can overload your metrics backend and your bill. High-cardinality values belong on spans, not metrics.
• Leaking PII. Emails, tokens, and card numbers in span tags or log attributes end up in third-party storage. Scrub them in code or in the Collector, which matters for GDPR, CCPA, and HIPAA compliance.
• Broken traces in background work. Task.Run keeps Activity.Current, but a fire-and-forget job that outlives the request produces confusing traces. Start a new root activity with a link back to the original using ActivityLink.
• Exporting to the console in production. The console exporter writes synchronously and is slow. Use it only for local debugging.
• Tracing health checks and static files. Filter them out, as shown earlier, so they don't flood your trace storage.

Conclusion: Key Takeaways for OpenTelemetry .NET

OpenTelemetry .NET is now the standard way to add observability to C# applications, and it's easy to adopt because the concepts are built into the .NET runtime. With a few lines of configuration you get distributed tracing across your microservices, correlated logs, and production-ready metrics, without tying yourself to one vendor.

• OpenTelemetry spans are Activity in .NET, tracers are ActivitySource, and meters are Meter. Your libraries don't need an SDK dependency.
• Always call AddService(), and register every custom source and meter with AddSource()/AddMeter().
• Use UseOtlpExporter() with OTEL_* environment variables so configuration stays portable.
• Put high-cardinality detail on spans and keep metric tags low-cardinality.
• Propagate context manually across queues that don't support OpenTelemetry, and use correct span kinds.
• In production, sample deliberately and route telemetry through the OpenTelemetry Collector.

Start small: add the ASP.NET Core and HttpClient instrumentation to one service today and view it in the Aspire dashboard. Then add custom spans where your business logic runs. After you've followed a slow request across five services in one view, you won't want to go back to reading log files.

Tags: #OpenTelemetry C# #distributed tracing .NET #ASP.NET Core observability #OpenTelemetry tracing tutorial #OTLP exporter .NET #.NET metrics and logging

About csharp-coder.com
Your go-to resource for C#, .NET, and modern software development. Follow along for daily tutorials, tips, and real-world examples.