Observability — Logs, Metrics & Distributed Tracing

#Introduction

A user reports "checkout is slow". The request passed through the gateway, the order, pricing, inventory and payment services and Kafka. Which one is slow? Without observability, you're guessing. Observability means being able to answer questions about a running system from the data it emits.

#The Three Pillars

They work best linked together: a dashboard shows a latency spike (metrics) → you open a slow trace (tracing) → you jump to that trace's logs (logs) by its trace ID.

#1. Logs — Centralized and Correlated

With 40 services × 3 instances, logging into machines to grep files is impossible. Ship every log line to one place, as structured JSON, with a trace ID on every line.

{"@timestamp":"2026-10-15T10:15:30.188Z","level":"ERROR","service":"payment-service","traceId":"4bf92f3577b34da6a3ce929d0e0e4736","spanId":"00f067aa0ba902b7","message":"Card declined","orderId":"ORD-1001"}

• Spring Boot 3.4+ emits structured logs with logging.structured.format.console=ecs (or logstash).
• With Micrometer Tracing on the classpath, traceId and spanId are added to the MDC automatically, so every log line can be correlated.
• Log at the right levels and never log secrets — see Logging in Spring Boot & Production.

#2. Metrics — Micrometer and Prometheus

Micrometer is the metrics facade in Spring Boot (like SLF4J for metrics). Actuator plus the Prometheus registry exposes metrics at /actuator/prometheus:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

management:
  endpoints:
    web:
      exposure:
        include: health,info,prometheus
  metrics:
    tags:
      application: ${spring.application.name}

Out of the box you get HTTP request timings, JVM memory and GC, thread pools, HikariCP connections, Kafka clients, Resilience4j and more. Prometheus scrapes each instance; Grafana draws dashboards.

Custom Business Metrics

@Service
public class CheckoutService {

    private final Counter ordersPlaced;
    private final Timer paymentTimer;

    public CheckoutService(MeterRegistry registry) {
        this.ordersPlaced = Counter.builder("orders.placed").tag("channel", "web").register(registry);
        this.paymentTimer = Timer.builder("payment.duration").publishPercentiles(0.95, 0.99).register(registry);
    }

    public void checkout(Order order) {
        paymentTimer.record(() -> paymentClient.charge(order));
        ordersPlaced.increment();
    }
}

What to Measure

Track latency as percentiles (p50, p95, p99), not averages — averages hide the slow requests users actually notice.

#3. Distributed Tracing

A trace follows one request across services. Each unit of work is a span; spans share a trace ID and form a tree:

The waterfall view in Zipkin or Jaeger shows instantly that the card gateway is the bottleneck.

Tracing in Spring Boot 3

Spring Cloud Sleuth has been replaced by Micrometer Tracing, with an OpenTelemetry (or Brave) bridge and an exporter:

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-zipkin</artifactId>   <!-- or the OTLP exporter -->
</dependency>

management:
  tracing:
    sampling:
      probability: 0.1          # trace 10% of requests in production (1.0 in development)
  zipkin:
    tracing:
      endpoint: http://zipkin:9411/api/v2/spans

Spring automatically:

• creates spans for incoming HTTP requests, RestClient / WebClient / Feign calls, @KafkaListener and KafkaTemplate, JDBC (with extra instrumentation) and @Observed methods;
• propagates the context in W3C traceparent headers — over HTTP, and in Kafka record headers when observation is enabled on the template and listener;
• puts traceId / spanId into log lines.

#Health Checks

management:
  endpoint:
    health:
      probes:
        enabled: true          # enabled automatically on Kubernetes
      show-details: when-authorized

#Dashboards and Alerting

• One dashboard per service: RED metrics, JVM, connection pools, Kafka consumer lag, circuit-breaker states.
• Alert on symptoms users feel — error rate above X%, p99 latency above the SLO, growing consumer lag — not on every CPU spike.
• SLOs (service-level objectives), e.g. "99.9% of checkout requests succeed within 1 second over 30 days", turn monitoring into clear targets, with error budgets guiding release speed.
• Every alert should link to a runbook describing what to check and how to mitigate.

#Key Takeaways

• Logs, metrics and traces answer different questions — link them with trace IDs.
• Centralize structured JSON logs with traceId on every line.
• Micrometer + Prometheus + Grafana for metrics; watch RED signals and percentiles; avoid high-cardinality tags.
• Micrometer Tracing with OpenTelemetry replaces Sleuth; context propagates through HTTP and Kafka.
• Separate liveness from readiness; alert on user-facing symptoms and SLOs.