Kubernetes Beginner Series 11 — Observability: Logs, Metrics, and Traces

Table of contents

• Operations Without Observability Is Gambling
• Logs — How They Flow in Kubernetes
  • Log Collection Pipeline
  • Structured Logging
• Metrics — Prometheus and Grafana
  • Installation — kube-prometheus-stack
  • ServiceMonitor — Registering App Metrics
  • PromQL — The Metrics Query Language
  • Alerts — PrometheusRule
• Distributed Tracing — Jaeger and OpenTelemetry
  • OpenTelemetry
  • Jaeger Installation
• The Fourth Axis — Events
• Practical kubectl Debugging
  • Pod Status Inspection
  • Log Tracing
  • Getting Inside a Container
  • No Shell in the Image? Use kubectl debug
  • Checking Resource Usage
  • Port Forwarding and Proxy
• Connecting the Three Axes
• Where to Start

Operations Without Observability Is Gambling

As a Kubernetes cluster grows more complex, it becomes harder to understand “what’s running and how.” Pods die and respawn, so everything looks fine on the surface. Traffic surges are masked by autoscaling. By the time a problem becomes visible, users have already been affected.

Observability is the tool that breaks through this opacity. It collects the three signals a system emits — logs, metrics, and traces — and correlates them so you can explain “why is the system in this state right now.”

flowchart LR
    A[Logs] -->|"What happened"| D[Observability Platform]
    B[Metrics] -->|"How much is<br/>being used"| D
    C[Traces] -->|"How did the<br/>request flow"| D
    D --> E[Dashboards]
    D --> F[Alerts]
    D --> G[Debugging]

Each signal answers different questions. You need to correlate all three to understand “why did that outage just happen.”

Logs — How They Flow in Kubernetes

Container logs should go to standard output (stdout/stderr) by design. If you write to files the old-fashioned way, those files are trapped inside the container filesystem. When the Pod dies, the logs die with it.

Kubernetes stores the logs that containers write to stdout as files in the node’s /var/log/containers/ directory. kubectl logs is a wrapper that reads those files.

# Logs from a specific Pod
kubectl logs my-pod

# Logs from a specific container in a multi-container Pod
kubectl logs my-pod -c sidecar

# Real-time follow
kubectl logs -f my-pod

# Logs from multiple Pods by label (-l + --tail=N)
kubectl logs -l app=web --tail=100 -f

# Previous container logs (after a crash)
kubectl logs my-pod --previous

kubectl logs is handy for debugging, but insufficient for operations. When the node rotates log files, old logs disappear. It’s hard to interleave logs from multiple Pods in chronological order, and there’s no search capability.

Log Collection Pipeline

In production, you deploy a log collection agent as a DaemonSet on each node, shipping the node’s log files to a central store.

flowchart LR
    A[App Pod<br/>stdout] --> B[Node filesystem<br/>/var/log/containers/]
    B --> C[Fluent Bit<br/>DaemonSet]
    C --> D[Elasticsearch]
    C --> E[Loki]
    C --> F[CloudWatch / GCP]
    D --> G[Kibana]
    E --> H[Grafana]

Several popular combinations exist:

• EFK stack: Elasticsearch + Fluentd/Fluent Bit + Kibana. The traditional ELK-based approach
• Loki stack: Loki + Promtail + Grafana. Indexes logs by Prometheus-style labels for lightweight operation
• Cloud managed: CloudWatch Logs, GCP Logging, Azure Monitor. Simple to set up but comes with cost

If you’re setting up Kubernetes for the first time, Loki is a good choice for experimentation. Since it only indexes labels, storage costs are low, and it integrates natively with Grafana so you can view metrics and logs side by side.

A simple Fluent Bit configuration looks like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Parsers_File parsers.conf

    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        Parser docker
        Tag kube.*
        Refresh_Interval 5

    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off

    [OUTPUT]
        Name loki
        Match *
        Host loki.logging.svc.cluster.local
        Port 3100
        Labels job=fluentbit, namespace=$kubernetes['namespace_name']

The key point is that the Kubernetes filter attaches metadata like Pod labels and namespace to each log entry. Later, you can easily filter for “only logs from the api Pod in the backend namespace.”

Structured Logging

To make logs truly useful, you should emit them in a structured format (JSON). Plain text is readable by humans but difficult to search and aggregate.

{
  "timestamp": "2026-04-20T10:15:32Z",
  "level": "ERROR",
  "service": "payment-api",
  "trace_id": "abc123def456",
  "user_id": "user-9284",
  "message": "Payment failed",
  "amount": 15000,
  "error": "Card declined"
}

Note the trace_id included here. When correlating with distributed tracing later, this single ID lets you jump between logs and traces.

Metrics — Prometheus and Grafana

Source: Prometheus official repo — Apache 2.0 License

Metrics are numerical data that change over time. Things like “request count per minute,” “current memory utilization,” and “error count.” In the Kubernetes ecosystem, Prometheus is the de facto standard for metrics.

Prometheus works on a pull model. Each Pod exposes metrics at an HTTP endpoint (/metrics), and Prometheus periodically scrapes them.

flowchart LR
    A[App A<br/>/metrics] -->|scrape| D[Prometheus]
    B[App B<br/>/metrics] -->|scrape| D
    C[kube-state-metrics<br/>/metrics] -->|scrape| D
    E[node-exporter<br/>/metrics] -->|scrape| D
    D --> F[Alertmanager]
    D --> G[Grafana]

Installation — kube-prometheus-stack

The fastest path is the official Helm chart. It includes the Prometheus Operator, which lets you manage scraping configuration via CRDs.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set grafana.adminPassword='admin123'

After installation, verify with port forwarding:

# Prometheus
kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090

# Grafana
kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80

This single chart installs Prometheus + Alertmanager + Grafana + node-exporter + kube-state-metrics all at once. Node resources, Pod status, container metrics, and kubelet metrics are collected by default, and dozens of Kubernetes dashboards come pre-installed in Grafana.

ServiceMonitor — Registering App Metrics

With the Prometheus Operator, configuring Prometheus to scrape your app’s /metrics endpoint is as simple as creating a ServiceMonitor CRD.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: myapp
  namespace: backend
  labels:
    release: monitoring    # Label that kube-prometheus-stack looks for
spec:
  selector:
    matchLabels:
      app: myapp
  endpoints:
    - port: http
      path: /metrics
      interval: 15s

Once applied, Prometheus begins scraping the port of Services labeled app=myapp. On the application side, add a Prometheus client library (Spring Boot Actuator, prom-client, prometheus_client, etc.) to expose /metrics.

PromQL — The Metrics Query Language

Prometheus metrics are queried using PromQL. It feels unfamiliar at first, but a few patterns cover most questions:

# Current CPU utilization per node
sum by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))

# Total memory usage by namespace
sum by (namespace) (container_memory_working_set_bytes{container!=""})

# 5-minute average HTTP request rate (per Pod)
sum by (pod) (rate(http_requests_total[5m]))

# Error rate (5xx ratio)
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

# Pods that entered CrashLoopBackOff in the last hour
kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"} == 1

rate() converts counter metrics into a per-second increase rate, and sum by aggregates by label. These same expressions are used when defining alerts.

Alerts — PrometheusRule

Define PrometheusRule to send alerts when thresholds are exceeded:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: app-alerts
  namespace: backend
  labels:
    release: monitoring
spec:
  groups:
    - name: app.rules
      rules:
        - alert: HighErrorRate
          expr: |
            sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
            /
            sum(rate(http_requests_total[5m])) by (service)
            > 0.05
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "{{ $labels.service }} error rate exceeds 5%"
            description: "5xx ratio over the last 5 minutes: {{ $value | humanizePercentage }}"

        - alert: PodCrashLooping
          expr: |
            rate(kube_pod_container_status_restarts_total[15m]) > 0
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: "{{ $labels.pod }} is crash-looping"

for: 5m means “the condition must remain true for 5 minutes before firing.” This prevents noisy alerts from momentary spikes. Alertmanager routes these alerts to Slack, PagerDuty, email, and more.

Distributed Tracing — Jaeger and OpenTelemetry

As the number of microservices grows, a single request passes through multiple services. With logs alone, it’s hard to tell “where did those 500ms go.” Distributed tracing solves this problem.

Each service attaches a trace_id and span_id when processing a request and passes them to the next service. All spans from all services are collected and reassembled.

flowchart TD
    A[Request in: trace_id=abc] --> B[Gateway<br/>span 1]
    B --> C[Auth Service<br/>span 2]
    C --> D[Order Service<br/>span 3]
    D --> E[Payment Service<br/>span 4]
    D --> F[Inventory Service<br/>span 5]
    E --> G[DB<br/>span 6]

Looking at a trace for a single request, you can see the call graph between services and the duration of each step at a glance. It immediately reveals which service is slow and whether there are unnecessary calls.

OpenTelemetry

Previously, Jaeger and Zipkin each had their own SDKs. Now OpenTelemetry (OTel) is the standard. Instrument your code with the OTel SDK, and the data format is unified under the OTel Protocol (OTLP). Choose any collection backend — Jaeger, Tempo, Honeycomb, or whatever you prefer.

The basic architecture looks like this:

flowchart LR
    A[App A<br/>OTel SDK] --> C[OTel Collector<br/>DaemonSet or Deployment]
    B[App B<br/>OTel SDK] --> C
    C --> D[Tempo / Jaeger]
    C --> E[Prometheus]
    C --> F[Loki]
    D --> G[Grafana]

The OTel Collector acts as an agent. Apps send data to the Collector, which processes, filters, and routes it to backends. The apps don’t care what the backend is — they only talk to the Collector.

Jaeger Installation

The Jaeger Operator is the simplest approach:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
  namespace: tracing
spec:
  strategy: production
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://elasticsearch.logging:9200

This deploys all of Jaeger’s components (Collector, Query, Agent) at once. Configure your apps to send data via OTLP, and search traces in the Jaeger UI.

The Fourth Axis — Events

Beyond logs, metrics, and traces, Kubernetes has one more signal unique to itself: Events.

# All events in a namespace
kubectl get events -n backend --sort-by='.lastTimestamp'

# Events related to a specific Pod
kubectl describe pod my-pod | tail -20

# Real-time event monitoring
kubectl get events --watch

Events capture Kubernetes-internal signals like Pod scheduling failures, image pull errors, OOMKilled, and health check failures. They’re the first clue in incident troubleshooting. It’s a good idea to collect and retain events like metrics, since by default they’re deleted after 1 hour. Use tools like kube-state-metrics or Event Exporter for long-term retention.

Practical kubectl Debugging

Let’s organize the commands frequently used during actual incidents. Memorizing this list lets you establish a direction within 5 minutes for most problems.

Pod Status Inspection

# Quickly find problem Pods
kubectl get pods -A | grep -vE "Running|Completed"

# Pod detailed status and events (the first command to run)
kubectl describe pod my-pod -n backend

# Why a container restarted
kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[*].lastState}'

# Full spec as JSON
kubectl get pod my-pod -o json | jq '.status'

The events section at the bottom of describe is especially useful. It immediately shows whether it’s a scheduling failure, image pull failure, or health check failure.

Log Tracing

# Chronological logs from multiple Pods by label
kubectl logs -l app=web --tail=100 --prefix=true

# Logs since a specific time
kubectl logs my-pod --since=10m
kubectl logs my-pod --since-time="2026-04-20T10:00:00Z"

# All container logs when sidecars are present
kubectl logs my-pod --all-containers=true

# Logs from the previously terminated container (during CrashLoopBackOff)
kubectl logs my-pod --previous

Getting Inside a Container

# Shell into a running container
kubectl exec -it my-pod -- /bin/bash
kubectl exec -it my-pod -c sidecar -- sh

# Run a single command
kubectl exec my-pod -- env
kubectl exec my-pod -- cat /etc/config/app.yaml

No Shell in the Image? Use kubectl debug

distroless or scratch images don’t have a shell. In that case, kubectl debug lets you attach a debugging sidecar.

# Temporarily add a debugging container to the same Pod
kubectl debug -it my-pod --image=busybox --target=my-container

# Launch a new Pod as a copy of the existing one with a different image
kubectl debug my-pod --copy-to=my-pod-debug --image=busybox -- sleep 3600

Network debugging is also possible:

# Spin up a temporary Pod on a node for network inspection
kubectl debug node/node-1 -it --image=nicolaka/netshoot

The netshoot image comes with networking tools like dig, curl, nslookup, and tcpdump, making it invaluable for diagnosing cluster network issues.

Checking Resource Usage

# Node resources
kubectl top nodes

# Per-Pod resources (requires metrics-server)
kubectl top pods -A --sort-by=cpu
kubectl top pods -A --sort-by=memory

# Per-container for a specific Pod
kubectl top pod my-pod --containers

Port Forwarding and Proxy

# Direct access to a Pod port from local
kubectl port-forward pod/my-pod 8080:80

# Service port forwarding
kubectl port-forward svc/my-service 8080:80

# API server proxy (after kubectl proxy, access http://localhost:8001/api/...)
kubectl proxy

Port forwarding is used routinely in development environments for connecting directly to databases or launching dashboards like Prometheus/Grafana.

Connecting the Three Axes

The real power of observability comes from connecting the three axes. In a single Grafana screen, a workflow like this becomes possible:

1. An alert fires — “payment-api error rate exceeds 5%”
2. Check the service’s graphs in the Prometheus dashboard — CPU/memory/request rate
3. Find logs from the same time window in Loki — what error messages were logged
4. Use the trace_id from an error log to open the trace in Tempo — which downstream service caused the failure
5. Jump back to the metrics of the root-cause service

When this workflow runs in a single tool, MTTR (Mean Time To Recover) drops dramatically. Without integration, you end up bouncing between systems wondering “wait, when did that happen?” With everything connected, it’s just a few clicks.

This is why the Prometheus + Loki + Tempo + Grafana combination excels at this correlation. Common labels and trace_id links tie everything together naturally.

Where to Start

Trying to set up everything at once is a recipe for burnout. Here’s a recommended order:

1. Prometheus + Grafana: Basic cluster metrics and dashboards. kube-prometheus-stack gets you there in 30 minutes
2. Basic alerts: Pod crashes, node disk, CPU/memory thresholds. The official chart installs default alerts too
3. Log collection: Loki + Promtail. Start light with 7-day retention
4. App metrics: Start with business-critical services exposing /metrics and defining alerts
5. Distributed tracing: Consider once you have 3+ microservices. Start with the OTel SDK

You don’t need perfection from the start. Fill in the gaps as they appear — that approach lasts longer.

The final part of this series ties everything back together. Instead of managing dozens of YAML files by hand each time, we’ll look at how to package applications with Helm.

-> Part 12: Helm and Package Management