Observability Is a Set of Questions, Not a Stack

Most observability advice starts with tools. Install Prometheus. Add OpenTelemetry. Buy the expensive APM. Import a dashboard with 40 panels and call it done.

That advice is backwards, and the bill arrives at 2am.

This article is about what I learned building an open-source ticketing platform for events that have to stay correct under pressure. It sells seats to flash sales: a few hundred people, a popular event, fifteen-minute holds, and a hard rule that no seat is ever sold twice.

Everything below is a pattern I now reach for on any service, not just that one. There are no invented dashboard screenshots, and where a number appears it came from a run I did.

If you are new to observability, read straight through: the first half builds the mental model. If you are experienced, skip to Break It on Purpose and The Parts of SRE That Survive Contact.

---

Part 0: The Failure That Costs Money

Here is the failure mode that shaped every decision in this article.

A seat is counted in exactly one bucket: available, held, or sold. When a guest reserves, a conditional update moves a seat from available to held. A background worker returns abandoned holds to sale after fifteen minutes. When a guest confirms, the seat moves from held to sold.

Now imagine the expiry worker stops.

Nothing crashes. No exception page appears. The API keeps returning 200s. Requests are fast, because the work that stopped was never on the request path. From the outside, the system looks healthier than it did a minute ago: traffic is steady, latency is flat, error rate is zero.

Meanwhile every abandoned hold is frozen. Inventory that should have been released is not. A popular event sells out and stays sold out, and the organizer cannot explain why. By the time someone notices, the event has passed its peak.

The most dangerous incident is the one where every dashboard is green.

This is not a hypothetical. It is the alert named ExpiryWorkerHeartbeatStale, and it is the reason the section on queues below exists. A system that is only good at reporting failures it can already see will never report this one.

---

Part 1: Observability Means Answering Questions

A useful definition, and the one this article is built on:

Monitoring tells you whether something you already predicted went wrong. Observability lets you ask a question you did not think of in advance and get an answer from the data you already collected.

The practical consequence is a design method. Do not start from a list of metrics. Start from the questions an on-call engineer will actually ask, and then decide what has to be recorded for each one to be answerable.

For a ticketing platform, those questions are:

1. Is the service up, and is it the target or the process that is wrong?
2. What fraction of requests are failing, among requests that are actually flowing?
3. Which endpoint is slow, not "the API is slow"?
4. Is the database connection pool saturated, or merely busy?
5. Is the expiry worker keeping up?
6. Is scheduled work backing up, and how late is it?

Six questions. Eleven metrics. That ratio is the whole design.

Before the table, one paragraph of vocabulary, because it carries most of the beginner mistakes. Metrics come in a few shapes. A counter only goes up, so you ask for its rate over a window. A gauge is a current value, so you read it directly. A histogram records observations into buckets, so you combine it into percentiles. A label splits a metric by a dimension, and each unique combination is its own time series. That last sentence is where most first Prometheus setups get hurt.

The metrics, and what each one answers

Eleven metrics, and they cover every question above. Breadth is not coverage.

Beginner trap 1: unbounded label values

The single most common way to destroy a Prometheus instance is a label with user-supplied values.

Consider the naive version:

http_requests_total{path="/v1/events/01J9WZ6Q2CX7F2YV8AX0P4TD7L"}

One series per event. Add a few thousand events plus one series per attendee ID, and the scrape payload stops fitting in memory. The failure mode is not gradual. Prometheus dies, and your monitoring dies with the thing it was monitoring.

Record the matched route template, never the raw path:

http_requests_total{route="/v1/events/{event_id}"}

Which means this query names one endpoint rather than the API as a whole:

topk(5, job:http_request_duration_seconds:p95_5m)

That is only possible because the route template was a deliberate label design decision made before the first request was served. The rule I follow: if a label's value can be influenced by a user, it does not belong in a metric. IDs, emails, raw URLs, tenant names, request IDs, all excluded on purpose. Request IDs live in logs, where they belong, because you look them up one at a time rather than aggregating over them.

Beginner trap 2: alerting on counts instead of ratios

A first attempt at an error alert usually looks like this:

rate(http_requests_total{status_code=~"5.."}[5m]) > 0

On a system serving three requests a minute, a single failure fires this. The on-call engineer investigates, finds nothing, and learns that the alert is noise. Two weeks later they stop reading it. That is the real damage: not a false positive, but a trained-in reflex to ignore the pager.

Record a ratio, and make every error and latency alert require the system to be under real load before it can fire:

(job:http_server_errors:ratio5m > 0.05)
  and on (job) (job:http_requests:rate5m > 1)

Five percent server errors, sustained thirty seconds, above one request per second. A quiet system cannot page anyone. A busy system with a real problem can.

Beginner trap 3: three definitions of "slow"

The dashboard says p95 is 800ms. The alert fires at 1s. The SLO says 500ms. Which one is true? All three, and that is the problem. The same concept was written three times in three places, and they have already drifted.

Define the expensive reductions once, as recording rules, and let everything downstream consume them:

- record: job:http_request_duration_seconds:p95_5m
  expr: |
    histogram_quantile(
      0.95,
      sum by (job, route, le) (rate(http_request_duration_seconds_bucket[5m]))
    )

Dashboards, alerts, and ad-hoc queries all read job:http_request_duration_seconds:p95_5m. There is one definition of "p95 latency", and if it is wrong it is wrong in one place.

The sum by (route, le) is load-bearing. Drop route and you get the average of all endpoints, which is a number that describes no user's experience.

---

Part 2: The Queue Is Where Incidents Live

Request paths fail loudly. Background workers fail silently. If you take one idea from this article, take this one.

Two alerts guard the expiry worker, and at first glance they look redundant:

On a dashboard, both render as an empty graph. They are completely different incidents. One is a deployment or configuration fault: check the revision, the environment, whether the container is even running. The other is a runtime fault: check the logs, the database, whether the process is wedged.

An engineer woken by a single "worker is not reporting" alert has to guess which one they are in. Two alerts with distinct names means the alert itself tells you where to start.

time() - worker_last_batch_timestamp_seconds{worker_name="reservation-expiry"}

Holds last fifteen minutes. If that value exceeds roughly the batch interval, expired inventory is not being returned to sale, which is the silent incident from Part 0, now visible as a number.

Depth without age is a burst; age is the number that hurts

Two gauges describe the queue:

outbox_due_jobs
outbox_oldest_due_job_age_seconds

Depth tells you how many jobs are waiting. A depth of 100 after a burst of expiries is normal and drains in seconds. Depth alone cannot tell you whether anyone is being hurt.

Age tells you how long the oldest due job has been waiting. That is the number that maps to a guest's experience: it is how late a hold is released. A backlog of three jobs where the oldest has been waiting two minutes is worse than a backlog of five hundred that is draining.

This is why the oldest-job alert is critical while the depth alert is only warning. The severity difference encodes a judgment about user impact, made deliberately and written down, rather than assigned by whoever wrote the YAML.

The incident drill that would have caught it

The expiry worker's own health is also a metric, not a log line:

worker_last_batch_succeeded{worker_name="reservation-expiry"}
worker_consecutive_failures{worker_name="reservation-expiry"}

One gauge says the last batch failed. The other says how many batches have failed in a row, which is the difference between "a blip" and "this has been broken for six hours and nobody noticed."

---

Part 3: Alert Rules Are Code, So Test Them

Most teams test their application and hope their alerts are correct. An alert expression is a small program with a boolean output, and it deserves the same treatment as any other program.

Prometheus ships a test harness. This is a real test from a rules test file:

rule_files:
  - rules.yml

evaluation_interval: 5s

tests:
  - name: API target down alert waits then fires
    interval: 5s
    input_series:
      - series: up{job="api", instance="api:8080"}
        values: "1 0 0 0 0 0 0 0"
    alert_rule_test:
      - eval_time: 10s
        alertname: ApiTargetDown
        exp_alerts: []
      - eval_time: 25s
        alertname: ApiTargetDown
        exp_alerts:
          - exp_labels:
              job: api
              instance: api:8080
              severity: critical
            exp_annotations:
              summary: API scrape target is down
              description: Prometheus has failed to scrape the API for at least 15 seconds.

Read what it asserts. At ten seconds the alert must not be firing: the for: 15s clause is being tested, not just the expression. At twenty-five seconds it must be firing, with those exact labels and those exact annotations.

That second assertion matters more than it looks. Annotation text is what the on-call engineer reads at 2am. If someone rewrites the summary, this test fails, and the change is deliberate rather than accidental.

CI runs three checks:

promtool check config prometheus.yml
promtool check rules rules.yml
promtool test rules tests/api.rules.test.yml

Dashboards as code, asserted too

Grafana dashboards can be provisioned from JSON in the repository, and CI can fetch them by UID and fail if a panel was deleted or renamed:

curl -sf --user "$GRAFANA_ADMIN_USER:$GRAFANA_ADMIN_PASSWORD" \
  'http://127.0.0.1:3002/api/dashboards/uid/api-overview'

A dashboard that quietly lost the panel everyone looks at first is worse than no dashboard, because it looks fine. Asserting the dashboard exists, by the identifier humans use to find it, is cheap and catches the drift.

---

Part 4: Break It on Purpose

An alert that has never fired is decoration.

This is the most useful sentence in the runbook, and it is the difference between having alerting and believing you have alerting.

Have five drills. Each produces exactly one alert, then proves recovery. Two of them, in full:

Drill 1: stall the expiry worker.

docker compose logs -f worker      # watch what you are about to cause
docker compose pause worker

Within about thirty seconds ExpiryWorkerHeartbeatStale fires as critical, and outbox_oldest_due_job_age_seconds climbs. Recover:

docker compose unpause worker

The alert resolves without intervention. Nobody acknowledges anything, nobody restarts anything. That is the assertion. A frozen process resumes, the heartbeat resumes, and the state machine closes itself. If the age does not fall back below thirty seconds, work genuinely backed up and you are watching a queue drain rather than an idle worker. Those are different situations that look the same on a graph.

Drill 2: make the worker's batches fail.

docker compose exec -T postgres psql -U app -d app \
  -c "ALTER TABLE orders RENAME TO orders_offline;"

ExpiryWorkerBatchFailing fires and worker_consecutive_failures rises. Crucially, ExpiryWorkerMissing does not fire, because the heartbeat is still being written. The worker is alive and failing, which is exactly the distinction those two alerts were designed to separate. The drill is what proves the separation works in practice rather than in theory.

Drill 5: take a scrape target down.

docker compose stop prometheus

ObservabilityTargetDown fires. This is the least satisfying drill, and the runbook says so, because it proves the point that makes monitoring trustworthy: a silent monitoring system alerts on its own silence instead of looking like a healthy system.

That drill exists because the alternative is genuinely dangerous. Prometheus down and everything fine produce the same dashboard: nothing. If you have never tested which one you are looking at, you do not know.

After every drill

up{job="api"}
worker_last_batch_succeeded{worker_name="reservation-expiry"}
time() - worker_last_batch_timestamp_seconds{worker_name="reservation-expiry"}

Confirm the system returned to baseline, not merely that it stopped alerting. An alert that resolves because the series disappeared has told you nothing.

---

Part 5: Logs and Traces, and the Failure Nobody Warns You About

Logs: one line in, one line out, correlated

The API writes structured JSON. The line at the end of a request carries the facts you need and nothing that identifies a person:

{"fields": {"message": "finished processing request", "status": 200, "latency": "10 ms"},
 "spans": [{"name": "http_request", "route": "/v1/events", "trace_id": "…",
            "span_id": "…", "parent_span_id": "…", "request_id": "…"}]}

Three rules are enforced here:

1. Structured, never interpolated. A log line is a JSON object so a query can filter on fields. Grepping free text works until the day you need to correlate.
2. Correlation identifiers on every line. trace_id, span_id, and a request ID in the response header (X-Request-ID) mean a support ticket can be turned into a query.
3. Zero PII, and no database detail in responses. Errors are logged with a SQLSTATE classification, such as deadlock_detected, pool_timed_out, or too_many_connections, rather than a driver string. A spike in one classification is the difference between "the database is saturated" and "the database is unreachable." Without it every 503 looks identical.

That last point is the one beginners skip. Logging the raw error message feels helpful, and it hands an attacker your schema and your query text. Classify the failure for the operator; return an opaque error to the client.

Traces: one browser request, one trace

The web tier takes a W3C traceparent header from its own span, rather than minting a new one, and passes it to the API. The result is a single trace: the web tier's request span at the root, its render and fetch spans beneath, and the API's http_request span below those. The same trace ID is on every log line either tier writes.

gcloud logging read 'jsonPayload.spans.trace_id="TRACE_ID"' --project=your-project --limit=20

That one identifier turns "the page was slow" into a waterfall showing whether the time went to rendering, the network hop, or a database query.

Two decisions worth copying:

Sampling is parent-based. The root span decides once, and every downstream span inherits the decision. Independent sampling per service produces partial traces that are worse than none, because they look complete.

Health probes are excluded on purpose. At a ten-second liveness interval, probes were 40 of 45 spans in a sampled window. The traffic worth looking at had become the exception in its own trace store. Excluding them is not an optimisation, it is signal preservation.

The trap: an export can fail silently

This one cost real time and is not in most tutorials.

The OpenTelemetry Node exporter resolves a non-2xx response from the collector to { status: 'failure' } and logs nothing. A rejected batch and a delivered batch are indistinguishable from inside the process. Both are quiet. The only difference is that one set of spans exists and the other does not.

The tier built, started, served 200s, and put nothing in the trace backend, with nothing in its logs either way. That reads like an application bug and is not necessarily one.

The fix was to make the exporter say what it is doing, at a level nobody has to remember to enable:

otlp span export enabled: service=web endpoint=https://telemetry.example.com project=your-project
otlp export failed, 12 span(s) not recorded

And a debugging order, because "no spans" has four causes that are not faults at all:

1. No enabled line. Instrumentation never ran. Export is opt-in; check the endpoint variable first.
2. enabled present, export failed present. The batch reached the exporter and was refused. The error carries the reason.
3. enabled present, no failure, still no spans. The batch never reached the exporter. Look at the flush rather than the POST: a five-second batch delay does not get spent by an instance that scaled to zero.
4. The service is simply idle. Scale-to-zero exports nothing. Send a request and look again.

Writing that order down took ten minutes and saved the next person a day.

---

Part 6: The Parts of SRE That Survive Contact

Correctness is a test, not a dashboard

The platform's defining property is that a flash sale cannot oversell. That property is not an alert. It is an assertion in the test suite, and it is verified under real concurrency.

The distinction matters:

• A dashboard showing "no oversell detected" is a claim about the metrics you happened to collect.
• A test asserting that sold + held <= capacity after 750 concurrent requests is a claim about the system.

The load test that backs this up: 750 reservation requests over 30 seconds, 600 seats, zero 5xx, zero oversell, p50 38.4 ms, p95 89.2 ms, p99 142.1 ms. The 150 rejections with HTTP 409 are a pass, not a failure. Inventory unavailable is the correct business outcome at capacity.

Which leads to a subtle SLI design point most teams get wrong:

sum(rate(http_requests_total{route=~".*/reservations.*", status=~"2..|409"}[5m]))
/
sum(rate(http_requests_total{route=~".*/reservations.*"}[5m]))

409 is counted as good. A sold-out event returning "no inventory" is the platform working. An SLI that counts it as a failure makes your error budget burn precisely when you are selling out, which is the moment you are succeeding.

SLOs are denominators

Four candidate SLIs, each with an exact denominator:

The oversell property is deliberately absent from that table. It is a correctness invariant, not a percentage. Some things should never be expressed as "99.9% of the time we do not double-book a seat."

Two rules that keep SLOs honest:

1. A low-traffic percentage is noise. Always report the request count next to the ratio. 100% of three requests is not evidence.
2. Do not exclude hard dependencies to protect the number. If payment is down and checkouts fail, the user journey failed. A budget whose denominator was chosen to look good measures nothing.

Burn rate, for when you have volume to justify it

For higher-traffic services, alert on budget consumption rather than raw thresholds. The Google SRE multi-window pattern:

Two windows, because either alone lies. A short window alone fires on a spike; a long window alone is slow to notice a total outage. Requiring both means the alert is fast and credible.

Local rules often use short for windows instead. Say so plainly: they are demonstration thresholds, accelerated for a drill, and not 30-day SLO evidence. Saying that out loud is part of the practice.

Capacity work is measurement, not intuition

The largest defect found in this codebase was not a crash. It was a sequential scan inside the expiry worker: 41 ms over a 151k-row queue, on every poll.

Bounding the candidate selection and covering the hot paths changed three numbers:

The lesson is about where to look. There is a temptation, when a system is slow, to reach for the most sophisticated layer available. But in a system whose write path serialises on a single Postgres row, kernel-level packet inspection optimises the layer that is not the constraint.

A small measurement script that builds a disposable database on the real schema, loads flash-sale volume, and prints the query plan for every hot statement is enough. Read the result as buffer counts and plan shape, not wall-clock on a laptop. Node counts transfer; a laptop's latency floor does not.

Sometimes the right decision is to decline

This deserves its own paragraph because it is the least-taught SRE skill.

An evaluation was done of migrating from Cloud Run to GKE to adopt eBPF, Cilium, and Hubble. It was declined, and the reasoning is recorded as an ADR:

• Cloud Run runs under gVisor, where CAP_BPF and raw socket access are unavailable. The benefit requires a full cluster.
• The minimum cluster cost was $60 to $150 per month against Cloud Run's idle cost of zero, for an event platform serving hundreds to thousands of attendees.
• The measured bottleneck was PostgreSQL. Optimising kernel packet routing would have optimised the layer that was not the constraint.

That ADR also records a revisit trigger: the conditions under which the decision should be reopened, so the choice can be re-examined on evidence instead of memory.

Recording what you declined, and why, is how a team avoids re-litigating the same architecture debate every quarter. It is also how you notice when the reason has expired.

Know what is not covered

The most credible section of any runbook is the list of gaps:

• The web tier exports spans but no metrics, so its request rate comes from the platform's built-ins, with no route-level breakdown.
• No synthetic probe hits the public URL. Availability is observed by asking, not by monitoring.
• There is no centralized log retention locally. Logs live in the container backend; the runbook says so rather than implying a searchable store exists.
• No Alertmanager receiver is configured, so alerts are visible but page nobody.
• A notification channel exists, but delivery is unverified, and an unverified channel accepts a policy and delivers nothing.

That last one is the most instructive. "The alert fired in the console" and "a human was told" are different states, and from inside the console they are indistinguishable. Until someone confirms delivery from the recipient's inbox, you have a monitoring system, not an alerting system.

---

Part 7: A Beginner's Checklist

If you are starting from zero, this is the order I would do it in. Each step produces something you can use before the next one.

1. Write down the questions before the metrics. Six to ten questions an on-call engineer would ask. If a metric does not answer one, do not add it.

2. Instrument the request path first, with a route template. Count, duration histogram, in-flight gauge. Template the label, never the raw path.

3. Instrument the background work second. Heartbeat timestamp, last-batch success, consecutive failures, queue depth, and queue age. This is where silent failures live and where most teams have nothing.

4. Use ratios, and gate alerts on load. errors / total, and do not fire on a system nobody is using.

5. Define reductions once, as recording rules. One definition of p95, consumed by every dashboard and alert.

6. Test your alert rules. promtool test rules costs an afternoon and catches the for: clause being wrong. Assert both that it fires and that it does not fire too early.

7. Write the runbook next to the alert. Every alert names one first check. The alert and its response belong in the same table, and the alert's annotation is what someone reads half-asleep.

8. Break it on purpose, then prove recovery. Five drills, one alert each. Confirm return to baseline, not just that the alert stopped.

9. Correlate with one identifier. One trace ID on every log line and in every downstream call. Parent-based sampling so traces stay whole.

10. Make exporters observable. An exporter that fails silently will cost you a day. Log export enabled and export failed, and write down the four ways an empty trace list can be correct.

11. Write down what you declined and what you have not covered. An ADR with a revisit trigger, and a gaps list in the runbook.

12. Do not confuse correctness with reliability. The property that a seat cannot be sold twice is a test, not a dashboard. Dashboards measure; tests prove.

---

Closing

The most valuable artifact in a system like this is not a dashboard. It is a document that says, in plain language: here is what is instrumented, here is the query that proves it, here is how to break it, and here is what is still not covered.

Observability is not a stack you install. It is a set of questions you have decided to be able to answer, plus the discipline to test that you can, including when the answer is "the worker stopped an hour ago and every graph is green."

If you take one thing away, take drill 1. Stop a background worker, watch the alert fire, and wait for it to resolve on its own. It takes ninety seconds, and it will teach you more about your system than any dashboard you import.