Monitoring Fundamentals in Six Lessons: From Golden Signals to Error Budgets

Summer is the perfect time to step back from the alert stream and revisit the ideas your whole monitoring practice quietly rests on. Not the newest tool or the shiniest dashboard — the fundamentals: what to measure, how to read it, when to wake someone up, and what to do when something breaks at 2 a.m.

We packed six of those fundamentals into a free, no-signup PDF — The DevOps & Monitoring Summer Workbook, complete with hands-on exercises and an answer key. This article is the reader’s-digest version: the core idea from each lesson, and why it matters. Grab an iced coffee and let’s refresh.

Lesson 1 — The four golden signals

Google’s SRE book distilled service health into four numbers. Watch these first; everything else is detail:

• Latency — how long requests take (report it as a percentile, not an average — more on that below).
• Traffic — how much demand you’re serving (requests per second, throughput).
• Errors — how often requests fail (the ratio of failed to total requests).
• Saturation — how full your resources are (CPU, memory, disk usage).

The reason this list is so durable is the split behind it: latency and errors describe what your users feel, while traffic and saturation describe what your system is doing. If you could only build four graphs for a service — even one you’ve never seen before — build these four. You can triage almost anything from them.

Lesson 2 — The three pillars of observability

Monitoring tells you when something is wrong. Observability helps you understand why. It rests on three data types, and the trap is thinking you must pick one:

• Metrics — cheap numbers over time. Great for dashboards and alerts. They tell you that something broke.
• Logs — timestamped events with rich context. They tell you what broke.
• Traces — a single request followed across services. They tell you where the time went.

The practical workflow is to move through them in order: start at the metric (an alert fires, a graph spikes), drill into the trace to find which service or call is slow, then confirm in the logs with the exact error. Each pillar answers a question the other two can’t — you need all three.

Lesson 3 — Why averages lie

If your average response time is 190 ms, are your users happy? You can’t tell. An average blends the fast majority with the slow tail into one comfortable, misleading number.

Percentiles put the pain back in view. The p95 is the value 95% of requests come in under; the p99 is the value 99% come in under. The p99 is where your angriest users live — and where a single slow database query hides.

Consider a real distribution: average = 190 ms, p50 = 120 ms, p95 = 840 ms, p99 = 2.1 s. The average says “190 ms, all good.” But one user in a hundred is waiting over two seconds. That’s the story the average erases.

The rule: alert and report on percentiles, not averages. “p95 latency under 300 ms” is a promise you can keep and verify; “average latency” is a number you can hide behind.

Lesson 4 — Anatomy of a good alert

The fastest way to be ignored is to alert too often. A good alert has four parts:

1. It watches a signal that matters.
2. It compares that signal to a sensible threshold.
3. It waits long enough — for a duration — to avoid reacting to momentary spikes.
4. It delivers an actionable notification: what broke, where, and how bad.

Anything else is noise, and noise trains your team to swipe alerts away. The single most important habit here is to alert on symptoms users feel, not on every internal cause. One alert for “checkout error rate > 5% for 5 minutes, paged to on-call with a dashboard link” beats twenty separate CPU alerts that nobody reads.

A quick gut check — which of these are good alerts, and which are noise?

• CPU went above 90% for one 10-second spike at 3 a.m. → noise (a momentary blip, no sustained impact).
• An alert whose entire message is “Something is wrong.” → noise (no context, not actionable).
• API 5xx rate > 5% for 5 minutes, paged to on-call with a dashboard link. → good (symptom-based, sustained, actionable).
• Every single failed request pages the whole team. → noise (this is how alert fatigue is born).

Lesson 5 — SLIs, SLOs and the error budget

This is where “we should be more reliable” becomes something you can actually measure — and spend.

• An SLI (Service Level Indicator) is a measurement, e.g. the percentage of successful requests.
• An SLO (Service Level Objective) is the target you set for it, e.g. 99.9% over 30 days.
• The gap between 100% and your SLO is your error budget: the amount of unreliability you’re allowed.

The maths is refreshingly concrete. Downtime allowed = (1 − SLO) × time period, and a 30-day month has 43,200 minutes:

• A 99.9% SLO allows ~43 minutes of downtime a month.
• A stricter 99.99% SLO allows only ~4.3 minutes.

The budget is a decision tool. Budget left → ship features and take risks. Budget spent → stop shipping and focus on reliability until it recovers. And beware the trap of a 100% SLO: it forbids all change, because any risk could break it. A realistic SLO gives you room to move fast and a clear signal for when to slow down.

Lesson 6 — The incident lifecycle

When something breaks at 2 a.m., a clear sequence beats heroics. Every incident follows the same arc:

1. Detect — an alert fires.
2. Triage — assess who and what is affected, and how badly.
3. Mitigate — stop customer pain, often with a rollback.
4. Resolve — fix the true root cause.
5. Learn — write a blameless postmortem so it can’t happen the same way twice.

The counter-intuitive, load-bearing rule: mitigate before you diagnose. Stopping the bleeding — a rollback, a feature flag, a failover — comes first. The perfect root-cause fix can wait until customers are safe. Two numbers keep you honest along the way: MTTA (mean time to acknowledge) and MTTR (mean time to resolve).

Bonus: what kind of SRE are you?

The workbook ends with a not-at-all-scientific personality quiz. Are you the Firefighter (calm in chaos, unbeatable under pressure), the Dashboard Artist (every metric has a home and a colour), the Zen Master (you’d rather delete a problem than fix it twice), or the Guardian (SLOs, runbooks and quiet nights)? Most of us are a blend — and the honest answer usually depends on how the last on-call went.

Take the workbook to the beach

These six ideas — golden signals, the three pillars, percentiles, alerting, error budgets and the incident lifecycle — are the fundamentals every reliable team leans on. Reading them is a good refresher; practising them is how they stick.

That’s what the workbook is for. Each lesson comes with a reusable schema, a hands-on exercise, and a worked solution in the answer key — no email, no form, just a PDF.

👉 Download The DevOps & Monitoring Summer Workbook (free PDF)

And if you’d rather have all of this running for you — golden-signal dashboards, percentile-based alerts and auto-discovery out of the box — that’s exactly what Bleemeo does. Start a free trial: one agent, a few minutes, no stack to maintain.