How to monitor nginx: what stub_status tells you, and what it hides

nginx exposes seven numbers through stub_status. Six of them are traffic counters that mostly tell you how busy you are. The seventh — the gap between connections accepted and connections handled — is the only one that tells you nginx is turning traffic away, and it is the one nobody graphs.

This guide covers what those numbers mean, the failure each one predicts, and the questions stub_status genuinely cannot answer so you know when to stop asking it.

What nginx is, briefly

nginx was written by Igor Sysoev and released in 2004. It is one of the most deployed web servers on the internet — Atlassian, LinkedIn, Apple and Bleemeo among its users — maintained by F5. Its event-driven architecture is why it serves far more concurrent connections per unit of memory than a process-per-connection design like the historic Apache httpd prefork model.

That architecture is also why its monitoring surface is so small: nginx does not keep per-request state around to be queried.

The metrics that matter

Accepted versus handled: the only real alarm

stub_status reports three cumulative counters on one line: connections accepted, connections handled, and requests served.

In normal operation accepted and handled are identical. They diverge for exactly one reason: nginx reached worker_connections and closed connections it had already accepted without serving them. No error is returned to the client in a way you will see aggregated, and nothing obvious appears in the error log at the volume you would notice.

So the metric to watch is the difference, or better its rate of change. A counter that has been three apart since a spike last March is history; a difference growing right now is active traffic loss.

The fix is worker_connections, and the arithmetic has a trap. Your ceiling is worker_processes × worker_connections, but a reverse proxy consumes two connections per client — one from the browser, one to the upstream. Configure 1024 connections across 4 workers and you can serve roughly 2048 proxied clients, not 4096. Raising it also requires worker_rlimit_nofile to be at least as high, or you trade one ceiling for another and get too many open files in the error log instead.

Requests per connection

Divide requests by handled connections. This is how many requests the average client got out of one TCP connection — your keepalive effectiveness.

A value near 1 means every request is paying for a fresh connection: a TCP handshake and, over HTTPS, a TLS handshake. Usually it means keepalive_timeout is too short, or something in front of nginx is not reusing connections. For upstreams specifically, an upstream block without a keepalive directive opens a new connection to the backend for every single request, which is a surprisingly common and completely invisible tax.

Active, reading, writing, waiting

stub_status breaks active connections into three states:

• reading — nginx is receiving the request headers.
• writing — nginx is sending a response, which for a proxy includes waiting on the upstream.
• waiting — idle keepalive connections, holding a slot and doing nothing.

The useful reading is proportional. A large and growing writing count on a reverse proxy means requests are piling up against a slow backend — nginx is fine, your application is not. A large waiting count is normally healthy (keepalive working) but it consumes connection slots, so it belongs in the worker_connections calculation.

A growing reading count is rarer and more interesting: clients that opened a connection and are sending headers slowly. That is either terrible mobile networks or a slowloris-shaped problem, and client_header_timeout is the lever.

What stub_status cannot tell you

This matters more than any of the above, because it defines where monitoring nginx by its status module stops:

• No status codes. You cannot see a 5xx rate. A backend returning 502 to every request produces a perfectly healthy stub_status, because nginx handled every connection and served every request — with an error body.
• No latency. No request duration, no percentiles, nothing.
• No upstream health. Which backend is slow, which is down, how often nginx failed over — none of it.
• No per-vhost breakdown. One set of counters for the whole server.

All of that lives in the access log. $status, $request_time and $upstream_response_time are the three log variables worth having in your format if they are not already, and the difference between the last two is time spent talking to a slow client rather than waiting on your application.

Collecting it with Bleemeo

After installing nginx from your distribution’s packages and installing the Bleemeo agent, the agent auto-discovers nginx and starts port checks for service availability.

For metrics it reads nginx’s stub status module, which you enable on the default virtual host — /etc/nginx/sites-enabled/default on Debian and Ubuntu:

location /nginx_status {
    stub_status on;
}

Like any nginx location, restrict it. Status endpoints reachable from the internet leak your traffic volumes and, on some setups, more:

location /nginx_status {
    stub_status on;
    allow 127.0.0.1;
    allow 172.16.0.0/16;
    deny all;
}

Note that this endpoint has to be on the default virtual host: with server_name-based routing, a request the agent makes to 127.0.0.1 lands there, not on whichever vhost you happened to add it to. More configuration detail is in the documentation.

The service dashboard you get for free

Once the agent can read the status endpoint, a service dashboard appears:

Building the dashboard you actually want

A custom dashboard for the rest — and usefully for business metrics alongside them, so a drop in checkouts sits next to the traffic that might explain it. The example graphs:

• Status of nginx
• Number of requests per second
• Number of client connections established per second
• Number of client connections processed per second

Those last two are the accepted and handled rates. Put them on the same graph: the moment they separate is visible instantly, and separately they look identical.

The complete list of available metrics is in the documentation, and you can add anything else Bleemeo collects, including metrics from other hosts:

• CPU
• Memory
• Number of packets sent per second

Correlating HTTP traffic with system metrics is what separates “nginx got slow” from “the host started swapping at 14:03 and nginx got slow”.

Which of these deserve an alert

Requests per second is a dashboard line. It is your traffic, not your health, and a quiet night looks exactly like a broken load balancer.