How to monitor RabbitMQ: queue depth, blocked publishers, and consumers that left

RabbitMQ almost never crashes. What it does instead is stop accepting messages — politely, by blocking publishers when it runs low on memory or disk — and the symptom shows up in your application as requests that hang. The broker is up. The health check passes. Nothing is wrong, except that half your services are stuck waiting on a basic.publish that will never return.

That, plus queues quietly growing because a consumer died, is most of what RabbitMQ monitoring is about.

What RabbitMQ is, briefly

RabbitMQ is an open-source message broker, one of the most widely deployed, used by startups and large enterprises alike. It speaks AMQP and sits between services that produce work and services that consume it.

Which is exactly why monitoring it is different from monitoring a database: a broker’s job is to absorb imbalance between producers and consumers. Some queue depth is normal and healthy. The question is always whether the depth is draining.

The metrics that matter

Queue depth, and the number it means nothing without

The first metric everyone graphs is messages in a queue. On its own it is nearly useless — you cannot tell a healthy backlog from a stalled one.

Watch these together, per queue:

• Messages ready — waiting for a consumer to take them.
• Messages unacknowledged — taken by a consumer that has not confirmed completion.
• Consumer count — how many consumers are attached.

The combinations tell you what is actually happening. Ready climbing with consumers at zero is a dead consumer, and the most common RabbitMQ incident there is. Ready climbing with consumers attached means they are too slow or too few. Unacknowledged climbing means consumers are accepting work and then dying, hanging, or throwing before they ack — and every one of those messages will be redelivered, which is how one bad message becomes an infinite loop.

The derivative matters more than the value: publish rate versus ack rate. If publishes exceed acks for any sustained period, the queue is growing, and it will keep growing until something gives. That comparison catches trouble long before an absolute depth threshold does.

The alarms that block your application

RabbitMQ protects itself with two watermarks:

• Memory alarm, at vm_memory_high_watermark (40% of system memory by default).
• Disk alarm, at disk_free_limit.

When either trips, the broker stops reading from publishing connections. Publishers block. Not error — block, which in most client libraries means the calling thread waits. Your API starts timing out while RabbitMQ reports itself perfectly healthy, because from its point of view it is: it is doing precisely what it was configured to do.

Alert on both alarm states directly, and on the memory and disk headroom that leads to them. This is the single highest-value RabbitMQ alert, and a plain TCP check on port 5672 will never see it.

Unroutable messages

Messages published to an exchange with no matching binding are dropped — silently, unless the publisher asked for mandatory delivery or an alternate exchange is configured. A non-zero rate of unroutable messages usually means a routing key changed on one side of a deploy and not the other. The publisher is happy, the consumer is idle, and the messages are gone.

File descriptors and sockets

RabbitMQ holds a file descriptor per connection and per queue index file. Both file descriptors used and sockets used, against their limits, are worth watching: hitting the limit refuses new connections, and the default ulimit on many distributions is far lower than a busy broker needs.

Cluster partitions

In a clustered deployment, a network partition leaves nodes unable to see each other while each remains individually healthy. Depending on the configured partition-handling mode, the cluster may keep running split — two halves each believing they are authoritative.

Every node reports itself up. The only way to see it is to check the reported partitions on each node, and to compare the cluster’s node count against what you expect. This is the RabbitMQ failure most likely to be discovered by a customer rather than by monitoring.

Consumer utilisation

RabbitMQ reports, per queue, how much of the time it had messages ready but no consumer available to take them. Low utilisation with a growing queue means your consumers are the bottleneck rather than the broker — useful for knowing which side to scale.

Collecting it with Bleemeo

Once the agent is installed, it creates a service check on TCP 5672 and gathers metrics through the RabbitMQ management API using the guest user. If you have disabled guest — and on anything internet-adjacent you should have — the documentation covers configuring a different user, along with the advanced options.

Give that user the monitoring tag, which is the role RabbitMQ provides for exactly this: it can read the management API without being able to publish, consume or reconfigure anything.

Note that the port check and the management API answer different questions. The TCP check tells you the broker accepts connections; only the API tells you whether it is blocking publishers. Both matter, and the second is the one that catches real incidents.

Building a dashboard

RabbitMQ gets no default service dashboard, so a custom dashboard is where to start. The example below graphs:

• Status of RabbitMQ
• Number of messages
• Number of context switches caused by the RabbitMQ process

The full metric list is in the documentation, and you can add anything else Bleemeo collects — including metrics from other hosts, which for a cluster is the point:

• Memory
• Number of packets sent per second
• CPU

Host memory belongs on the same screen as the broker’s own numbers, because the memory alarm is a function of both.

Which of these deserve an alert

Total message count is a dashboard line, not an alert. A queue at ten thousand that drains every minute is healthy; a queue at fifty that has not moved in an hour is not.