Why We Built Qarote: A Queue Saturation War Story

The message came at 2:47am. Not a PagerDuty alert — a Slack message from a customer.

“Hey, are you guys having issues? Our jobs haven’t been processing for the past hour.”

An hour.

I opened the RabbitMQ management plugin. The queue depth chart was a vertical line. 847,000 messages. At our normal drain rate, that was six hours of backlog. We’d been adding to it for sixty-three of them. Our primary job queue had been filling, unattended, for over an hour. Two consumers were connected. Neither was processing anything.

I had four tabs open within thirty seconds: the management plugin, Grafana, CloudWatch, and a Slack thread where my colleagues were trying to understand the same thing from different dashboards.

Nobody knew what had happened.

(I’m Brice, co-founder and CTO of Qarote. This is the incident that led us to build it.)

What the RabbitMQ management plugin couldn’t tell me

The management plugin told me what I already knew from looking at the chart: the queue was deep, the message rate was zero, and two consumers were listed as connected.

What it couldn’t tell me:

• Which consumers were those two? Were they the workers I expected, or zombie connections from a previous deployment that hadn’t cleaned up?
• Why were they connected but not processing? Were they stuck on a single message? Crashing silently and reconnecting? Blocked waiting on a downstream service?
• When did the backlog start? The chart resolution was too coarse to pinpoint the moment things went wrong. I couldn’t tell if the accumulation started ten minutes ago or two hours ago.
• Was this isolated or cascading? I had fifteen queues. The plugin showed me one at a time.
• What did the messages look like? I couldn’t inspect a single message without writing a throwaway consumer script — which meant I couldn’t confirm whether the messages themselves were malformed or the consumers were the problem.

I switched to Grafana. The RabbitMQ Prometheus metrics we’d set up gave me time-series data for queue depth and consumer count, but the scrape interval was 60 seconds — meaning I was already seeing data that was a minute stale. And the dashboards were organized by broker, not by what I actually needed: a list of queues behaving anomalously right now.

CloudWatch had our application logs. I searched for error patterns across the consumer services. Found nothing obvious in the first pass — which either meant the consumers were healthy and the problem was upstream, or the error was being swallowed somewhere.

Forty minutes into the incident, I finally isolated the cause: a database connection pool had exhausted under load. Consumers were picking up messages, failing silently on the first DB call, nacking without logging the error properly, and then requeueing — creating a tight queue backlog that looked, from the management plugin, like “two consumers connected, zero throughput.”

The fix took five minutes. The diagnosis took forty.

The real problem

I wrote the post-mortem at 5am. The root cause section took two sentences. The “how did we miss this for an hour” section took two pages.

We had every metric in the building. We had zero diagnosis.

The problem wasn’t that we had bad tools. The management plugin is genuinely useful for day-to-day visibility. Prometheus + Grafana is a legitimate monitoring stack. The problem was that none of these tools were built to answer the question I was actually asking at 3am: what is wrong, right now, and what do I do about it?

The management plugin shows you state. Grafana shows you history. Neither shows you causality.

To diagnose a RabbitMQ incident properly, you need to correlate things that live in different places: queue depth with consumer health, consumer health with message ack rates, ack rates with the specific queues where consumers are stuck. You need to see which consumers are actually processing versus which ones are connected-but-frozen. You need to see whether a queue’s backlog started growing at the same time consumer count dropped, or before it — because those are different incidents.

None of that was surfaced automatically. It had to be assembled manually, tab by tab, during an incident, by someone who already knew where to look.

The standard monitoring stack for a production RabbitMQ deployment at that point looked like:

• Management plugin — web UI, decent for manual inspection
• Prometheus exporter — turns RabbitMQ metrics into scrapeable endpoints
• Alertmanager — routes metric-based alerts to Slack or PagerDuty
• Grafana — dashboards for trend analysis
• Custom scripts — usually Python, usually checking DLQ depth because none of the above do it well by default

Five tools. For one broker. Each requiring configuration, maintenance, and someone who knows which tab to look at when things go wrong.

That’s not a monitoring stack. That’s archaeology.

What a purpose-built RabbitMQ monitoring tool needs

After the incident, I started designing what a tool would look like if the only question it had to answer was: what’s wrong right now?

A few things felt non-negotiable:

Zero Prometheus required. Not because Prometheus is bad — it’s excellent for general infrastructure monitoring. But standing up a full Prometheus + Grafana + Alertmanager stack just to monitor a single RabbitMQ cluster is a significant investment. For teams that aren’t already running that stack, the barrier is too high. And even for teams that are, the dashboards aren’t built around RabbitMQ semantics — they’re built around raw metrics.

Rate of change, not just depth. A queue at 50,000 messages growing at +500/sec is a 3am page in 90 minutes. The same queue shrinking at -1,000/sec is healthy. Depth alone is a lagging indicator. You need velocity to know if you’re heading toward an incident or recovering from one.

Consumer health as a first-class signal. Not just “how many consumers are connected” but “are they actually processing?” A consumer that’s connected but not acking is broken. That distinction matters and it’s invisible in most monitoring setups.

Queue history you can actually use post-incident. RabbitMQ’s built-in retention is coarse and short. What I needed that night was a 5-minute-resolution depth chart going back 7 days — something that showed me exactly when the backlog started, not just that it was currently large. That precision changes how you scope an incident: was this a slow leak or a sudden drop?

The ability to inspect messages without disturbing them. That night, I couldn’t confirm whether the messages themselves were valid without writing a throwaway consumer. I needed to peek inside the queue — read headers, check payload structure — without consuming a single message. That capability alone would have cut 15 minutes off the diagnosis.

Automated incident correlation, not manual tab-switching. The forty-minute diagnosis wasn’t forty minutes of hard thinking. It was forty minutes of assembling context that already existed in three different tools. A tool that correlates queue backlog spikes, consumer drops, and publish rate changes into a single timeline isn’t doing anything magic — it’s doing the mechanical part of incident diagnosis automatically.

Alerts that fire on leading indicators, not just lagging ones. An alert that fires when your queue hits 500,000 messages is mostly useful for confirming you’re already in an incident. An alert that fires when your queue is growing faster than your drain rate — when you have five minutes of lead time — is actually useful.

A daily health summary delivered to your inbox. Not because I want to check dashboards every morning, but because small accumulations need to be caught before they become 3am calls. A daily digest of queue depths, consumer trends, and anomalies from the past 24 hours means you never go into an incident blind.

Why we built a dedicated RabbitMQ monitoring tool

I looked at the existing options seriously before writing a line of code. There are RabbitMQ Grafana dashboard templates you can import. There are SaaS APM platforms with RabbitMQ integrations. There are commercial RabbitMQ monitoring products. And of course there’s the management plugin itself, which many teams use as their primary — and only — RabbitMQ monitoring tool.

The Grafana dashboards are a good start, but they don’t solve the causality problem — they show you metrics, not a diagnosis. Datadog and similar APMs can display queue depth charts, but they’re priced for general observability and not built around RabbitMQ semantics. The commercial products are either locked to a specific hosting provider or priced at enterprise tiers that don’t make sense for smaller deployments.

None of them were built as a RabbitMQ management plugin alternative — a replacement for the tab you have open during incidents, not an additional layer on top of your existing stack.

None of them were built from the premise that the most important thing is: tell me what’s wrong, not just what’s happening.

So we built Qarote.

Qarote is a self-hosted RabbitMQ monitoring tool that connects directly to the management HTTP API — no Prometheus plugin, no YAML, no agents to deploy. MIT-licensed core, self-hostable in about two minutes with a single Docker command.

Here’s what that incident would have looked like with Qarote running:

• Minute 1: Consumer connected-but-not-acking flag raised. Qarote polls every 15 seconds — no 60-second scrape lag.
• Minute 1: Queue growth rate alert triggered. The backlog was accelerating, not just deep.
• Minute 2: Incident Diagnosis Engine correlated the queue backlog spike, the consumer ack rate drop, and the application error spike — same 4-minute window, automatically. No tab-switching.
• Minute 5: Message Spy confirmed the messages in the stuck queue were structurally valid — payload intact, headers correct. The messages weren’t the problem. The consumers were.
• Minute 5: Narrowed to DB or network. Fix deployed by minute 10.

That’s ten minutes, not forty.

And the next morning, the Daily Digest would have included the incident in the 24-hour anomaly summary — so the whole team would have known about it before they opened Slack, without anyone having to write a post-mortem retrospective email.

If that same queue had been accumulating slowly for days before the incident — a slow leak instead of a sudden drop — Queue History would have shown it: 5-minute snapshots retained for 7 days, graphable against consumer count and publish rate. The kind of context that turns a surprise outage into a capacity planning conversation.

---

If you’re running RabbitMQ in production and you’ve had the experience of staring at the management plugin during an incident not knowing what’s actually wrong — I built this for you.

The free tier is unrestricted for a single broker. Point it at your management API and it starts working. No gatekeeping on the core.

If you want to inspect the code before you run it, it’s open source on GitHub.