Running a message queue
A queue decouples the thing that accepts work from the thing that does it. Which queue you want depends almost entirely on what happens when something crashes.
Choosing
- Redis lists or streams
- Fastest and simplest, and you probably already run Redis. Durability depends on your persistence settings — see Redis and caching. Right for work you can afford to lose or recreate.
- RabbitMQ
- A real broker: acknowledgements, routing, dead-letter queues, per-message durability. The default choice for task queues where losing a job matters.
- Kafka or Redpanda
- A durable log rather than a queue. Messages are retained and replayable, many consumers read independently, ordering is guaranteed per partition. Right for event streams and analytics, heavy for a job queue.
- Postgres as a queue
- With
SELECT ... FOR UPDATE SKIP LOCKEDthis is genuinely good up to a surprising volume, and it is one fewer system to run. Consider it before adding a broker.
Durability costs what it costs
In RabbitMQ, a message survives a restart only if the queue is durable, the message is persistent and the publisher waits for a confirm. Miss one of the three and you have an in-memory queue with a reassuring name.
Persistence means writing to disk on every message, so the queue becomes bound by disk latency. Put it on NVMe — see SAS, SATA and NVMe — and measure with the real message size, not a benchmark.
What it needs from the machine
- Memory for the backlog. A consumer outage means the queue grows; size for the worst case you will tolerate, and set a limit so it fails loudly rather than taking the machine. RabbitMQ's memory watermark exists for this.
- Fast disk if messages are persistent.
- Few, fast cores rather than many slow ones for most brokers.
- A private network, always. Never expose a broker port — see connecting your own servers.
The failure to design for
Consumers stop and the queue fills. That is not an edge case, it is Tuesday. Decide in advance:
- A maximum queue length, and what happens at it — reject new work, or drop the oldest
- A dead-letter queue, so poison messages leave the main flow instead of being retried forever
- Retry with increasing delays, and a cap
- Idempotent consumers, because at-least-once delivery means you will process a message twice
That last one is the most commonly skipped and the most expensive to retrofit.
Watch the right number
Queue depth and consumer lag, not CPU. A queue that is growing steadily is the earliest signal that something downstream is broken, usually long before anyone notices a symptom. Alert on it — see Prometheus and Grafana.
Clustering
Possible for all of them, and more operational work than running one well-monitored node with a tested restore. Be honest about which problem you are solving — see high availability, honestly.
Still stuck? Mail support@novogara.com — an engineer answers, at any hour. Back to the knowledge base