Resilience
Articles on keeping Node.js services healthy under load and during deploys: back-pressure, graceful shutdown, zero-drop deploys, delivery guarantees, and what survives a Redis restart.
9 articles
When Redis restarts: persistence, eviction and reconnects for a Node.js service fleet
Sooner or later the Redis under your services restarts: an upgrade, a moved node, a process killed for memory. Whether your queued messages are still there afterwards depends on three Redis settings most teams never chose, and what your services do in the meantime depends on how the framework reconnects. Here is what we measured on a real Redis, the five lines of configuration that decide it, and what to do in code for the seconds the broker is gone.
read →Least privilege for your Redis broker: the ACL a Node.js service fleet actually needs
TLS decides who can listen to your broker. It says nothing about what a connected process may do after it is in — and on a Redis left at its defaults the answer is everything, including FLUSHALL. Here is the exact rule set an @imqueue fleet needs, the permissions that look optional and quietly are not, and how to rotate the password without a restart storm. Every behaviour is measured.
read →One event, every instance: pub/sub over a Redis broker fleet with @imqueue
You need to tell every instance of a service something at once — a price list changed, a tenant was suspended, an order was created and three concerns want to know. Pub/sub is the tool, and a fleet of brokers changes how it behaves. Here is what a clustered subscription does across several Redis brokers, how to publish and subscribe so brokers joining and leaving never cost you a handler, how to write a handler that stays correct when it gets one copy per broker, and how to see that it works.
read →Guaranteed delivery, tested with kill -9: what at-least-once actually promises when a worker dies
"Will I lose messages if a worker dies?" is the right question, and for a year this page answered it wrong. Here is what guaranteed (at-least-once) delivery really promises — measured by killing workers mid-job — what the one deadline actually governs, what a drain does and doesn't save, what it costs, and how to choose per workload.
read →Autoscale on queue depth, not CPU
CPU tells you a process is busy, not whether anyone is waiting. An @imqueue service's backlog is a Redis list, so KEDA can scale on it with no exporter, no Prometheus and no code change — and the metrics server covers the case where you need more.
read →Nothing threw, and the job never ran: silent failures in a Node.js message queue
A send() that resolves with a message id nobody will ever deliver. A push() that returns cleanly onto a queue redis refused to write. Neither throws, and neither can — the caller has already moved on by the time redis answers. That silence is over: here is what each of those failures now writes to the log instead, and the one upgrade note that costs you a stack trace.
read →One notification, every replica: the LISTEN/NOTIFY duplicate problem
LISTEN/NOTIFY is a broadcast, not a queue. Scale a Node app to three replicas and the same notification gets handled three times — no error, no warning, three charges on the card. Here's why, and what an inter-process lock actually does about it.
read →Graceful shutdown and zero-drop deploys
Every deploy sends a kill signal to a process that is probably in the middle of something. Nothing 500s, no dashboard turns red, and the work is gone anyway. Here's what actually happens to an in-flight message on SIGTERM, and the built-in drain that keeps it.
read →Back-pressure for Node.js services
When a downstream service slows down, HTTP tends to turn that into a cascading failure. A queue absorbs the spike instead. Here's the difference, and the trade-offs to watch.
read →