RabbitMQ vs AWS SQS

You, • development
Back

Recently, I had the opportunity to debug an issue rising from RabbitMQ and refresh my memory on its implementation. After working with AWS SQS for awhile,

Context

RabbitMQ

The legacy part of the code makes use of RabbitMQ. We have self-hosted RabbitMQ servers and makes use of rex as an intermediary service that connects to the queue, listens for messages, forwards the message to another CLI software (in our case, our API) and acknowledges the message depending on the execution result.

AWS SQS

Since we migrated to AWS Cloud services, we started taking advantage of utilizing several AWS's managed services - and AWS SQS is one of them. One of the main features that I implemented (asynchronous export1) is designed with AWS SQS.

Pros and Cons

Administration [Winner - SQS]

There is a bit of steep learning curve for administering and understanding how RabbitMQ is set up as it's most likely self-hosted. (There is Amazon MQ if you wish to have it managed by AWS.) I had to take some time to locate the RabbitMQ clusters in order to access its management interface. A detailed documentation is a must especially if RabbitMQ is a part of legacy code as its likely set up manually rather than using code to manage cloud infrastructures like AWS CDK or Terraform.

Accessing AWS SQS management interface is very straightforward as all the services managed by AWS is consolidated under single AWS Management Console.

Coding [SQS]

This is a close one. Both services, being established and widely used, have a great resources, documentation and a plenty of examples. However in this specific case, the consumer logic interacting with the queue, especially around positive/negative acknowledgment of message, was abstracted in rex and out of the codebase - making it challenging to debug consumer-specific issue.

AWS SQS offers SDKs in all major programming languages including PHP. Easy to follow both publish/consume logic available right in the codebase.

Purging Messages [SQS]

RabbitMQ's purge does not delete unacknowledged messages. This means that if you somehow run into a bug where queue is blocked2 due to a pile of unacknowledged messages, the only way to quickly remove3 stuck messages are either close the channel and purge the queue or delete and recreate the queue - both very dangerous in production.

AWS SQS handles the unacknowledged messages differently using visibility timeout, separate DeadLetter queue (if you use one) and DeadLetter Receive count. AWS SQS's purge does exactly what it's told - it simply clears all messages in the queue.

  1. Case Study available here↩
  2. RabbitMQ has a configuration for consumer prefetch count which helps to limit the number of unacknowledged message on a channel. Once the number of unacknowledged message equals the set count, the queue would stop releasing the Ready messages.↩
  3. The fact that there is stuck, unacknowledged message in the queue means that there is probably a consumer stuck recycling the message due to a processing error which would most definitely require a code change. However, you might want to briefly unblock the queue by clearing the queue depending on the situation.↩
© Hannah Kim.