AWS-SQS

What is AWS SQS?

Amazon Simple Queue Service is a distributed message queuing service introduced by Amazon.com in late 2004. It supports programmatic sending of messages via web service applications as a way to communicate over the Internet.

How does SQS work?

SQS It is used to decouple various components of the application. To understand in simple terms, it is like a buffered storage in between various software components.

To understand the concept how SQL works, let’s discuss with a use case.

Consider a sender component ‘A’ and receiver component ‘B’. Now A is sending a messages to B at a higher rate than the rate receiver B can process. This creates a serious problem of losing messages in transit.

To prevent messages getting lost, a SQS queue can be introduced to act as a buffer to prevent loss of messages. The messages are stored in the queue in transit and then delivered to the receiver.

This is one of the use-cases for SQS, other use cases can be decoupling the application, asynchronous processing etc.

Key features of SQS

SQS is a cloud service and can be used by any type of software, application, or other service. SQS works at its own independent service in the cloud. A software connects with SQS using a connection by passing the credentials and queue names. SQL also allows applications to create and delete custom queues.

• At-Least-Once Delivery — A message in the queue is delivered at least once. Message delivery is guaranteed, no message is lost.[Text Wrapping Break]
• Multiple components can work on a single queue. SQS uses a lock mechanism, if one component is using a message, it is made hidden to other components. Upon successful processing, message is deleted from the queue. If the message processing fails, it stays in the queue and is made visible to all the components. This feature is called Visibility Timeout.
• There are two types of queues — Standard and FIFO. In standard queue the messages are picked up randomly. It might not be in the order it entered the queue while FIFO queue uses first-in-first-out, it ensures the order.
• For the messages that cannot be processed are kept in dead-letter queue.
• Billing is done based on the number of requests to the queue. SQS is a good service to be used for applications to increase efficiency, reliability and performance.

Benefits of SQS

Here is a list of key benefits of SQS.

SQS is easy to setup without any overhead

Implementing SQS, there is no upfront cost or administrative cost. It is all configurable in AWS cloud and APIs are provide to manage queues programmatically. AWS’s pay-as-you-go model allows you to pay for what you use only.

SQL is reliable

Use Amazon SQS to transmit any volume of data, at any level of throughput, without losing messages or requiring other services to be available. SQS lets you decouple application components so that they run and fail independently, increasing the overall fault tolerance of the system. Multiple copies of every message are stored redundantly across multiple availability zones so that they are available whenever needed.

SQS is secure

You can use Amazon SQS to exchange sensitive data between applications using server-side encryption (SSE) to encrypt each message body. Amazon SQS SSE integration with AWS Key Management Service (KMS) allows you to centrally manage the keys that protect SQS messages along with keys that protect your other AWS resources. AWS KMS logs every use of your encryption keys to AWS CloudTrail to help meet your regulatory and compliance needs.

SQS is scalable and and cost-effective

Amazon SQS leverages the AWS cloud to dynamically scale based on demand. SQS scales elastically with your application so you don’t have to worry about capacity planning and pre-provisioning. There is no limit to the number of messages per queue, and standard queues provide nearly unlimited throughput. Costs are based on usage which provides significant cost saving versus the “always-on” model of self-managed messaging middleware.

Example of Amazon SQS in Action

Let’s walk through a simple example of using AWS SQS, using the line at the DMV (Department of Motor Vehicles) as the example subject matter. The DMV is notorious for long waits, forcing people to corral themselves into some form of a line. While this isn’t an actual use case anyone would (presumably) solve using AWS SQS, it will allow us to quickly demo their capabilities, with a real-world situation most are all too familiar with.

While AWS SQS has SDK libraries for almost any language you may want to use, I’ll be using their REST interface for this exercise (with my trusted REST side kick Postman!).

Authorization

Postman makes it easy to setup all the necessary authorization using Collections. Configure the AWS authorization in the parent collection with the Access Key and Secret Access Key found in the AWS Console:

Then reference that authorization in each request:

Using this pattern, it’s easy to quickly spin up requests and put AWS SQS through its paces.

Creating a Queue

When people first walk in the door, any DMV worth their salt will give them a number to begin the arduous process. This is your main form of identification for the next few minutes/hours (depending on that day’s “volume”), and it’s how the DMV employees think of you (“Number 14 over there sure seems a bit testy!”).

Let’s create our “main queue” now, with the following REST invocation:

Request:

GET https://sqs.us-east-1.amazonaws.com?Action=CreateQueue&DefaultVisibilityTimeout=0&QueueName=MainLine&Version=2012-11-05

Response:

https://sqs.us-east-1.amazonaws.com/612055710376/MainLinefa178e12-3178-5318-8d90-da20904943f0

Good deal. Now we’ve got a mechanism to track people as they come through the door.

Standard vs FIFO

One important detail that should be mentioned — there are two types of queues within AWS SQS:

• Standard — higher throughput, with “at least once delivery”, and “best effort ordering”.
• FIFO (First-In-First-Out) — not as high throughput, but guarantees on “exactly once” processing, and preserving the ordering of messages.

Long story short, if you need things super fast, can tolerate messages out of order, and possibly sent more than once, Standard queues are the answer. If you need absolute guarantees on order of operations, no duplication of work, and don’t have huge throughput needs, then FIFO queues are the best choice.

We’d better make sure we create our MainLine queue using FIFO! While a “mostly in order” guarantee might suffice in some situations, you’d have a riot on your hands at the DMV if people started getting called out of order. Purses swinging, hair pulling — it wouldn’t be pretty. Let’s add “FifoQueue=true” to the query string to indicate that the queue should be FIFO:

Request

https://sqs.us-east-1.amazonaws.com?Action=CreateQueue&DefaultVisibilityTimeout=0&QueueName=MainLineFIFO&Version=2012-11-05&FifoQueue=true

Send Message

Now that we’ve got a queue, let’s start adding “people” to it, using the “SendMessage” action. Note that when using REST, we need to URL encode the payload. So something like this:

{
"name": "Ronnie Van Zandt",
"drivers_license_number": "1234"
}

Becomes this:

%7B%0A%20%20%20%22name%22%3A%20%22Ronnie%20Van%20Zandt%22%2C%0A%20%20%20%22drivers_license_number%22%3A%20%221234%22%0A%7D

There are many ways of accomplishing this, I find the urlencoder site to be easy and painless.

Here’s the final result:

Request

https://sqs.us-east-1.amazonaws.com/612055710376/MainLineFIFO?Version=2012-11-05&Action=SendMessage&MessageBody=%7B%0A%20%20%20%22name%22%3A%20%22Ronnie%20Van%20Zandt%22%2C%0A%20%20%20%22drivers_license_number%22%3A%20%221234%22%0A%7D

Response:

00ad4e10-4394-450f-8902-4a9cf4b96b95b9f28edc9c6dc9fe2a86f5ae8efb236497a41dd4-5d15-59e0-b9f5-49e02fb4384d

After this call, we’ve got young Ronnie standing in line at the DMV. Thanks to AWS’s massive scale and performance, we can leave Ronnie there as long as we’d like. And we can add as many people as we’d like — with AWS SQS’s capacity, we could have a line around the world. But that’s horrible customer service, someone needs to find out what Ronnie needs!

Receive Message

At the DMVs I’ve been to, there’s usually a large electronic sign on the counter that will display the next lucky person’s number. You feel a brief pulse of joy when your number displays, and rush to the counter on a pillow of euphoria, eager to get on with your life. How do we recreate this experience in AWS SQS?

Why, “ReceiveMessage”, of course! (Note we are invoking it using the actual QueueUrl passed back by the CreateQueue call above)

Request

https://sqs.us-east-1.amazonaws.com/612055710376/MainLineFIFO?Action=ReceiveMessage&Version=2012-11-05

Response

00ad4e10-4394-450f-8902-4a9cf4b96b95AQEBjq8apWDfLXE0pCbpABh6Wdx70ZbszY0k38t9u8Mrny1Jz+Q522Vwvvf4xLqzQHfjoHQd56JJJEM67LJG5tQ/YSCibFSNCg8jfadyNMbqBH48/WxmpYunI3w1+GbDCL2tlKkDz/Lm9akGasgDZEBtw6U9jw1Bu6XbzNuNiw5jfVzjC99E38KSvxvZMHfmSi3Wo2XOBAcfU0oTpLmGMwccGiRUOp4XtS38nMXHhBdtKSS+U11N38cJAtlnxHQJkXmTAk7ZdvpxJNtnOrXmeGN00vtf6OSyLJzRJJieYHNtxIyxojcGZcnJQ6dTveMWQ1A1FOzschRuavl3wtftDS/YSt5sDNeBcjEOE+Y0QE+18qiWaDZc+nlaetcBvqmt6Hbtb9f28edc9c6dc9fe2a86f5ae8efb2364{
"name": "Ronnie Van Zandt",
"drivers_license_number": "1234"
}6a43b589-940c-52a4-bc62-e1bde75e22e4

One thing to keep in mind — ReceiveMessage doesn’t actually REMOVE the item from the queue — the item will remain there until explicitly removed. Visibility Timeout can be used to ensure multiple readers don’t attempt to process the same message.

So how do we permanently mark the item as “processed”? By deleting it from the queue!

Delete Message

The DeleteMessage action is what removes items from a queue. There’s not really a good analogy with the DMV here (thankfully, DMV employees can’t “delete” us), so we’ll just go with an example. DeleteMessage takes the ReceiptHandle returned by the ReceiveMessage endpoint as a parameter (once again, encoded):

Request

https://sqs.us-east-1.amazonaws.com/612055710376/MainLineFIFO?Action=DeleteMessage&Version=2012-11-05&ReceiptHandle=AQEBjq8apWDfLXE0pCbpABh6Wdx70ZbszY0k38t9u8Mrny1Jz%2BQ522Vwvvf4xLqzQHfjoHQd56JJJEM67LJG5tQ%2FYSCibFSNCg8jfadyNMbqBH48%2FWxmpYunI3w1%2BGbDCL2tlKkDz%2FLm9akGasgDZEBtw6U9jw1Bu6XbzNuNiw5jfVzjC99E38KSvxvZMHfmSi3Wo2XOBAcfU0oTpLmGMwccGiRUOp4XtS38nMXHhBdtKSS%2BU11N38cJAtlnxHQJkXmTAk7ZdvpxJNtnOrXmeGN00vtf6OSyLJzRJJieYHNtxIyxojcGZcnJQ6dTveMWQ1A1FOzschRuavl3wtftDS%2FYSt5sDNeBcjEOE%2BY0QE%2B18qiWaDZc%2BnlaetcBvqmt6Hbt

Response

a69c7042-d0e2-546a-bdf7-2476a30b89df

And just like that, Ronnie is able to leave the DMV with his newly printed license, all thanks to AWS SQS!

Companies Using SQS

680 companies reportedly use Amazon SQS in their tech stacks, including

How RedBus used SQS as a service from AWS:

The redBus is an Indian travel agency that specializes in bus travel throughout India by selling bus tickets throughout the country. Tickets are purchased through the company’s Website or through the Web services of its agents and partners. The company also offers software, on a Software as a Service (SaaS) basis, which gives bus operators the option of handling their own ticketing and managing their own inventories. To date, the company says they have sold over 30 million bus tickets and has more than 1750 bus operators using the software to manage their operations.

The Challenges

The biggest problem was that the infrastructure could not effectively handle processing fluctuations, which had a negative impact on productivity. Additionally, the procurement of servers or upgrading the server configuration was an extremely time-consuming endeavor. Over time, redBus realized that a better solution was imperative — a solution that offered scalability to handle the company’s processing fluctuations. redBus looked to Amazon Web Services (AWS) for a solution.

The Benefits

AWS the company with its many magnificent characteristics, perhaps the most significant to redBus is the ability to “instantly replicate the whole setup on-demand for testing by creating and destroying instances on demand for experimentation, thereby reducing the time to market.” Less time to market translates to increased profitability and success.

The travel agency anticipates expanding the AWS solution to include Amazon Simple Notification Service (Amazon SNS) and Amazon Simple Queue Service (Amazon SQS) for monitoring, alerts, and intercommunication. SQS helps to provide more security by securing the sensitive data of users of redBus. In addition, the increasing amount of users of redBus are benefitted from the reduction in latency provided by SQS.