Amazon Simple Queue Service – SQS


Amazon Simple Queue Service (SQS) is a distributed queue system that enables web services to quickly and reliably queue messages that one component in the application generates to be consumed by another component. A queue is a temporary repository for messages that await processing.

HOW IT WORKS

Below sample architecture we are going to use to understand how SQS works. There are two SQS Queue created, one for Request and another for Response.

Web Server who act as producer, who creates a message and puts it into request queue. it also read from response queue as well.

You can have multiple producers who can add multiple messages to the queue at the same time. You don’t have to worry about the traffic or peaks. SQS handles that for you.

sqs-as-workflow-diagram

Queued messages are processed by consumer. The consumer is requesting new messages periodically from the queue. You can have multiple consumers, but you have to remember that each message can be processed only once. It means that you can have multiple instances of the same consumer, but you can’t read the same message from one queue in two different components. Each of these components should use a separate SQS queue.

After a consumer processes the message, it has to be deleted from the queue. Deleting is important because SQS assumes that processing can fail. To prevent that, after the consumer receives a message, it is hidden from the queue for a defined period of time and after that, if it is not deleted, the message shows up in the queue again.

HOW TO RUN MULTIPLE EC2 INSTANCES SIMULTANEOUSLY WITH AMAZON EC2 TO HANDLE INCREASED LOAD? 

Two specific features of Amazon SQS make this possible:

  • A single Amazon SQS queue can be shared by multiple servers simultaneously.
  • A server that is processing a message can prevent other servers from processing the same message at the same time by temporarily “locking” a message. The server can specify the amount of time the message is locked. When the server is done processing the message, it should delete the message. If the server fails while processing the message, another server can get the message after the lockout period.

These two features ensure that the number of processing servers can be easily changed dynamically to handle varying load. The entire process can be automated to ensure that at any given time the optimal number of EC2 instances is running. This practice is commonly referred to as “auto-scaling.”

TYPES OF QUEUE 

Standard Queue

  • Available in all regions.
  • Unlimited Throughput – Standard queues support a nearly unlimited number of transactions per second (TPS) per API action.
  • At-Least-Once Delivery – A message is delivered at least once, but occasionally more than one copy of a message is delivered.
  • Best-Effort Ordering – Occasionally, messages might be delivered in an order different from which they were sent.
  • When to use? – Send data between applications when the throughput is important, for example:
    • Decouple live user requests from intensive background work: let users upload media while resizing or encoding it.
    • Allocate tasks to multiple worker nodes: process a high number of credit card validation requests.
    • Batch messages for future processing: schedule multiple entries to be added to a database.

FIFO Queue

  • Available in the US East (N. Virginia), US East (Ohio), US West (Oregon), and EU (Ireland) regions.
  • High Throughput – FIFO queues support up to 300 messages per second (300 send, receive, or delete operations per second). When you batch 10 messages per operation (maximum), FIFO queues can support up to 3,000 messages per second. To request a limit increase, file a support request.
  • Exactly-Once Processing – A message is delivered once and remains available until a consumer processes and deletes it. Duplicates aren’t introduced into the queue.
  • First-In-First-Out Delivery – The order in which messages are sent and received is strictly preserved.
  • When to use – Send data between applications when the order of events is important, for example:
    • Ensure that user-entered commands are executed in the right order.
    • Display the correct product price by sending price modifications in the right order.
    • Prevent a student from enrolling in a course before registering for an account.

USE CASES OF SQS

  • File processing – image scaling, video re-compression
  • Asynchronous communication with external services
  • Sending emails
  • Order Processing Application.

POST AND READ MESSAGE IN/FROM SQS USING .NET CORE 

I followed these steps to create an IAM user for authentication in SQS and then create a queue with that user having full access.

This is a very simple application that posts a message to a queue, and receives one, just by passing the credentials in the code. I have noticed that AWS nicely handles duplicates so if you send the same message multiple times, it only seems to show up once.

            

//the url for our queue
var queueUrl = "https://sqs.eu-west-1.amazonaws.com/[USERID]/[QUEUENAME]";

Console.WriteLine("Queue Test Starting!");

Console.WriteLine("Creating Client and request");

//Create some Credentials with our IAM user
var awsCreds = new BasicAWSCredentials("[ACCESSKEY]", "[SECRETKEY]");

//Create a client to talk to SQS
var amazonSQSClient = new AmazonSQSClient(awsCreds, Amazon.RegionEndpoint.EUWest1);

//Create the request to send
var sendRequest = new SendMessageRequest();
sendRequest.QueueUrl = queueUrl;
sendRequest.MessageBody = "{ 'message' : 'hello world' }";

//Send the message to the queue and wait for the result
Console.WriteLine("Sending Message");
var sendMessageResponse = amazonSQSClient.SendMessageAsync(sendRequest).Result;

Console.WriteLine("Receiving Message");

//Create a receive requesdt to see if there are any messages on the queue
var receiveMessageRequest = new ReceiveMessageRequest();
receiveMessageRequest.QueueUrl = queueUrl;

//Send the receive request and wait for the response
var response = amazonSQSClient.ReceiveMessageAsync(receiveMessageRequest).Result;

//If we have any messages available
if(response.Messages.Any())
{
    foreach(var message in response.Messages)
    {
        //Spit it out
        Console.WriteLine(message.Body);

        //Remove it from the queue as we don't want to see it again
        var deleteMessageRequest = new DeleteMessageRequest();
        deleteMessageRequest.QueueUrl = queueUrl;
        deleteMessageRequest.ReceiptHandle = message.ReceiptHandle;

        var result = amazonSQSClient.DeleteMessageAsync(deleteMessageRequest).Result;
    }
}

For more details, please refer “How Amazon SQS Queues Work” page. I have read documentation provided by amazon for SQS and believe me its really useful and all the things are very well documented.

I hope this will help !!!!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s