Choosing between Apache Kafka, Amazon Kinesis, Microsoft Event Hubs and Google Pub/Sub

Distributed log technologies such as Apache Kafka, Amazon Kinesis, Microsoft Event Hubs and Google Pub/Sub have matured in the last few years, and have added some great new types of solutions when moving data around for certain use cases.

To help you choose which one to use when, here is a decision flow chart. It is more of a general guide to which technologies to consider, and a few decision points to help you eliminate some technologies.

Ultimately Apache Kafka is the most flexible, and has great performance characteristics. But Apache Kafka clusters are challenging to setup, scale, and manage in production.

If you wish to go for a fully managed solution – Kinesis, Event Hubs and pub/sub offer alternative options depending on whether ordering and blob size are important to you.

On AWS, Amazon Managed Streaming for Apache Kafka (Amazon MSK) service available which is fully managed, highly available, and secure. You can also evaluate this option along with above all if you are using AWS Cloud for your workloads.

Confluent cloud has also re-engineered Apache Kafka for the cloud. They have given more focus on building apps and not managing clusters with a scalable, resilient and secure event streaming platform. It has made event streaming with Kafka simple on AWS, Azure and GCP clouds.

Hopefully this blog post will help you choose the technology that is right for you.

Amazon RDS Proxy makes it easier to use Relational database in serverless

In this post, you’ll learn how Amazon RDS Proxy and other improvements are making relational databases work in serverless.

In the past two years, AWS has addressed some of the key problems which make relational database work better in serverless applications.

Let us discuss them one by one.

Problem 1 – Pricing Model

  • Relational databases were priced hourly by instance size, whether you’re using it or not.
  • AWS released Amazon Aurora Serverless. This is a serverless version of the proprietary Amazon Aurora database that can automatically scale up and down according to your usage. This release helped with the pricing model issues around using a relational database.

Problem 2 – Cold-starts

  • Relational databases should be network-partitioned in a private subnet of your VPC where it cannot be accessed from the public internet. However, this means that your Lambda functions need to be in a VPC, which meant occasional cold-starts of up to 10 seconds.
  • AWS announced improved VPC networking for AWS Lambda functions. This update greatly decreased the coldstart latency for Lambda functions that use a VPC. This makes it more acceptable to use VPC Lambda functions in user-facing applications.
  • AWS has launched Provisioned Concurrency, a feature that keeps Lambda functions initialized and hyper-ready to respond in double-digit milliseconds.
    • When you invoke a Lambda function, the invocation is routed to an execution environment to process the request. When a function has not been used for some time, when you need to process more concurrent invocations, new execution environments are created.
    • The creation of an execution environment takes care of installing the function code and starting the runtime. Depending on the size of your deployment package, and the initialization time of the runtime and of your code, this can introduce latency for the invocations that are routed to a new execution environment.
    • When you enable Provisioned Concurrency for a function, the Lambda service will initialize the requested number of execution environments so they can be ready to respond to invocations.

Problem 3 – Connection limits

  • Relational databases were built for an era of a lower number of long-running compute instances. They don’t fit this world of a high number of connections. AWS Lambda users may run into connection limits when trying to connect to their relational databases.
  • Amazon RDS Proxy announced recently handles the connection limit problem. Rather than managing connections in your Lambda functions, you can offload that to the Amazon RDS Proxy. All pooling will happen in the proxy so that you can handle a large number of connections.

Having said that, there are still a few issues with relational databases in serverless applications. The pricing model is better, but it’s still not perfect. Amazon Aurora serverless scaling isn’t as quick as it needs to be and you’re still not getting anything like the DynamoDB On-Demand pricing. Additionally, the Amazon RDS Proxy is priced on an hourly basis.

But we can still hope that in future AWS will make this serverless journey better, by providing improvements and new tools and services to us.

I hope this will help !!!!

Secure database connection string while using AWS Lambda Function with Amazon RDS using AWS Secret Manager

In this blog post, we will discuss architecture style which help us to use AWS Secret Manager to secure your database credential and send them to Lambda function that will use them to connect and query the backend database service Amazon RDS, without hard coding the secrets in code or passing them through environment variables.

This approach will help you secure secrets and protect your backend databases. Manually changing the passwords would be cumbersome and AWS Secret Manager helps by managing and rotating the RDS database passwords.

Above architecture diagram can be easily implemented using .NET Core Lambda function along with any Amazon RDS like SQL, MongoDB , MySQL

  1. Mobile apps, Websites or Services call the Web API hosted on AWS API Gateway.
  2. API Gateway executes the Lambda function.
  3. Lambda function retrieves the database connection string using the Secret Manager API.
  4. Lambda function connect to the database using database secrets from Secrets Manager.
  5. Lambda function execute query & return the results to API Gateway.
  6. API Gateway with provide response to Clients (Mobile apps, Websites or Services).

I believe you guys can use this architecture style to securely connect with database using lambda functions and design your solutions.

I hope this will help !!!

AWS API Gateway – API Creation with Lambda Proxy Integration and Web Page Redirection (302) using .NET Core 2.0

In this article, we will see how to create and test an API with Lambda proxy integration using the API Gateway console. We will also see how a Lambda backend parses the query string request and implements app logic that depends on the incoming query string parameters. we will also see how to create 302 Redirect response using Lambda function and redirect caller directly to web page.

Topic to cover in this article

  1. How to create Lambda function for Lambda Proxy Integration using .NET core 2.0?
  2. How to create Resource?
  3. How to create HTTP GET Method with Lambda Proxy Integration?
  4. How to setup Request query string parameter?
  5. How to setup Response redirect (HttpStatus=302) with Location header parameter?

Step-1: Create Lambda function for Lambda Proxy Integration using .NET core 2.0

Create New AWS Lambda Project “ApiLambdaVerifyEmail” using Visual Studio 2017 version 15.5.6:


Add following NuGet Packages for Lambda function which will be needed for API Gateway Proxy Request and Response and other Lambda features:


Update Function Handler Code with below code snippets:

public APIGatewayProxyResponse FunctionHandler(APIGatewayProxyRequest input, ILambdaContext context)
    //Ready query string parameter
    string queryString ;
    input.QueryStringParameters.TryGetValue("pageKey", out queryString);

    //Set Default URL if no match found
    string redirectUrl = @"";

        Console.WriteLine("pageKey : " + queryString);

    //Use Query String Parameters to do some DB Operation 

    //Based on database operation redirect page to X Web Page or Y Web Page or Z Web Page
    switch (queryString)
        case "google":
            redirectUrl = @"";
        case "twitter":
            redirectUrl = @"";
        case "sandeep":
            redirectUrl = @"";

    Console.WriteLine("URL : " + redirectUrl);

    //Redirect to Web Page using 302 Response Code and URL
    var response = new APIGatewayProxyResponse
        StatusCode = (int)HttpStatusCode.Redirect,
        Headers = new Dictionary { { "location", redirectUrl } }

    return response;


What above code does is:

  • Read query string parameters – pageKey
  • Based on query string parameter values – set redirect URLfor the response.
  • Create APIGatewayProxyResponse object with page redirection (302) Status code and add header location for the page where to redirect.

Publish Lambda function by right clicking Project on solution explorer


Next, It will display Upload Lambda function popup where you can select/define your profile, region and Lambda function name and press Next button:


Next, It will display popup to select permissions for Lambda function you are uploading and once you select Role then press Upload button:


Note: Create Role for Lambda function if you haven’t created yet using IAM console as per your Lambda function Access Requirements. For this article we are not accessing any other AWS Service from Lambda function so no role as such required.

Next step is to create API Gateway API with Lambda Proxy Integration.

Step-2: Create Resource – TestRedirectPage :

How to setup Resource guide from AWS?


Step-3: Setup GET Method with Lambda Proxy Integration: 

How to setup HTTP Method guide from AWS?


3-Add permission to lambda function-confirmation

After GET Method configuration is saved it will show all request / response configuration on screen:


Step-4: Setup Query String Parameters for API Gateway – pageKey:


Verify GET Integration Request Type is LAMBDA_PROXY:


Step-5: Setup GET Method Response redirect (HttpStatus=302) with Location header parameter:

When setting up the API Gateway Method Response, start by declaring the status code in the Method Response section. 200 will already be there so delete it. Add your 302 here instead and add the Location response header.


After above all configurations are done, Method Execution will look like:


Test GET Method we configured by clicking Test link.

  • Set Query Strings textbox “pageKey=sandeep” and press test button.


Deploy API

  • Select GET method
  • then click on Actions button on top
  • then select deploy api option
  • select/create Dev stage and press Deploy button


Get Invoke URL from Dev environment

  • select GET method of testredirectpage resource
  • then it will display invoke URL
  • by using this URL we can hit API gateway resource and execute method from browser or Postman.


Run this Invoke URL in Browsser by Adding Query String pageKey=sandeep


Browser will invoke GET API method of testredirectpage resource on API gateway and then run our lambda function endpoint, then lambda function will parse this query string value, then send response to redirect Url= , then browser will redirect page.


Similarly, you can test with other Query String parameters and see which web page opens 🙂

I hope this is useful. 

Amazon Simple Queue Service – SQS

Amazon Simple Queue Service (SQS) is a distributed queue system that enables web services to quickly and reliably queue messages that one component in the application generates to be consumed by another component. A queue is a temporary repository for messages that await processing.


Below sample architecture we are going to use to understand how SQS works. There are two SQS Queue created, one for Request and another for Response.

Web Server who act as producer, who creates a message and puts it into request queue. it also read from response queue as well.

You can have multiple producers who can add multiple messages to the queue at the same time. You don’t have to worry about the traffic or peaks. SQS handles that for you.


Queued messages are processed by consumer. The consumer is requesting new messages periodically from the queue. You can have multiple consumers, but you have to remember that each message can be processed only once. It means that you can have multiple instances of the same consumer, but you can’t read the same message from one queue in two different components. Each of these components should use a separate SQS queue.

After a consumer processes the message, it has to be deleted from the queue. Deleting is important because SQS assumes that processing can fail. To prevent that, after the consumer receives a message, it is hidden from the queue for a defined period of time and after that, if it is not deleted, the message shows up in the queue again.


Two specific features of Amazon SQS make this possible:

  • A single Amazon SQS queue can be shared by multiple servers simultaneously.
  • A server that is processing a message can prevent other servers from processing the same message at the same time by temporarily “locking” a message. The server can specify the amount of time the message is locked. When the server is done processing the message, it should delete the message. If the server fails while processing the message, another server can get the message after the lockout period.

These two features ensure that the number of processing servers can be easily changed dynamically to handle varying load. The entire process can be automated to ensure that at any given time the optimal number of EC2 instances is running. This practice is commonly referred to as “auto-scaling.”


Standard Queue

  • Available in all regions.
  • Unlimited Throughput – Standard queues support a nearly unlimited number of transactions per second (TPS) per API action.
  • At-Least-Once Delivery – A message is delivered at least once, but occasionally more than one copy of a message is delivered.
  • Best-Effort Ordering – Occasionally, messages might be delivered in an order different from which they were sent.
  • When to use? – Send data between applications when the throughput is important, for example:
    • Decouple live user requests from intensive background work: let users upload media while resizing or encoding it.
    • Allocate tasks to multiple worker nodes: process a high number of credit card validation requests.
    • Batch messages for future processing: schedule multiple entries to be added to a database.

FIFO Queue

  • Available in the US East (N. Virginia), US East (Ohio), US West (Oregon), and EU (Ireland) regions.
  • High Throughput – FIFO queues support up to 300 messages per second (300 send, receive, or delete operations per second). When you batch 10 messages per operation (maximum), FIFO queues can support up to 3,000 messages per second. To request a limit increase, file a support request.
  • Exactly-Once Processing – A message is delivered once and remains available until a consumer processes and deletes it. Duplicates aren’t introduced into the queue.
  • First-In-First-Out Delivery – The order in which messages are sent and received is strictly preserved.
  • When to use – Send data between applications when the order of events is important, for example:
    • Ensure that user-entered commands are executed in the right order.
    • Display the correct product price by sending price modifications in the right order.
    • Prevent a student from enrolling in a course before registering for an account.


  • File processing – image scaling, video re-compression
  • Asynchronous communication with external services
  • Sending emails
  • Order Processing Application.


I followed these steps to create an IAM user for authentication in SQS and then create a queue with that user having full access.

This is a very simple application that posts a message to a queue, and receives one, just by passing the credentials in the code. I have noticed that AWS nicely handles duplicates so if you send the same message multiple times, it only seems to show up once.


//the url for our queue
var queueUrl = "[USERID]/[QUEUENAME]";

Console.WriteLine("Queue Test Starting!");

Console.WriteLine("Creating Client and request");

//Create some Credentials with our IAM user
var awsCreds = new BasicAWSCredentials("[ACCESSKEY]", "[SECRETKEY]");

//Create a client to talk to SQS
var amazonSQSClient = new AmazonSQSClient(awsCreds, Amazon.RegionEndpoint.EUWest1);

//Create the request to send
var sendRequest = new SendMessageRequest();
sendRequest.QueueUrl = queueUrl;
sendRequest.MessageBody = "{ 'message' : 'hello world' }";

//Send the message to the queue and wait for the result
Console.WriteLine("Sending Message");
var sendMessageResponse = amazonSQSClient.SendMessageAsync(sendRequest).Result;

Console.WriteLine("Receiving Message");

//Create a receive requesdt to see if there are any messages on the queue
var receiveMessageRequest = new ReceiveMessageRequest();
receiveMessageRequest.QueueUrl = queueUrl;

//Send the receive request and wait for the response
var response = amazonSQSClient.ReceiveMessageAsync(receiveMessageRequest).Result;

//If we have any messages available
    foreach(var message in response.Messages)
        //Spit it out

        //Remove it from the queue as we don't want to see it again
        var deleteMessageRequest = new DeleteMessageRequest();
        deleteMessageRequest.QueueUrl = queueUrl;
        deleteMessageRequest.ReceiptHandle = message.ReceiptHandle;

        var result = amazonSQSClient.DeleteMessageAsync(deleteMessageRequest).Result;

For more details, please refer “How Amazon SQS Queues Work” page. I have read documentation provided by amazon for SQS and believe me its really useful and all the things are very well documented.

I hope this will help !!!!