Ordered queue processing in Azure Functions with Sessions

Previously ordered processing in Azure Functions was only possible with event streams like Azure Event Hubs, but today I want to show how you can preserve order for Service Bus queues and topics as well.

On the surface it seems pretty straight-forward: I want to be able to process messages from a queue in the exact order that I received them. For a simple service running on a machine, it’s pretty easy to achieve. However, how do I preserve the ordering of queue messages when I want to process at scale? With something like Azure Functions I may be processing messages across dozens of active instances, how can I preserve ordering?

Read full article by Jaff here.

I hope this help !!!

Spying on .NET Garbage Collector with .NET Core EventPipes

This post of the series shows how to generate GC logs in .NET Core with the new event pipes architecture and details the events emitted by the CLR during a collection.

  • Part 1: Replace .NET performance counters by CLR event tracing.
  • Part 2: Grab ETW Session, Providers and Events.
  • Part 3: CLR Threading events with TraceEvent.
  • Part 4: Spying on .NET Garbage Collector with TraceEvent.
  • Part 5: Building your own Java GC logs in .NET

To read full article click here.

I hope this will help !!!

Secure database connection string while using AWS Lambda Function with Amazon RDS using AWS Secret Manager

In this blog post, we will discuss architecture style which help us to use AWS Secret Manager to secure your database credential and send them to Lambda function that will use them to connect and query the backend database service Amazon RDS, without hard coding the secrets in code or passing them through environment variables.

This approach will help you secure secrets and protect your backend databases. Manually changing the passwords would be cumbersome and AWS Secret Manager helps by managing and rotating the RDS database passwords.

Above architecture diagram can be easily implemented using .NET Core Lambda function along with any Amazon RDS like SQL, MongoDB , MySQL

  1. Mobile apps, Websites or Services call the Web API hosted on AWS API Gateway.
  2. API Gateway executes the Lambda function.
  3. Lambda function retrieves the database connection string using the Secret Manager API.
  4. Lambda function connect to the database using database secrets from Secrets Manager.
  5. Lambda function execute query & return the results to API Gateway.
  6. API Gateway with provide response to Clients (Mobile apps, Websites or Services).

I believe you guys can use this architecture style to securely connect with database using lambda functions and design your solutions.

I hope this will help !!!

How to find missing Index in Sql Server

The below query will show missing index suggestions for the specified database. It pulls information from the sys.dm_db_missing_index_group_stats, sys.dm_db_missing_index_groups, and sys.dm_db_missing_index_details DMVs. You can also narrow it down to a specified table by uncommenting the AND statement and specifying the table name.


USE YourDatabase
GO
 
SELECT db.[name] AS [DatabaseName]
    ,id.[object_id] AS [ObjectID]
	,OBJECT_NAME(id.[object_id], db.[database_id]) AS [ObjectName]
    ,id.[statement] AS [FullyQualifiedObjectName]
    ,id.[equality_columns] AS [EqualityColumns]
    ,id.[inequality_columns] AS [InEqualityColumns]
    ,id.[included_columns] AS [IncludedColumns]
    ,gs.[unique_compiles] AS [UniqueCompiles]
    ,gs.[user_seeks] AS [UserSeeks]
    ,gs.[user_scans] AS [UserScans]
    ,gs.[last_user_seek] AS [LastUserSeekTime]
    ,gs.[last_user_scan] AS [LastUserScanTime]
    ,gs.[avg_total_user_cost] AS [AvgTotalUserCost]  -- Average cost of the user queries that could be reduced by the index in the group.
    ,gs.[avg_user_impact] AS [AvgUserImpact]  -- The value means that the query cost would on average drop by this percentage if this missing index group was implemented.
    ,gs.[system_seeks] AS [SystemSeeks]
    ,gs.[system_scans] AS [SystemScans]
    ,gs.[last_system_seek] AS [LastSystemSeekTime]
    ,gs.[last_system_scan] AS [LastSystemScanTime]
    ,gs.[avg_total_system_cost] AS [AvgTotalSystemCost]
    ,gs.[avg_system_impact] AS [AvgSystemImpact]  -- Average percentage benefit that system queries could experience if this missing index group was implemented.
    ,gs.[user_seeks] * gs.[avg_total_user_cost] * (gs.[avg_user_impact] * 0.01) AS [IndexAdvantage]
    ,'CREATE INDEX [IX_' + OBJECT_NAME(id.[object_id], db.[database_id]) + '_' + REPLACE(REPLACE(REPLACE(ISNULL(id.[equality_columns], ''), ', ', '_'), '[', ''), ']', '') + CASE
        WHEN id.[equality_columns] IS NOT NULL
            AND id.[inequality_columns] IS NOT NULL
            THEN '_'
        ELSE ''
        END + REPLACE(REPLACE(REPLACE(ISNULL(id.[inequality_columns], ''), ', ', '_'), '[', ''), ']', '') + '_' + LEFT(CAST(NEWID() AS [nvarchar](64)), 5) + ']' + ' ON ' + id.[statement] + ' (' + ISNULL(id.[equality_columns], '') + CASE
        WHEN id.[equality_columns] IS NOT NULL
            AND id.[inequality_columns] IS NOT NULL
            THEN ','
        ELSE ''
        END + ISNULL(id.[inequality_columns], '') + ')' + ISNULL(' INCLUDE (' + id.[included_columns] + ')', '') AS [ProposedIndex]
    ,CAST(CURRENT_TIMESTAMP AS [smalldatetime]) AS [CollectionDate]
FROM [sys].[dm_db_missing_index_group_stats] gs WITH (NOLOCK)
INNER JOIN [sys].[dm_db_missing_index_groups] ig WITH (NOLOCK) ON gs.[group_handle] = ig.[index_group_handle]
INNER JOIN [sys].[dm_db_missing_index_details] id WITH (NOLOCK) ON ig.[index_handle] = id.[index_handle]
INNER JOIN [sys].[databases] db WITH (NOLOCK) ON db.[database_id] = id.[database_id]
WHERE  db.[database_id] = DB_ID()
--AND OBJECT_NAME(id.[object_id], db.[database_id]) = 'YourTableName'
ORDER BY ObjectName, [IndexAdvantage] DESC
OPTION (RECOMPILE);

This script also generates a CREATE INDEX script for each record, so that you can take a better look at the index and decide if you want to use it.

It’s important to mention that these Missing Index statistics get reset every time SQL Server is restarted, so if you’ve recently rebooted the server you may not have many index stats accumulated yet.

I hope this help !!!

Which one to Choose – Azure Durable Function or Logic App?

If you are architecting a solution that requires serverless and stateful workflows using Azure Platform then there are two choices you have

  1. Azure Durable Functions
  2. Logic Apps

Azure Durable Functions is a new programming model based on Microsoft serverless platform Azure Functions. It allows you to write a workflow as code and have the execution run with the scalability and the reliability of serverless with high throughput.

Azure Logic Apps is a cloud service that helps you automate and orchestrate tasks, business processes, and workflows when you need to integrate apps, data, systems, and services across enterprises or organisations. Logic Apps simplifies how you design and build scalable solutions for app integration, data integration, system integration, enterprise application integration (EAI), and business-to-business (B2B) communication, whether in the cloud, on premises, or both.

In this blog post, I discuss the main differences between these two event-driven Azure services and I provide some guidance to help you to make the right decision.

Feature Comparison

Let us compare different features supported by these two platform like different workflows, connectors, logging and exception handling and few more. Below comparison table will put more lights on these features:

Durable Functions Logic Apps
Connectors or Bindings The list of supported bindings is here. Some of these bindings support triggering a function, or are inputs or outputs. The list of bindings is growing, especially for the Functions runtime version 2.

Logic Apps provide more than 200 connectors, and the list just keeps growing. Among these, there are protocol connectors, Azure Services connectors, Microsoft SaaS connectors, and third-Party SaaS Connectors.

Actions Can orchestrate Activity Functions (with the ActivityTrigger attribute). However, those Activity Functions could call other services, using any of the supported bindings.

Additionally, orchestrations can call sub-orchestrations.

Many different workflow actions can be orchestrated. Logic Apps workflows can be calling actions of the more than 200 connectors, workflow steps, other Azure Functions, other Logic Apps, etc.
Flow Control The workflow’s flow is controlled using the standard code constructs. E.g. conditions, switch case statements, loops, try-catch blocks, etc. You can control the flow with conditional statementsswitch statements, loopsscopes and controlling the activity chaining with the runAfter property.
Fan-Out / Fan-In Pattern Functions can be executed in parallel and the workflow can continue when all or any of the branches finish. You can fan-out and fan-in actions in a workflow by simply implementing parallel branches, or ForEach loops running in parallel.
Async HTTP APIs and Get Status Pattern Client applications or services can invoke Durable Functions orchestration via HTTP APIs asynchronously and later get the orchestration status to learn when the operation completes. Additionally, you can set a custom status value that could be query by external clients. Client applications or services could call Logic Apps Management API to get the instance run status. However, either the client has to have access to this API or you would need to implement a wrapper of this.

Custom Status value is not currently supported out-of-the-box. If required, you would need to persist it in a separate store and expose it with a custom API.

Programmatic Instance management Client applications or services can monitor and terminate instances of Durable Functions orchestrations via the API. Client applications or services could call Logic Apps Management API to monitor and terminate instances of Logic App Workflows. However, either the client has to have access to this API or you would need to implement a wrapper.
Concurrency Control Concurrency throttling is supported. Concurrency control can be configured at workflow level or loop level.
Lifespan One instance can run without defined time limits. One instance of Logic Apps can run for up to 90 days.
Error Handling Implemented with the constructs of the language used in the orchestration. Retry policies and catch strategies can be implemented.
Orchestration Engine Orchestration functions and activity functions may be running on different VMs. However, Durable Functions ensures reliable execution of orchestrations. To support this, check-pointing is implemented at each await statement. Additionally, the orchestration replays every time after resuming from an await call until it reaches the last activity check-pointed to rebuild the in-memory state of the instance. For high throughput scenarios, you could enable extended sessions. In Logic Apps the runtime engine breaks down the different tasks based on the workflow definition. These tasks are distributed among different workers. The engine makes sure that each task is executed at least once, and that tasks are not executed until their dependencies have finished with the expected status.

Pricing

Even though both options offer a serverless option where you only pay for what you use, there are some differences to consider as described below.

Durable Functions Logic Apps
In the consumption plan, you pay per-second of resource consumption and the number of executions. More details described here. For workflows you pay per-action and trigger (skipped, failed or succeeded). There is also a marginal cost for storage.

In case you need B2B integration, XML Schemas and Maps or Liquid Templates, you would need to pay for an Integration Account.

Durable Functions can also be deployed on App Service Plans or App Service Environments where you pay per instance. At the moment there is no option to run Logic Apps on your dedicated instances.

It’s also worth mentioning that in most cases, the operation costs of Logic Apps tend be higher than those of Durable Functions, but that would depend case by case. And for enterprise-grade solutions, you should not pick a platform based on price only, but you have to consider all the requirements and the value provided by the platform.

Management and Monitoring 

Its really important to choose platform which we can manage and monitor effectively after we deploy our solution in Production. So management and monitoring also play key role while making decision and below points might help you in choosing between them:

Durable Functions Logic Apps
Tracing and Logging The orchestration activity is tracked by default in Application Insights. Furthermore, you can implement logging to App Insights. The run history and trigger history are logged by default. Additionally, you can enable diagnostic logging to send additional details to Log Analytics. You can also make use of trackedProperties to enrich your logging.
Monitoring To monitor workflow instances, you need to use Application Insights Query Language to build your custom queries and dashboards. The Logic Apps blade and Log Analytics workspace solution for Logic Apps provide very rich and friendly visual tools for monitoring.
Resubmitting There is no out-of-the-box functionality to resubmit failed messages. Failed instances can easily be resubmitted from the Logic Apps blades or the Log Analytics Workspace.

Deployment Process

Durable Functions Logic Apps
CI/CD Durable Functions builds and deployments can be automated using VSTS build and release pipelines. Additionally, other build and release management tools can be used. Logic Apps are deployed using ARM Templates only.
 Versioning Versioning strategy is very important in Durable Functions. If you introduce breaking changes in a new version of your workflow, in-flight instances will break and fail. You can find more information and mitigation strategies here. Logic Apps keep version history of all workflows saved or deployed. Running instances will continue running based on the active version when they started.
Runtime Azure Functions can not only run on Azure, but be deployed on-premises, on Azure Stack, and can run on containers as well. Logic Apps can only run on Azure.

Summary: 

Logic Apps are better suited when:

Building integration solutions and leveraging extensive list of connectors required. It reduce time-to-market and ease connectivity.

Visual tools to manage and troubleshooting workflow required.

Durable Functions are a better suited when:

The list of available bindings is sufficient to meet the requirement.

The logging and troubleshooting capabilities are sufficient, and you can build your custom monitoring tools.

You prefer to have all the power and flexibility of a robust programming language.

 

I hope this will help !!!