Build event-driven, serverless monitoring for Azure — in just 15 minutes

quarter-of-one

Microsoft Azure provides multiple Platform as a Service (PaaS) products that allow you and your development teams to focus on quickly building awesome solutions for your customers. One of those products, Azure Service Bus, allows you to separate tiers of a solution and introduce asynchronous processing, which is so important in public clouds to ensure durable applications that scale reliably.

Unfortunately, when it comes to monitoring, Azure Service Bus is lagging behind other Azure services, since at the moment there is no alerting capability provided on the Azure level at all, beyond logs and metrics integration with Operations Management Suite (OMS). There is also no integration available with Azure Monitor. Hopefully this will change soon and Azure Service Bus capabilities will be updated to match those of other Azure services. (The Azure Monitor Metrics dashboard preview is already available in the Azure Portal, so that’s a positive sign.)

Does it mean that unless we use OMS, we cannot monitor Azure Service Bus at all today?

Fortunately, the answer is “no,” since Azure Service Bus exposes REST API, which can be used to provide custom monitoring and alerting. It allows us to analyze the values of the following metrics:

  • Active message count
  • Dead letter message count
  • Scheduled message count
  • Transfer message count
  • Transfer dead letter message count

To visualize those metrics, we can use another Azure PaaS product — Azure Applications Insight. It provides monitoring capabilities that can be easily integrated into applications –including apps not running in Azure — and is a perfect fit for our needs.

The last component we need is a glue between Azure Service Bus and Applications Insight. In the past we would need to have a virtual machine (VM) running, on which code is executed, e.g. with cron or Windows Scheduler, depending on which OS you prefer. Then we would need to patch the VM, upgrade it when needed, manage users and their access and do all the other painful, non-differentiating activities that are required when you own a server. This is where serverless and Azure Functions come to the rescue.

Thanks to Azure Functions we don’t need to provision a VM to run our code. We just write the function code, upload it to Azure, define the schedule when it should be triggered and we’re done. Azure Functions will take care of everything else. Did I mention that it’s also very cost-effective (you pay for what you really use) and offers a huge free tier (1 million requests and 400,000 GBs of resource consumption per month)?

With all those components, our custom Service Bus monitoring solution would look like this:

Rest API

Since I am a big supporter of open source, and Microsoft shines in this area, let’s use Node.js for writing the Azure Functions “glue” component. It has a very fast starting time and works well with internet endpoints, so it’s perfect for our little exercise. Make sure you have Node.js and Azure Functions local development environment installed before proceeding.

You also need to understand the basics of Azure Functions project creation and development.

Coding

I will use a free IDE provided by Microsoft, Visual Studio Code, but any IDE will do the trick. On a high level we want our function to perform the following steps:

  1. Connect to Service Bus and Application Insights.
  2. Get a list of Service Bus queues.
  3. For each queue, get the metrics values and push them to Application Insights.

To interact with Azure APIs, we need to add the following dependencies:


"dependencies": {
"applicationinsights": "^0.22.0",
"azure-sb": "^0.10.4"
}

As for the code:
1. Connect to Service Bus and Application Insights.


// connect to service bus
var azure = require('azure-sb');
var serviceBusService = azure.createServiceBusService(process.env['MonitoredServiceBusEndpoint']);

// connect to AppsInsights
var appInsights = require("applicationinsights");
appInsights.setup().setAutoDependencyCorrelation(false).start();
var client = new appInsights.TelemetryClient(process.env['ApplicationInsightsWithSBData']);

Note that you must configure the following environment variables:

  • MonitoredServiceBusEndpoint as in the Service Bus -> Shared Access Policies -> Primary or Secondary Connection String
  • ApplicationInsightsWithSBData used to store custom metrics (you can still use default Client and APP_INSIGHTS parameter for monitoring Azure Functions)
  • AzureWebJobsStorage used for local Azure Functions testing

Back to our code, for convenience let’s define the list of metrics and their humanly readable names:


// name - human readable name in AppsInsights, id - what is exposed by service bus
const metrics = [
{ name: "Active Message Count", id: "d2p1:ActiveMessageCount" },
{ name: "Dead Letter Message Count", id: "d2p1:DeadLetterMessageCount" },
{ name: "Scheduled Message Count", id: "d2p1:ScheduledMessageCount" },
{ name: "Transfer Message Count", id: "d2p1:TransferMessageCount" },
{ name: "Transfer Dead Letter Message Count", id: "d2p1:TransferDeadLetterMessageCount" }
];

2. Get a list of Service Bus queues.


serviceBusService.listQueues(function(error, listqueuesresult, response) {
// code here
});

3. For each queue, get the metrics values and push them to Application Insights.


// in the list of queues we already receive all the metrics
listqueuesresult.;">forEach(function(element) {
// send data to apps insight
metrics.forEach(function(metric) {
client.trackMetric({
name: element.QueueName+" - "+metric.name,
value: element.CountDetails[metric.id]
});
});
}, this);

That’s it for the code (a complete example can be found in GitHub). Just make sure it’s triggered by a schedule (e.g. every five seconds).

If there are no errors, you can switch to Azure Application Insights and enjoy the metrics coming from Service Bus (visible in Custom Metrics). Now we can monitor Service Bus with Azure native tools (Application Insights) and a low-cost serverless gluing component — all in less than 15 minutes of work.


Lech-Migdal-headshotLech Migdal is a Cloud Solutions Architect in the Azure Center of Excellence at DXC Technology, providing advice and technical consulting to DXC clients in the adoption of Microsoft Azure and AWS clouds. Connect with Lech on LinkedIn.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s

%d bloggers like this: