Microsoft Azure Message Exchange or Tips for Successful Cloud Communication

24 July 2015

My acquaintance with the Azure cloud began with some practice in DataArt. On the third month of my training our colleague, Anton Boyko, visited my mentor, Dmitry Andrusenko, and told him that he needed someone for a small demonstration project. That’s how I met him.

Anton is an enthusiastic and an active person with the ability to inspire people with ideas. My internship ended a long time ago, and Anton left DataArt soon after I finished it. But he still expresses an interest promotes Microsoft Azure, often consults us on Azure technologies, and invites us to take part in different events. For example, he holds different workshops, trainings, and meeting dedicated to the cloud. Not so long ago in Kiev, the first Ukrainian Azure Community was created.

The Ukrainian Azure Community is a group of like-minded people, who are interested in this technology, share their acquired knowledge and talk about Azure. Once every one or two months they meet to discuss different topics. Usually those meetings take place in DataArt’s Kiev office.

The main theme of the meeting where I made a presentation was exchanging messages with different Azure services. We discussed Azure Storage Queues, Azure Service Bus Queues, Azure Service Bus Topics, & Subscriptions.

I’ll elaborate on the technical part of this topic. There are three models of a message exchange.



Pic. 1

In model A the messages is sent directly from the sender to the recipient. It’s the simplest model, yet it has its flaws:

  • Difficulties with scaling (yes, it’s Clouds);
  • Easy to lose messages (e.g. when a receiver goes offline);
  • Hard to implement a retro policy.

Those downsides are the reason Azure uses mainly models B (where the message goes to the broker, and the recipient takes it from there) and C (the message goes to the broker, and then the broker sends it to the recipient).

Azure Storage Queues

This is a standard FIFO queue, and it follows model B.

Within an Azure account you can create any number of queues with unique names. The queue’s name should be URL-friendly. Every queue is able to contain any number of messages, yet every message should be no more than 64kb. I.e. the queues are not intended for data sharing, only for exchanging tasks.

(Here’s a question for you: what would you do if a message for some reason totals more than 64kb? In the context of Azure, of course)



Pic. 2

The message sending looks like this:

// Retrieve storage account from connection string
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(_cloudConnectionString);

// Create the queue client
CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();

// Retrieve a reference to a queue
CloudQueue queue = queueClient.GetQueueReference("somecorrectqueuename");

// Create the queue if it doesn't already exist
queue.CreateIfNotExists();

string messageToSend = string.Format("message #{0}", i);

// Create a message and add it to the queue.
CloudQueueMessage message = new CloudQueueMessage(messageToSend);
queue.AddMessage(message);

Azure’s bandwidth capability for sending to a queue is 20 000 messages per second. The peculiarity of this type of queue is a definite timeout (30 seconds by default) for the processing of a single message. Which means, a message isn’t taken out of the queue when being read, but becomes invisible for other recipients. If a message is processed correctly, it must be deleted from the queue afterwards. If a read error or any other situation occurs (i.e. after reaching timeout a message is not deleted), a message appears again in the head of the queue and is available for different recipients to read.

Proper processing of a message looks like this:

// Retrieve storage account from connection string
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(_cloudConnectionString);

// Create the queue client
CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();

// Retrieve a reference to a queue
CloudQueue queue = queueClient.GetQueueReference("somecorrectqueuename ");
while (true)
{
    // Get the next message
    CloudQueueMessage retrievedMessage = queue.GetMessage();

    if (retrievedMessage != null)
    {
      //Process the message in less than 30 seconds, and then delete the message
      queue.DeleteMessage(retrievedMessage);
    }
    else
    {
      Thread.Sleep(2000);
    }
}

You can set a unique timeout for every message (on a receiver’s side). Messages can contain only single lines, which are coded in base64 for the sending. There’s also the possibility to “see” the contents of a message without taking it from the queue. You can even read messages in packs, several at a time.

A REST API is provided for working with the Azure Storage Queue, which allows working with queues regardless of a platform. By the way, it’s the only message technology in Azure at the moment, which allows working with it directly, using the JS.

As you see, this service provides limited yet distinct functionality. However, it provides an opportunity to raise the probability of message delivery to the recipient compared with sending messages via Model A (without an intermediary). This queue fits architectures in which the receiver cannot always be online. Here’s an example: a sender conducts some activity during the day, logging his activity in a queue, and at the end of the day a recipient appears, processes this data, generates reports, and "goes to sleep " until the next day.

And last, but not the least – Azure Storage Queue is the least expensive message server in Azure.

Azure Service Bus Queues, Azure Service Bus Topics & Subscriptions

Azure Service Bus is a service, which provides several means of sending messages. There are two types:

  • Brokered Bus – asynchronous messaging; it stores messages and sends them when the recipient is ready to read them, using two mechanics: “queues” and “topics and subscription”
  • Relayed bus – synchronous messaging; messages are not stored anywhere, they are sent to the “environment”, and if the recipient is ready, he will read them, if not – the messages are lost.

Azure Service Bus Queues (pic. 3) uses Model B (pic. 1). It’s different from Azure Storage Queues in messages from the broker are sent to the recipient when they are ready and not vice versa. There’s also the possibility to send several messages at once (using transactions). In addition, if Azure Storage Queues support At-Least-Once delivery warranty, i.e. messages at least would be read once (in some scenarios more than once), Azure Service Bus Queues support At-Least-Once At-Most-Once guarantee, and that means a messages will be read only once.



Pic. 3

Azure Service Bus Topics & Subscriptions (Pic. 4) also implement Model B, but in a different way.



Pic. 4

The so-called topics are created in the Service Bus Namespace. This is also how subscriptions are created. A message from the sender is passed to a particular topic (the sender decides which topic to send it to) and then it is automatically filtered in the topic and copied to each subscription that satisfies the filter. The message is not stored in the topic. If there’s no suitable subscription, it disappears. If there are several, it is copied to each one.

Recipients create subscriptions. There should be a filter specification for each one (a filter can be empty, which means every message that arrives with the appropriate topic will be copied to the subscription). A filter can be changed without changing the subscription and the messages that are already there will not be lost and new ones will arrive based on to the updated filter.

A message can read by a recipient via two mechanisms: ReceiveAndDelete and PeekLock. The first one involves the usual reading of messages from the queue.  A request for the message is made and once read it is removed from the queue. The second one (just like in Azure Storage queues) involves a time-out for message processing. And if by the end there’s no confirmation that the message has been successfully processed, it again becomes visible to the other recipients.

You can read more about Azure Storage Queues here: http://azure.microsoft.com/uk-ua/documentation/articles/storage-dotnet-how-to-use-queues/
Azure Service Bus Queues: http://azure.microsoft.com/uk-ua/documentation/articles/service-bus-dotnet-how-to-use-queues/
Comparing Azure Storage Queues to Azure Service Bus Queues: https://msdn.microsoft.com/en-us/library/azure/hh767287.aspx
Azure Service Bus Topics & Subscriptions:  http://azure.microsoft.com/uk-ua/documentation/articles/service-bus-dotnet-how-to-use-topics-subscriptions/

The method of implementing a messaging system for your own project is up to you. First of all, you need to consider the level of importance of a message sending and how much are you ready to pay for it.

In conclusion, I would like to point out that Microsoft Azure technologies are in active development; there are many new features, some older ones are now depreciated, and that’s why it’s better to check for info about them on the official site.

Tags: ,


Add Comment

Name Mail Website Comment