Understanding RabbitMQ Exchange and Queue

In previous post, I compared the difference of ActiveMQ (JMS) and RabbitMQ (AMQP)’s design philosophy. RabbitMQ has its beauty of simplicity in its design with 5 exchange types. In this post, I want to clarify the understanding of RabbitMQ’s exchange & queue and their relationship - binding.

In brief, exchanges are the only places where messages could be published to; while queues are the only places where messages could be consumed from. And the configurations for how messages should be routed from exchanges to queues are called bindings.

Imagine a message as a mail, then an exchange is like a post office, and a queue is kind of a mailbox in front of someone’s house. Different exchange types are like different types of mails which will be delivered in different ways. For instances, if a school wants to send the same notification mail to every student’s mailbox (kind of broardcast), it is exchange type - “fanout”; if a mail should only be sent to one specific student’s mailbox, it is exchange type - “direct”, and the key of the binding between the exchange and the queue should be the address of the student’s mailbox which the post office could understand. The mail to be sent to the post office should have the same address written on it, this address is the “routing key” of the message.

In RabbitMQ, it is not possible for a publisher to send a message to a queue directly. Even when you send a message without specifying an exchange, the message is actually sent to the global default exchange, who has bindings to each queue with queue name as binding key.

Messages can only be consumed from queues, consuming from exchanges directly are not allowed. An exchange has no storage. So if you send a message to an exchange while there are no queues bound to it or no bindings match the routing key of the message, the message will be thrown away immediately.

About designing exchanges, there is an interesting discussion on StackOverflow for modeling a classic scenario. One exchange, or many? Which one is better? My opinion is similar to derick’s. It really depends on your system needs. RabbitMQ’s exchange types are simple, but flexible enough to support different modellings for the same scenario. There are always tradeoffs with each one. But no matter one or many exchanges you choose, I agree with Jason Lotito’s comments there about routing keys and naming convention of queues: “Routing keys are elements of the message, and the queues shouldn’t care about that (he means the naming of queues should not care about value of routing keys of published messages). That’s what bindings are for.”, “Queue names should be named after what the consumer attached to the queue will do. What is the intent of the operation of this queue.”.

Behind RabbitMQ Exchange Types

Before touching RabbitMQ, I used MSMQ & ZeroMQ much in real projects before. I also played Apache ActiveMQ together with Camel a little bit (but not in real projects) when evaluating different ESB solutions before.

Like everyone, my first glance of RabbitMQ is the Get Started tutorials, and the feeling of the RabbitMQ design philosophy is, it is quite like the design philosophy of ZeroMQ - simple & consistent interface and super flexible configuration. Among all of the configuration options, the most interesting part is the “exchange types”. You might want to say that the concept of “exchange types” is from the AMQP protocol, not invented by RabbitMQ. Yes, you are right. But no doubt that RabbitMQ is one of the most popular AMQP implementations.

The 6 examples in the Get Started tutorials show the flexibility of RabbitMQ with the consistent and simple Publish & Receive interface. And I believe you could already imagine many more usages of them to match many other common scenarios.

But, what’s the underlying philosophy behind “exchange types”? In a word, it is all about implementing integration patterns in a manner of simple, stupid.

If you ever played Apache ActiveMQ and Camel, you must be familar with the Enterprise Integration Patterns. The examples of Apache Camel even exactly match all of the integration patterns. The design philosophy of ActiveMQ is, it provides the basic queue and pub/sub ability, and, together with Camel, to give ultimate flexibility for implementing all the integration patterns.

The design philosophy of ZeroMQ and RabbitMQ is quite different. Ultimate flexibility is cool, but performance and simplicity are also important. ZeroMQ chooses ultimate performance and simplicity rather than more integration features, while RabbitMQ is kind of in between of ActiveMQ and ZeroMQ. Its interface is as simple as ZeroMQ; the performance is not as super as ZeroMQ, but good enough; and it provides minimal but elegant configuration options to support most of the common integration patterns. One of the essences of its design is just the “exchange types”, which abstracts most common message routing & consuming scenarios with only 5 easy-to-understand types: default (no routing), direct, fanout (broadcast), topic and header.

Modeling & programming is the art of abstracting complexity. A real elegant design must be simple, stupid! That’s what behind the idea of “exchange types”.

Integrate PrettyPhoto with jekyll-picture-tag plugin

In previous post, I talked about installed jekyll-picture-tag plugin to display pictures responsively in posts. It works very well in different browsers and resolutions. But sometimes, we might want a photo gallery. Instead of displaying the full size of photos, we might want to display thumbnails, and on click, popup a lightbox style slideshow window for displaying the full size of photos.

So, in this post, I want to talk about how I have integrated PrettyPhoto with jekyll-picture-tag plugin. Our goal is no change to jekyll-picture-tag plugin, and to use the same jekyll-picture-tag’s liquid tag format, only to add PrettyPhoto effect with addtional attributes when necessary.

The live example, is already in the previous post, an image is clicked, the PrettyPhoto slideshow window will popup.

The only addtional attribute we add to the jekyll-picture-tag liquid tag is “group”. You could see the sample liquid tag below. Images marked with the same group value, will be displayed together in the same PrettyPhoto slideshow window. So easy!

{% picture 2015-1-20-1.jpg alt="picture 1" group="1" %}
{% picture 2015-1-20-2.jpg alt="picture 2" group="1" %}

Besides the addtional attribute in liquid tag. We also need to add necessary css and js to our page template. So, in the header include file, we need to add the following:

  <!-- Pretty Photo -->
  <script src="/public/js/jquery-1.6.1.min.js" type="text/javascript" charset="utf-8"></script>
  <link rel="stylesheet" href="/public/css/prettyPhoto.css" type="text/css" media="screen" charset="utf-8" />
  <script src="/public/js/jquery.prettyPhoto.js" type="text/javascript" charset="utf-8"></script>  
  <script type="text/javascript" charset="utf-8">
    $(document).ready(function(){
	  $("picture img").each(function(){
	    if (!$(this).attr('group')) return;
		
	    var parent = $(this).parent();
	    parent.attr('rel', 'prettyPhoto[group_' + $(this).attr('group') + ']');
		parent.attr('href', parent.children().first().attr('srcset'));
	  });
      $("picture[rel^='prettyPhoto']").prettyPhoto({theme:'pp_default',slideshow:5000,autoplay_slideshow:false,social_tools:false});
    });
  </script>  

Also, we need to add necessary css, js and image files to the jekyll repo. All the files could be downloaded from the official PrettyPhoto site.

Installed jekyll-picture-tag plugin

Installed jekyll-picture-tag plugin to display pictures responsively in posts.

Steps to install:

Step 1: install gem;

gem install jekyll-picture-tag

Step 2: edit _config.yml according to: jekyll-picture-tag _config.yml example;

Step 3: add liquid tag to post;

{% picture 2015-1-20-1.jpg alt="picture 1" group="1" %}
{% picture 2015-1-20-2.jpg alt="picture 2" group="1" %}

How to Change RabbitMQ Queue Parameters in Production?

RabbitMQ does not allow re-declaring a queue with different values of parameters such as durability, auto delete, etc. Some parameters could be configured both by queue parameter and server-side policies, but if both are set, queue parameters win. So as long as queue parameters are used, it is the same problem.

So, what if, in production environment, we do want to do the change?

Firstly, if your system could accept temporary downtime, then the easiest way is apparently to stop all your publisher and subscriber apps, delete the queue, and create the new one with new parameters.

Secondly, if possible, instead of applying incompatible parameters on existing queue, it is always recommended to add a version number to your queue (meaning creating a new queue with the same name prefix as the old one, but to add/increase the version number as part of the queue name). So that after released the config or code and restarted all the publisher and consumer apps, you only need to move all the pending messages from the old queue (if any) to the new queue, and then delete the old queue.

If you really want to change some incompatible parameters of an existing queue in production environment in a safe way (no message lost and system hang), some tricky manual steps have to be executed. RabbitMQ is so flexible, you could have many different ways to reach the same goal, so here I just try to give some examples, from which, you could gain some clues to benefit your real scenarios.

Example 1, if the dead letter exchange and the message TTL option of the old queue is not configured with queue parameters, meaning we could configure them with server-side policies, and also all the publisher apps only work with exchanges forwarding messages to the old queue, not publishing messages to the old queue directly:

1. We could temporarily create a new queue and a new exchange bound to it, set the temporary exchange as the dead letter exchange of the old queue and set the message TTL of the old queue to 0 with server-side policies. From now on, all the new messages published to the old queue will automatically be forwarded to the new queue.
2. As long as the old queue have no pending messages any more, you could change the upstream exchanges of the old queue to bind to the new queue instead.
3. Now you could recreate the old queue and repeat step 1 on the new queue (dead letter exchange and TTL 0) to change back to use the recreated queue.
4. Delete all temporary exchanges and queues.

Example 2, similar to example 1, the difference is we could not configure dead letter exchange or message TTL on old queue, and if our subscribers could tolerate receiving duplicated messages:

1. We could create a temporary queue and bind to all the upstream exchanges of the old queue, so that duplicated messages are forwarded to both the old queue and the new queue now.
2. Then we delete & recreate the old queue with the same bindings.
3. After restarted all the apps (they are still talking to the old queue), let’s move all the pending messages in the new queue to the old queue with the shovel plugin (please realize that there might be some duplicated messages received by subscribers).
4. Delete the temporary queue.