Elasticsearch Mantanence Lessons Learned Today

The elasticsearch cluster was down! Today I troubleshooted an Elasticsearch-cluster-down issue. We have a 3-node Elasticsearch cluster receiving hundreds of Giga of tracking data every day. And this afternoon, it was suddenly down and all our kibana dashboards failed to load any data from it. From elasticsearch-kopf monitor, we could see more than half of the shards are unallocated, so it sounds like at least 2 nodes were just restarted for some reason....

June 17, 2016 · 4 min · 807 words · Teddy

RabbitMQ Exchange and Queue Design Trade-off

In previous post, I mentioned the discussion on StackOverflow regarding designing exchanges. Usually, when people ask about best practice of designing exchanges, you will get some useful suggestions, but finally you will be told that the final solution always “depends on your system needs”. Fair enough & safe enough answer, but not really helpful enough. So I want to extend this discussion a little bit here with more detailed backgrounds....

February 23, 2016 · 7 min · 1362 words · Teddy

Understanding RabbitMQ Exchange and Queue

In previous post, I compared the difference of ActiveMQ (JMS) and RabbitMQ (AMQP)’s design philosophy. RabbitMQ has its beauty of simplicity in its design with 5 exchange types. In this post, I want to clarify the understanding of RabbitMQ’s exchange & queue and their relationship - binding. In brief, exchanges are the only places where messages could be published to; while queues are the only places where messages could be consumed from....

February 22, 2016 · 3 min · 472 words · Teddy

Behind RabbitMQ Exchange Types

The underlying philosophy behind RabbitMQ ’exchange types'

February 18, 2016 · 2 min · 384 words · Teddy

How to Change RabbitMQ Queue Parameters in Production?

Tips to change RabbitMQ Queue Parameters in Production

January 19, 2016 · 3 min · 559 words · Teddy