Blogs

The motivations, strategies, and challenges of migrating classic ASP.NET applications to containers

Licenses of this site

Created a new hugo based blog site

In recent 3 months, I have spent most of my casual time in studying machine learning algorithms. I’m really a new comer to machine learning. And left college for more than 15 years, no surprise, I have successfully forgotten most of my maths knowledge for such as calculus, linear algebra, probabilities, statistics, etc. How I began my learning was from reviewing all the maths by reading books, doing exercises, then watching online machine learning videos, and finally following books & tutorials in action. ...

Background In modern application development, there are so many scenarios of dataflow processing, such large volume data processing, image/video conversion, data migration, etc. In .NET, thanks to Microsoft for providing the TPL Dataflow library which is a flexible and high performance library for this area. TPL dataflow is really flexible, but when we want to build complex dataflows, we might have to create bunch of blocks, manually wire them, be careful to avoid data blocking, write a lot of tricks even for implementing the simplest branching and merging cases. If you are ever a dataflow developer, do you remember how many times you ever wait for your dataflow to complete but it never happens and you have to debug line by line to find out where the message is blocked and why. ...

The elasticsearch cluster was down! Today I troubleshooted an Elasticsearch-cluster-down issue. We have a 3-node Elasticsearch cluster receiving hundreds of Giga of tracking data every day. And this afternoon, it was suddenly down and all our kibana dashboards failed to load any data from it. From elasticsearch-kopf monitor, we could see more than half of the shards are unallocated, so it sounds like at least 2 nodes were just restarted for some reason. Coz of our cluster setting is each index has one primary and one replica, until at least the primary shards are allocated, the indices are not able to be loaded. The shards are being slowly allocated automatically. If I’m patient enough and just wait for a while, it should be recover by itself in my understanding. So I try to wait. After 10 minutes, some dashboards could display, which looks good. But after 30 minutes, from kopf, I could see the HEAP of the master node keeps increasing, and eventually full. And the entire cluster becomes no responsive again. Restart the master node, but the HEAP still keeps increasing and be full and cluster down again. ...

In previous post, I mentioned the discussion on StackOverflow regarding designing exchanges. Usually, when people ask about best practice of designing exchanges, you will get some useful suggestions, but finally you will be told that the final solution always “depends on your system needs”. Fair enough & safe enough answer, but not really helpful enough. So I want to extend this discussion a little bit here with more detailed backgrounds. Before discussing any solutions, I’m trying to suggest some rules first. No matter which solutions to choose, these rules should be correct in most cases and should be followed if there are no compelling reasons. ...

In previous post, I compared the difference of ActiveMQ (JMS) and RabbitMQ (AMQP)’s design philosophy. RabbitMQ has its beauty of simplicity in its design with 5 exchange types. In this post, I want to clarify the understanding of RabbitMQ’s exchange & queue and their relationship - binding. In brief, exchanges are the only places where messages could be published to; while queues are the only places where messages could be consumed from. And the configurations for how messages should be routed from exchanges to queues are called bindings. ...

The underlying philosophy behind RabbitMQ ’exchange types'

Tips to change RabbitMQ Queue Parameters in Production