I <3 Violet UML

I <3 Violet UML

Badass JavaScript: CoffeePhysics: A Fast New Physics Engine Written in CoffeeScript

Badass JavaScript: CoffeePhysics: A Fast New Physics Engine Written in CoffeeScript

Good presentation by SpringSource around the rise of AMQP & RabbitMQ as messaging solutions


Outerthought - A First Exploration of SolrCloud

Outerthought - A First Exploration of SolrCloud

What are you looking at?: Elasticsearch as nosql

What are you looking at?: Elasticsearch as nosql

Getting off the CouchDB... or Lessons Learned while Experimenting in Production

Getting off the CouchDB... or Lessons Learned while Experimenting in Production

RabbitMQ + Spring = Easy Integration

If you’ve never heard of RabbitMQ, it is a robust messaging system (written in Erlang) based on the AMQP standard, which is a wire-level protocol designed to be cross platform and language agnostic. We utilize RabbitMQ here at Traackr in our back-end technology stack, where we offload work that is not required for immediate consumption by our system. Not only have we found RabbitMQ to be incredibly fast and durable, but it also offers a slick UI and administrative controls for examining the state of your message queues and server.

We chose to integrate with RabbitMQ using the Spring AMQP project (http://www.springsource.org/spring-amqp). The Spring AMQP project provides all the necessary classes for setting up RabbitMQ transaction managers, connection factories, message templates, listeners, etc. As always, Spring provides terrific documentation and getting your application to successfully send and receive messages is a piece of cake - http://static.springsource.org/spring-amqp-net/docs/1.0.x/reference/html/amqp.html

Now, regarding message consumers, Spring recommends creating your listener containers with a bean definition (via XML) so that it can simply run in the background:

<object name="MessageListenerContainer" type="Spring.Messaging.Amqp.Rabbit.Listener.SimpleMessageListenerContainer, Spring.Messaging.Amqp.Rabbit">
    <property name="ConnectionFactory" ref="RabbitConnectionFactory"/>
    <property name="Queue" value="some.queue"/>
    <property name="MessageListener" ref="SomeListener"/> </object>

However, we did things a little bit differently here at Traackr, and here is why you should too. We create and manage our own listener containers via our own Service, which resembles something like:

@Service
public class MyRabbitMQListenerService
{
    ...
    private Map<String, SimpleMessageListenerContainer> messageListenerContainers = new HashMap...
    ...
    @Override
    public void afterPropertiesSet() { // spring bean initializer
        // pseudo-code
        for (String queue : my_list_of queues) {
            SimpleMessageListenerContainer container = SimpleMessageListenerContainer();
            container.queue = queue;
        container.MessageListener = new listener_delegate;
           ... set other stuff
           messageListenerContainers.put(queue, container);

             // Start by default (or not if you're no fun)
            container.start();
        }
    }
   
    public SimpleMessageListenerContainer getListenerContainerForQueue(String queue) {
        return this.messageListenerContainers.get(queue);
    }
}

This allows us to quickly access the SimpleMessageListenerContainer registered to each queue. One of the great things about the SimpleMessageListenerContainer class is that it provides methods for starting and stopping message consumption…

Now, suppose we expose two REST-ful endpoints that look something like this:

@RequestMapping(value = "/startMessageContainer/{queue}", method = RequestMethod.GET)
@RequestMapping(value = "/stopMessageContainer/{queue}", method = RequestMethod.GET)

With some extra code to connect the dots, we can now pause and resume message consumption for any queue at will … Awesome stuff!

@RequestMapping(value = "/stopMessageContainer/{queue}", method = RequestMethod.GET)
@ResponseBody
String stopMessageContainer(@PathVariable("queue") String queue) {
   // Quick & dirty
   myRabbitMQListenerService.getListenerContainerForQueue(queue).stop();
   return "success";
}

What’s the advantage of this, you might ask? Well, suppose the responsibility of the message consumer is to take the message payload and send that information to a 3rd party service. So, what happens when that 3rd party service goes down? Well, most likely an exception will be raised by the message consumer, triggering a roll-back by the RabbitMQ transaction manager, and the message will get put back on the queue. Now if this 3rd party service is down for a day, this cycle of roll-backs would continue until the external service was back up (unfortunately, RabbitMQ has no notion of a dead letter queue). If we were aware of the problem, we could easily shut down consumption on this queue and prevent all the unnecessary network chatter.

Even better, what if the message consumer imposed thresholds for certain types of errors that when met would automatically stop themselves. Using the above approach of encapsulation, we can achieve that, and that is some VERY powerful stuff.


Suggestions with SOLR

As you might have guessed from previous posts, we do use SOLR quite a bit at Traackr. It’s one of the main piece of technology that drives our search platform.

While we do not have any immediate need for suggestions capabilities, this is an interesting article that introduces some very useful concepts. Worth a read.

http://www.searchworkings.org/blog/-/blogs/different-ways-to-make-auto-suggestions-with-solr


The 10gen Blog on MongoDB and NoSQL: MongoDB at O'Reilly Strata Conference 2012

The 10gen Blog on MongoDB and NoSQL: MongoDB at O'Reilly Strata Conference 2012

myNoSQL: A Question About NoSQL Managed Hosting

myNoSQL: A Question About NoSQL Managed Hosting