Push jobs to Resque without Resque

July 11, 2013

At Wuaki.tv we are moving to a services oriented architecture and the first service we developed was the users service. We had to move all the users related logic to a new service and make our applications to use a service oriented model instead of the ActiveRecord model we had.

Actually we are using Errbit to collect the errors/crashes from our applications so we have the Airbrake gem as dependency in all the projects we need to monitor the errors. The Airbrake configuration we usually set, normally in a initializer, is something like this:

Airbrake.configure do |config|
  # ...
  config.async do |notice|
    OurAmaizingQueue.add(AirbrakeDeliveryWorker, notice.to_xml)
  end
end

In the code above we see three dependencies, Airbrake, OurAmaizingQueue and AirbrakeDeliveryWorker. As you may know, Resque jobs need to be a module or class with a class method called perform. So even the Resque workers are executed by another application, we have a dependency with the job class constant. Of course, we have a dependency with OurAmaizingQueue that is just a wrapper to enqueue into Resque.

We wanted to avoid Resque and OurAmaizingQueue (that lives in a common internal gem) dependencies in our super-tiny-micro-awesome users service. So…what if we can push the job into the correct Resque queue without the Resque gem and our internal huge gem? We tried it and this is the result:

require 'multi_json'

module MicroAwesomeService
  module Queue
    extend self

    def redis
      MicroAwesomeService.redis
    end

    def encode(object)
      if MultiJson.respond_to?(:dump) && MultiJson.respond_to?(:load)
        MultiJson.dump object
      else
        MultiJson.encode object
      end
    end

    def add(queue, klass_name, *args)
      push(queue, class: klass_name, args: args)
    end

    def push(queue, item)
      watch_queue(queue)
      redis.rpush("resque:queue:#{queue}", encode(item))
    end

    def watch_queue(queue)
      redis.sadd("resque:queues", queue.to_s)
    end
  end
end

The ‘magic’ is easy, we only need to encode a Hash into JSON with a class key that stores the class name of the job to be enqueued and a args key with the arguments for the job stored like an Array. This JSON is pushed into the tail of the list identified by the key resque:queue:name_of_the_queue_to_use.

Another important thing we shouldn’t forget is to add the queue name to use into the resque:queues set to make Resque listening this queue.

The Airbrake initializer changes a bit cause we don’t have the job class constant and we need to enqueue the job using a string.

Airbrake.configure do |config|
  # ...
  config.async do |notice|
    MicroAwesomeService::Queue.add('normal', "AirbrakeDeliveryWorker", notice.to_xml)
  end
end

Cause we don’t have the job class, we need to use explicity the queue name and the job class name. All these stuff was working fine while we only needed to enqueue jobs but finally our users service needs to perform its own Resque jobs, so we added Resque gem as dependency :-) and MicroAwesomeService::Queue became just a wrapper to enqueue jobs.

Discussion, links, and tweets

comments powered by Disqus

I'm a software developer at Wuaki.tv. Follow me on Twitter or on GitHub.

Follow @jondeandres