Presenting Baton

We, at Digital Science, are pleased to announce that we open sourced one of the key pieces of our technical infrastructure, baton, which is a general purpose tool to query and run commands on our servers. It provides a basic framework that is meant to be extended in order to fit specific needs such as deployment tasks, monitoring tasks, etc.

This is the first step Digital Science takes towards contributing to the open source community. We have always been trying new technologies and therefore we rely on many open source projects. Thus, contributing to the open source community is a natural step for our company and we very much hope this will be the first of many other projects that we will share.

Why baton?

We strongly believe that automating daily tasks can dramatically increase productivity. One of our goals was having one click deployment of code to development/production servers or quick service discovery. After trying existing tools, we found nothing that was good enough.

That’s when we decided to build baton.

Since then, we created a number of baton extentions that have been powering code deployments, service discoveries, handle alert messages, etc.

Internals

Baton is a ruby gem. It relies on EventMachine and AMQP for message passing. The gem defines a basic set of classes operating on top of RabbitMQ.

The initial configuration will setup an input exchange and an output exchange.

On the input exchange, baton will wait for meaningful messages to perform actions (described by each service) and it will output messages to the output exchange.

For example, one could send a message to discover services in production. Baton would receive that message on the input exchange, and every service that responds to that message would write an output message to the output exchange.

API

This is the entry point for input messages. One should extend the API class and add meaningful methods that ultimately use `publish` to publish messages to the input exchange. One example can be found here.

Executable script

Any baton extension should have an executable script that will start the extension service. Here is an example.

Service

The Service is the starting point of any baton extension. The idea of the service is to setup consumers for the input messages arriving from the API. By implementing `setup_consumers` one will allow the consumers to receive messages. Here is an example.

Consumer Manager

This class is an orchestration class that attaches observers to the consumers (like logger, etc), binds the input queues to the correct exchanges, dispatches the received messages to the consumers and updates the observers on changes. One doesn’t need to extend this class unless one wants to change its behaviour.

Consumer

This class must be extended in order to process each received message. One should implement `process_message` at least, in order to give meaning to each received message. One can also override `routing_key` in order to listen to specific messages. Here is an example of an implementation.

Channel

Like the consumer manager, this class doesn’t need to be extended. It provides functionality to setup the exchanges and add consumers.

Observer

The observer class provides methods to notify observers. It is by default included in the consumers so that the output exchange (and possibly loggers, etc) receive the output messages.

baton-deploy, an extension

One of the most important extensions created so far was baton-deploy, which is a an extension that allows one to deploy code to our servers.

We have installed baton-deploy on each server (development environments included). Each baton-deploy deamon will read a configuration file which will provide information concerning which applications are expected to be running on that server.

Baton-deploy then creates queues with specific routing keys (e.g. such as application_name.current_environment). When the services start listening on those queues for incoming messages, we then use a web interface to interact with baton’s api. One could then deploy a given application to all production servers, all development servers, or ultimately target specific servers.

Your turn

You can easily extend baton to perform your own tasks. baton-ping provides what we consider to be a minimal extension to baton. One should note that there is an extra class on `baton-ping` called monitor. This is a good example of what to do with output messages from a baton extension. Together with baton-ping-monitor, it provides a standard way of consuming output messages and do something relevant with them.

If you would like to create your own extension, simply install baton (`gem install baton`) and run the following command:

batonize gem GEMNAME -b

This will create a basic gem structure with the necessary files to create a minimum viable baton extension.