Running Hubot in Production

Angus Williams 7th January 2016

In a previous blog post, we spoke about the basics of ChatOps and Hubot, and how they can be used to make your workflow more efficient. In this blog, we’ll take a slightly more in-depth look at running Hubot in production. If you’d like to know more about setting up and basic scripting of a ChatOps bot, please read the previous part of this blog.

In this post we’ll share some of our experiences and thoughts on running Hubot in a production environment, and go through some practical examples of how to achieve this. Please note that this post is focussed primarily on Linux systems.

This guide starts with the assumption that you’ve already created a Hubot instance using Yeoman. If you’re unsure of how to do this, read the instructions here. All the files mentioned in this post can be found on Github here.

Version Control

Once your Hubot instance has been created, you should commit your changes to a version control system of your choosing. Any changes to Hubot should be committed and updated from version control. See this link for some useful information from the Hubot documentation.

Run Hubot as its own user

From a security standpoint, we advise that you run Hubot as its own user. In Linux you can create a system with the following command:

$ useradd -r hubot

Creating a system user is also good practice, since system users aren’t able to login and don’t have home directories which has some security benefits.

Updating Hubot

At ECS Digital, we don’t update our Hubot instances all that often, so we don’t use an automated deployment process. To update the code-base on the production Hubot server, we do the following:

hubot1.png

Of course, this process could be automated, but as we don’t update Hubot too often so we’re happy with this method for now. Ideally, though, we would write a script for Hubot to update itself!

Ensuring the Hubot process is run at startup (and kept running)

I’m personally a big fan of Supervisord. Supervisord is an excellent project which can control processes for you.

Some of the benefits you get from Supervisor are:

  • Log handling for stderr and stdout – this includes log rotation options.
  • Automatic restarts when a process dies.
  • Remote web interface and XML-RPC API for remote controlling processes.
  • Config is much easier to deal with than init or upstart scripts.

Supervisor is available as an .rpm package for Redhat Linux variants and a .deb package for Debian Linux variants. It can also be installed via the Python pip package manager.

As we’re running Hubot on an Ubuntu 14.04 AWS instance, the supervisor package is available in the standard repos and can be installed with the following command:

$ sudo apt-get install supervisor

Supervisor can also be installed via pip, which will ensure a more up-to-date package. You may have to install Python 2.7 and pip if your distribution doesn’t come with Python installed already. You may need to run this command as root:

$ pip install supervisor

Config files for supervisor generally reside in /etc/supervisor. Here is an example config for running Hubot via supervisor:

/etc/supervisor/conf.d/my-hubot.conf
[program:my-hubot]
command=bin/hubot --adapter slack ; command to execute
directory=DIR/WHERE/HUBOT/IS ; cwd for program
; Log file handling
stdout_logfile=/var/log/%(program_name)s.log
stderr_logfile=/var/log/%(program_name)s-stderr.log
stdout_logfile_backups=10
stderr_logfile_backups=10
user=hubot ; user to run hubot as
startsecs=10
autorestart=true
; Add any environment vars needed below
environment =
    HUBOT_SLACK_TOKEN="SLACK-TOKEN-HERE",
    HUBOT_AUTH_ADMIN="AUTH,TOKENS,HERE",

As you can see, in the example above we are doing the following:

  • Defining a command
  • Defining a working directory
  • Logfile handling for stdout and stderr output and logfile rotation. Note the %(program_name)’s Python variable expansion in the log names.
  • Telling supervisor to run the process as user Hubot
  • Telling supervisor to restart Hubot upon death.
  • Defin a few environment variables to pass to the process.

Once you’ve created or updated config for a program, run the following command:

$ sudo supervisorctl update

Then run this command to ensure Hubot is started.

$ sudo supervisorctl status

To restart Hubot after updating it, run the following command, replacing my-hubot with the name you’ve chosen for your program:

$ sudo supervisorctl restart my-hubot

See here for more information on supervisor config options.

For our production instance, we commit the supervisor config to the Hubot repo and then simply symlink the file into /etc/supervisor/conf.d/my-hubot.conf. That way, our supervisor config is nicely versioned and can easily be rolled back if something breaks.

Handling role based permissions with hubot-auth

Sometimes you want to lock certain Hubot functionality to a particular group of users. Although Hubot has no support for this by default, we can add this functionality with the hubot-auth plugin. The hubot-auth plugin uses Hubot’s “brain”. If you’re using this plugin, you’re going to want to make sure that you’ve connected Hubot up to redis so the “brain” is persistent. Install instructions are on the github page.

You may have noticed the HUBOT_AUTH_ADMIN environment variable in the supervisor configs. This defines which administrators have permission to add or remove users from roles. If you’re using Slack, you’ll need to get the userid – not the username. See here for a more detailed summary.

Once you’ve installed the plugin and started Hubot again, you’ll be able to do things like this:

So, as you can see, I have the ‘admin’ role which allows me to set and remove roles from users. Next, I added myself to the role ‘new-role’. I now have two roles: admin and new-role. Slackbot has none.

To use these roles, we have to create some logic when we are using Hubot scripts. Here’s an example script:

# Description:
# Hubot auth example
 
module.exports = (robot) ->
  robot.respond /am [iI] authed/i, (res) ->
    user = res.envelope.user
    if robot.auth.hasRole(user, "a-role")
      res.reply "You sure are #{res.message.user.name}!"
    else if robot.auth.hasRole(user, "admin")
      res.reply "Nope, but you are an admin. Add yourself!"
    else
      res.reply "NO! Get outta here"

And here’s the script in action:

hubot3.png
You may have noticed the slight caveat here: you are going to have to retrofit authorisation logic into any script which requires some form of authentication. Unfortunately, we’ve yet to find a better solution for user authentication with Hubot.

Handling end-to-end testing of Hubot

Note: This section focuses only on using Hubot with slack.

If you need to make sure your Hubot is up and responding with a tool like Sensu, Nagios or Icinga, you can use the following workflow:

Requirements:

The basic premise is that we create a private slack channel which consists of you, Hubot and Slackbot. Next, we use the Slack remote response API to trigger Hubot using the echo command:

hubot4.png
We then access the Slack message APIs using Hubot’s API token and retrieve the last message from the API to ensure that it matches the message we sent.

Once you’re happy the test is working correctly, you can leave the Slack channel to avoid being notified about the test every time it runs.

You can find a Sensu plugin in the Github repo for this blog. I’m not a coder by trade so please don’t hold my terrible ruby code against me! If you have any suggestions on how it can be improved, feel free to contact me with your ideas.

To find out more about ECS Digital, and our unique take on DevOps, check out the training courses that we offer on our website.

Found this interesting? Why not share it: