Deploying Hubot to Heroku like a boss

Tuesday, November 01, 2011

What is Hubot?

Hubot was an almost mythical GitHub campfire bot. They use it for deploying, automate a lot of tasks, and to be a source of fun in the company. Was, because they open sourced it some time age.

Hubot & Heroku

When I decided to give hubot a try on Heroku I googled for it and found a few tutorials and blog posts. All of them advise to download (or clone) main hubot repository and deploy it to heroku. In may opinion this is not the best way to do it. This post describes how to create separated deployable hubot application.

Tools needed

You will need a ruby, git, node.js, npm and a heroku gem installed. Ruby and git is pretty common. You will install heroku gem by:

$ gem install heroku

then node.js with Homebrew

$ brew install node

and npm

$ curl http://npmjs.org/install.sh | sh

Things done locally

Clone hubot repository and create a new directory that will deployed to heroku.

$ git clone git://github.com/github/hubot.git
$ cd hubot
$ npm install    # install all required dependencies
$ bin/hubot --create ../acmebot

If you go to the created directory you should see file structure similar to this:

$ cd ../acmebot
$ ls -l
~/www/blog/hubot/acmebot  
total 32
-rw-r--r--   1 martinciu  staff    36 31 paź 21:28 Procfile
-rw-r--r--   1 martinciu  staff  3411 31 paź 21:28 README.md
drwxr-xr-x   3 martinciu  staff   102 31 paź 21:28 bin
-rw-r--r--   1 martinciu  staff    56 31 paź 21:28 hubot-scripts.json
-rw-r--r--   1 martinciu  staff   518 31 paź 21:28 package.json
drwxr-xr-x  12 martinciu  staff   408 31 paź 21:28 scripts

This will be your hubot application that you will deploy to heroku. First create a new git repository.

$ git init .
$ git add .
$ git commit -m "initial commit"

Now you can create a new heroku app.

$ heroku create acmebot --stack cedar

And now you can deploy your own hubot to heroku.

$ git push heroku

Heroku configuration

It is deployed, but hubot won’t join any campfire room because it doesn’t know anything about it. You have to tell him what room(s) should he go to. You can do it by heroku configuration variables.

# Campfire user's token. You can find it on user's profile pages.
# You should probably have additional campfire user for a hubot 
$ heroku config:add HUBOT_CAMPFIRE_TOKEN=secret
# room ids coma-separated (you can find room id in room URL)
$ heroku config:add HUBOT_CAMPFIRE_ROOMS=123 
# your campfire account subdoamin
$ heroku config:add HUBOT_CAMPFIRE_ACCOUNT="acme"

For some scripts Redis is required. You can add a free RedisToGo service by typing:

$ heroku addons:add redistogo:nano

All is set up. Now you can start hubot:

$ heroku ps:scale app=1 

It’s alive!

Now if you go to you campfire room and type

$ Hubot help 

you should get a list of commands that your Hubot is familiar with.

When it is not alive :/

If hubot doesn’t speak to you, it means that something went wrong. In that case you can check heroku logs by typing in console:

$ heroku logs 

You can also check application status by typing:

$ heroku ps 

More scripts

You have just deployed a basic hubot set up. If you want to add more commands you can find them on the hubot-scripts repository. It is already added to you your copy of hubot. To turn it on you should edit hubot-scripts.json file. It is simple JSON file with list of custom scripts that should be loaded. At the time of writing some of the scripts that are available in the hubot-scripts repository are not yet available on the npm. So if your hubot doesn’t start after adding some custom scripts, check it’s log files to see what scripts can’t be found.

Robot’s name

Hubot only talks to you if you call him by name. And it is a new of the user who’s token hubot uses. You can specify that name in the Procfile

app: bin/hubot --adapter campfire --name acmebot --enable-slash

And if you don’t want to talk to hubot by name you can add --enable-slash option. It will allow to replace robot’s name with /. Example:

/mustache me lady gaga 

Does your app have API?

Yes? That’s awesome! But are you sure that it is safe? And I don’t mean if you secured access with api_key, Basic HTTP or anything like this. I mean if it is safe for your severs? At zubibu we are going to add API for updating items in customer shops. It is pretty simple and obvious method. But if some API client use it in bad way it can harm us. For example, if an online store with 400 thousands items decided to decrease all prices by 2% for each item, and if they implemented zubibu API call after each item update - it could harm us. 400k requests in a very short period of time could be dangerous for many apps.

How to protect your API?

You can educate your clients :), you can implement queue for handling API calls on your side. But in most cases (or at least in case described above) queue should be done on the client side. To force implementing queue on the client side you could use API throttling. It should as light as possible and invoked as soon as possible in request processing. So the Rack app sound perfect for this task. You can write your own api throttling app or you can use rack-throttle gem. It is simple but has almost all required features and is pretty easy to extend. We at zubibu needed some custom features, so we decided to extend Rack::Throttle::Limiter. This is our implementation of API throttle. I home comments in code are enough to understand it.

# lib/api_defender.rb
require 'rack/throttle'
# we limit daily API usage
class ApiDefender < Rack::Throttle::Daily

  def initialize(app)
    host, port, db = YAML.load_file(File.dirname(__FILE__) + '/../config/redis.yml')[Rails.env].split(':')
    options = {
      # we already use Redis in our app, so we reuse it's 
      # config file here
      :cache => Redis.new(:host => host, :port => port, :thread_safe => true, :db => db),
      # if you use staging environment on the same redis server
      # it is good to have separete key prefix for this
      :key_prefix => "zubibu:#{Rails.env}:api_defender",
      # only 5000 request per day
      :max => 5000
    }
    @app, @options = app, options
  end

  # this method checks if request needs throttling. 
  # If so, it increases usage counter and compare it with maximum 
  # allowed API calls. Returns true if a request can be handled.
  def allowed?(request)
    need_defense?(request) ? cache_incr(request) <= max_per_window : true
  end

  def call(env)
    status, heders, body = super
    request = Rack::Request.new(env)
    # just to be nice for our clients we inform them how many
    # requests remaining does they have
    if need_defense?(request)
      heders['X-RateLimit-Limit']     = max_per_window.to_s
      heders['X-RateLimit-Remaining'] = ([0, max_per_window - (cache_get(cache_key(request)).to_i rescue 1)].max).to_s
    end
    [status, heders, body]
  end

  # rack-throttle can use many backends for storing request counter.
  # We use Redis only so we can use it's features. In this case 
  # key increase and key expiration
  def cache_incr(request)
    key = cache_key(request)
    count = cache.incr(key)
    cache.expire(key, 1.day) if count == 1
    count
  end

  protected
    # only API calls should be throttled
    def need_defense?(request)
      request.host == API_HOST
    end

end

To add this to your application you to add rack-throttle to your Gemfile

# Gemfile
#...
gem 'rack-throttle'

and run bundle install

after that add, ApiDefender middleware to your app’s rack stack. It should go as high in the stock as possible:

# config/applicaiton.rb
require 'lib/api_defender'
module SomeAwesomeApp
  class Application < Rails::Application
    # ...
    config.middleware.insert_after Rack::Lock, ApiDefender
  end
end

And thats it! Now, when you access your API you should get response like this:

$ curl -I http://api.someawesomeapp.com/whatever 
HTTP/1.1 200 OK
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4999

but if you exceed 5000 API calls you will get this response:

HTTP/1.1 403 Forbidden
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 0

And voilà, your API is more safe now!

More modifications

I suggest you reading rack-throttle’s README file as well as it’s source code to find out what else you could easily modify for your needs. For example your ApiDefence middleware could extend Rack::Throttle::Hourly instead of Rack::Throttle::Daily. Your could use different counter store, or identify your clients differently by overriding client_identifier method.

Upgrading to Rails 3.1.0.rc5 (from rc4)

Tuesday, August 02, 2011

Rails 3.1.0.rc5

Rails 3.1.0.rc5 is out. If your app doesn’t use rails 3.1 yet you may want to read this and probably this. If you have balls and you already use version 3.1, you probably would like to use rc5 ASAP.

Surprise

Alright RC releases could not be much different from each other. You may think that simple change in Gemfile make you even more cool.

gem 'rails', '3.1.0.rc5'

Not this time :). After changing Gemfile and running

$ bundle update rails

this line of my template:

# app/views/layouts/application.html.haml
= stylesheet_link_tag :contents, :media => "screen,print"

throws this error:

No expansion found for :contents

What has been changed?

After each major rails upgrade it is always good to generate dummy app and look around what has been changed. I did it and found two little things.

There is a new group in Gemfile called assets.

# Gems used only for assets and not required
# in production environments by default.
group :assets do
  gem 'sass-rails', "~> 3.1.0.rc"
  gem 'coffee-rails', "~> 3.1.0.rc"
  gem 'uglifier'
end

And config/application.rb file has been changed in the way how bundler require gems:

# config/application.rb
Bundler.require *Rails.groups(:assets) if defined?(Bundler)

After implementing these changes my app is working under rails 3.1.0.rc5. Hooray!

There may be other differences but none of them breaks my app.

Deprecation warning

After you upgrade to Rails 3.1 you may be attacked by deprecation warnings similar to this one:

DEPRECATION WARNING: class_inheritable_attribute is deprecated, please use class_attribute method instead. Notice their behavior are slightly different, so refer to class_attribute documentation first. (called from included at base.rb:2)

In most case it will probably be all right to replace your all class_inheritable_hash, class_inheritable_array, etc method with class_attribute. Unfortunately it won’t always work. There is a “slight” difference mentioned by warning above.

What is the difference?

With class_inheritable_hash (the old way):

class Base
  class_inheritable_hash :mappings
  self.mappings = {}
end

class FirstChild < Base
  self.mappings[:foo] = :bar
end

class SecondChild < Base
  self.mappings[:bar] = :baz
end

Base.mappings           # {}
FirstChild.mappings     # {:foo=>:bar}
SecondChild.mappings    # {:bar=>:baz}

Alright, this is the expected behavior. How does this code works with class_attribute? Let’s see!

class Base
  class_attribute :mappings
  self.mappings = {}
end

class FirstChild < Base
  self.mappings[:foo] = :bar
end

class SecondChild < Base
  self.mappings[:bar] = :baz
end

Base.mappings           # {:foo=>:bar, :bar=>:baz}
FirstChild.mappings     # {:foo=>:bar, :bar=>:baz}
SecondChild.mappings    # {:foo=>:bar, :bar=>:baz}

It is far from the how it supposed to work. I would’t say that class_attribute behavior is “slightly” different than “class_inheritable_hash”. It is completely different method! But there is no place for complaining. We have to deal with it. How can we do it? Actually quite simple. Just call dup method on an inheritable attribute:

class Base
  class_attribute :mappings
  self.mappings = {}
end

class FirstChild < Base
  self.mappings = self.mappings.dup
  self.mappings[:foo] = :bar
end

class SecondChild < Base
  self.mappings = self.mappings.dup
  self.mappings[:bar] = :baz
end

Base.mappings           # {}
FirstChild.mappings     # {:foo=>:bar}
SecondChild.mappings    # {:bar=>:baz}

Exactly as expected! Hooray!

Disclaimer

Please note that VMware Cloud Foundry is under heavy development right now. It means that some of the things written below might not be valid at time you are reading it. Please leave a comment if you find any outdated stuff.

Errbit

Errbit is an open source, self-hosted error catcher. It is Hoptoad API compliant so you can just point the Hoptoad notifier at your Errbit server if you are already using Hoptoad.

Because Errbit is self-hosted solution you need to have a decent server to run it. You can run it on you own server, you can run it on heroku - details in Errbit’s Readme or you can use Cloud Foundry. This post is a simple tutorial how to do it.

What is VMware Cloud Foundry

Cloud Foundry is the open platform as a service project initiated by VMware. It can support multiple frameworks, multiple cloud providers, and multiple application services all on a cloud scale platform.

You can download from Cloud Foundry from GitHub repository and set up your own cloud infrastructure or you can use hosted by VMware one. In this tutorial we use hosted one.

Cloud Foundry account and VMC tools

At first we need a Cloud Foundry account. They was at closed beta at the time I sign up. I had to wait a few days to get an account. I don’t know if it still apply. After you get an account install Cloud Foundry VMC tools. VMC is a command line client for Cloud Foundry.

$ gem install vmc

Then log in to Cloud Foundry using credentials.

$ vmc login  
Email: marcin.ciunelis@gmail.com
Password: ******
Successfully logged into [http://api.cloudfoundry.com]

You may want to change automatically generated password. You can do that using VMC tools:

$ vmc passwd  
Changing password for 'marcin.ciunelis@gmail.com'
New Password: ******
Verify Password: ******

Successfully changed password

Errbit

Errbit will not work out of the box on Cloud Foundry. We need to modify it a little. I won’t write here how to set up Errbit on your local box. You can find it on project README file. I’m going to focus on Cloud Foundry specific issues mainly. First, clone it from GitHub repository.

$ git clone git://github.com/jdpace/errbit.git

When you open Gemfile you will notice that redmine_client gem is taken from git repository, not from official gem release because official one does not work with Rails3. If you are not going to use Redmine integration you can safely comment that line out. If want to use Redmine you we will need to figure out something smarter :)

# Gemfile
gem 'lighthouse-api'
# gem 'redmine_client', :git => "git://github.com/oruen/redmine_client.git"
gem 'mongoid_rails_migrations'
#...

The other thing that Cloud Foundry currently does not support is running rake task. Errbit requires running rake errbit:bootstrap which copy config/config.yml, config/mongoid.yml files and seed the database. You can run this task on your local box to create config files.

$ rake errbit:bootstrap
Copying example config files...
-- Copying config/config.example.yml to config/config.yml
-- Copying config/deploy.example.rb to config/deploy.rb
-- Copying config/mongoid.example.yml to config/mongoid.yml

Seeding database
-------------------------------
Creating an initial admin user:
-- email:    errbit@errbit.example.com
-- password: password

Be sure to change these credentials ASAP!

Generating indexes for App
Generating indexes for Err
Generating indexes for Notice
Generating indexes for User

Edit config/config.yml file amending host (it must be uniqe) and email_from lines. Also production section in your config/mongoid.yml file should looks like this:

production:
  host: <%= JSON.parse( ENV['VCAP_SERVICES'] )['mongodb-1.8'].first['credentials']['hostname'] rescue 'localhost' %>
  port: <%= JSON.parse( ENV['VCAP_SERVICES'] )['mongodb-1.8'].first['credentials']['port'] rescue 27017 %>
  database:  <%= JSON.parse( ENV['VCAP_SERVICES'] )['mongodb-1.8'].first['credentials']['db'] rescue 'errbit_development' %>
  username: <%= JSON.parse( ENV['VCAP_SERVICES'] )['mongodb-1.8'].first['credentials']['username'] rescue '' %>
  password: <%= JSON.parse( ENV['VCAP_SERVICES'] )['mongodb-1.8'].first['credentials']['password'] rescue '' %>
  

Because we use JSON here to decode mongodb configuration, you will add the following line to your Gemfile

gem 'json'

and run bundle install command

To seed database we need to use a little trick. Errbit already use mongoid_rails_migrations so we can use rails migration to do this. Generate migration:

$ rails generate migration seed_database

and modify to look like this:

class SeedDatabase < Mongoid::Migration
  def self.up
    Rake::Task['db:seed'].invoke
    Rake::Task['db:mongoid:create_indexes'].invoke
  end

  def self.down
  end
end

One more thing before deploying. Errbit will send an email when error occur in your app. So you should have an valid delivery_method configured. It may be your own SMTP server, Amazon SES or email app like Sendgrid or PostmarkApp

Deploy

To run Errbit we need two servers. One for web-server and one for MongoDB database. Setting this up is really simple. To setup the app and database server type:

$ vmc push errbit --path=. --url=martinciu-errbit.cloudfoundry.com --mem=128M --runtime=ruby19
Detected a Rails Application, is this correct? [Yn]: 
Creating Application: OK
Would you like to bind any services to 'errbit'? [yN]: y
The following system services are available::
1. mongodb
2. mysql
3. redis
Please select one you wish to provision: 1
Specify the name of the service [mongodb-e776e]: 
Creating Service: OK
Binding Service: OK
Uploading Application:
  Checking for available resources: OK
  Processing resources: OK
  Packing application: OK
  Uploading (9K): OK   
Push Status: OK
Staging Application: OK                                                         
Starting Application: OK

--url should be same as the one you entered in config/config.yml file and it should be unique. That’s it you should have now a working application. You can check working apps by

$ vmc apps

+-------------+----+---------+-----------------------------------+---------------+
| Application | #  | Health  | URLS                              | Services      |
+-------------+----+---------+-----------------------------------+---------------+
| errbit      | 1  | RUNNING | martinciu-errbit.cloudfoundry.com | mongodb-e776e |
+-------------+----+---------+-----------------------------------+---------------+

It also shows connected services - mongodb here. You can now visit your errbit app in your browser. You should see a log in screen, but you can not log in because migrations was not run. Cloud Foundry runs migration only if config/database.yml exists. We don’t have such file because mongoid uses config/mongid.yml file by default. To force running migration just create an empty database.yml file and update app:

$ touch config/database.yml
$ vmc update errbit
Uploading Application:
  Checking for available resources: OK
  Processing resources: OK
  Packing application: OK
  Uploading (9K): OK   
Push Status: OK
Stopping Application: OK
Staging Application: OK                                                         
Starting Application: OK

Sometimes update doesn’t success. In that case stopping and starting service may help.

$ vmc stop errbit  
Stopping Application: OK

$ vmc start errbit  
Staging Application: OK                                                         
Starting Application: OK

Alright you should be able to log in to your own error catcher app. Just visit chosen URL (http://martinciu-errbit.cloudfoundry.com/ in my case) and log in with default credentials which are email: errbit@your-app-url.cloudfoundry.com and password: password. You may want to change these default. You can do this by clicking Edit profile button.

Configuration

You can now configure your production app that you want be monitored by Errbit. Errbit is compatible with hoptoad_notifier gem, so if you are familiar with this it should be pretty obvious to you. If you don’t use hoptoad you should add it to your Gemgile

gem "hoptoad_notifier"

and run bundle install command

You can create new app on the homepage, define who should receive notifications and create a config/errbit.rb file from the code provided by Errbit. In my case this file looks like this:

HoptoadNotifier.configure do |config|
  config.api_key = '3aa6c74ccabaf61295c1d10813575705'
  config.host    = 'martinciu-errbit.cloudfoundry.com'
  config.port    = 80
  config.secure  = config.port == 443
end

Unfortunately Errbit overwrites default hoptoad_notifier settings, so you can not use both error catchers at same time. Alright you should now be able to send a test error notification. Just go to you app folder and run:

$ rake hoptoad:test

You should a lot of dumped XML data and a while later you should receive email notification from Errbit. You can also review the test error on your errbit app. And that it! You have up and running your own Hoptoad clone!

Something went wrong?

If something went wrong (it probably will :P) you can find these tools and resources useful:

  1. vmc logs errbit
  2. vmc files errbit logs
  3. vmc help
  4. Cloud Foundry Forum
  5. Cloud Foundry source code on GitHub
  6. Errbit source code on GitHub

Didn't find what you were looking for? There's more in the archive.