Open Sourcing With Rails Engines

We wanted to open-source Rearview so that it could easily be used by the community. When we were discussing how to best do that, we knew our solution had to be as adaptable as possible. It needed to answer questions like:

How am I going to separate configuration from code? For example you might be using mysql but typically you would want your users to select any Rails supported database. Or maybe you use Capistrano for deployment with recipes and settings specific to your infrastructure.

What about gems that may not be appropriate for some users? This problem can occur with instrumentation like NewRelic and AirBrake. You do not want to impose them on all your users, but these might be required for others. Users should easily be able to add gems and features independent of the engine.

How can I create well-defined extension points for my application? For example even though you open source most of your code, there may be some code you want to keep internal. You also want to make it easy for other users to customize the application.

These questions can be answered with Rails engines.

Introduction to Engines

Rails engines are really just normal Rails applications that can be embedded into another Rails applications, referred to as the engine host. An engine can provide views, models, controllers, helpers, and other logic in any combination or amount as is warranted. The Rails guides refers to an engine as a “miniature application”. I find this statement misleading. An engine can be as small or as big as you want.

With that in mind, before we get into using Rails engines to build stand-alone applications, I’d like to cover some of the common use cases for Rails engines.

Common Use Cases For Engines

Mini-Applications

Sometimes its useful to create an engine that can be embedded as a miniature application inside another Rails project. For example, Resque is a Ruby based library for creating background jobs. Resque includes resque-web, a Rails engine providing an admin user interface for managing Resque queues. Although you can deploy resque-web by itself, as we do at LivingSocial, in smaller environments it might make sense to deploy it embedded within another Rails application.

Plug-in For Other Rails Applications

Devise is a authentication framework that is meant to be integrated into a pre-existing Rails project, and not intended to run stand alone. Providing a cross-cutting concern such as authentication is a common theme for engines that fall into this category.

Composition

Engines can be used to build components in an SOA, where various service end points can share things like domain models, preventing code duplication. Last year at Rocky Mountain Ruby, Ben Smith gave a good talk on using engines with this approach.

Stand Alone Application

A stand alone application can be rolled up into a Rails engine. This allows you to separate configuration from code, creates distinct extension points, and makes it easy for you and your users to customize the application independently from the main code base. This is the use case I’m covering in this post.

Refactoring as an Engine

Changing a stand alone application to run as an engine takes some work, but the payoff makes it well worth it. I’ll outline the basic approach we used when we were working on the Rearview engine. Taking a similar approach should be enough to get you started in the right direction.

Your setup will consist of two distinct and separate projects: the engine, and the host. Each will have its own directory and repository. The engine will ultimately be a gem you include in your host projects Gemfile. The host will be a full-blown (but mostly bare) rails application. Your users will download your host from git or however you choose to bundle it. The engine then is just a mostly transparent dependency (from the view point of the user.)

The Engine

The engine will house all your models, views, controllers, helpers, and other logic. The engine will be packaged as a gem and included as a dependency in the host. Start off by creating a new rails engine project:

1
2
3
4
5
6
7
8
9
10
11
12
% rails plugin new coyote --mountable

...
Using rails (4.0.2)
Using coyote (0.0.1) from source at /tmp/coyote
coyote at /tmp/coyote did not have a valid gemspec.
This prevents bundler from installing bins or native extensions, but that may not affect its functionality.
The validation message from Rubygems was:
  "FIXME" or "TODO" is not an author

Your bundle is complete!
Use `bundle show [gemname]` to see where a bundled gem is installed.

The validation failure message seen above can be easily corrected by editing the gemspec file, coyote.gemspec,and replacing the self-explanatory TODO text. Once this is complete you can get started. Soon you’ll be shuffling your code from your original project into the engine. If your an apt git user, you can create a feature branch, remove all your files, and run the command above. As you move forward you would unstage files and refactor them as necessary. You may also just choose to copy files in from your original project.

Engine Dependencies

Copy the gem dependencies from your original projects Gemfile into your engines gemspec. The gemspec file allows for both runtime and development dependencies, but does not provide for gem groups the same way bundler does. At this point your gemspec will look something like the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
$:.push File.expand_path("../lib", __FILE__)

# Maintain your gem's version:
require "coyote/version"

# Describe your gem and declare its dependencies:
Gem::Specification.new do |s|
  s.name        = "coyote"
  s.version     = Coyote::VERSION
  s.authors     = ["Carroll Shelby"]
  s.email       = ["shelby@ford.com"]
  s.homepage    = "http://github.com/shelby/coyote"
  s.summary     = "A fast engine for use in mustangs."
  s.description = "A fast engine for use in mustangs, and built for mods."

  s.files = Dir["{app,config,db,lib}/**/*", "MIT-LICENSE", "Rakefile", "README.rdoc"]
  s.test_files = Dir["test/**/*"]
  s.add_dependency "rails", "~> 4.0.2"
  s.add_dependency "devise", "~> 3.2.2"
  s.add_dependency "ancestry", "~> 2.0.0"
  s.add_dependency "state_machine", "~> 1.2.0"
  s.add_dependency "protected_attributes", "~> 1.0.5"
  s.add_dependency "httparty", "~> 0.12.0"
  s.add_dependency "celluloid", "~> 0.14.1"
  s.add_dependency "broach", "~> 0.3.0"
  s.add_dependency "jbuilder", "~> 1.5.2"
  s.add_dependency "statsd-ruby", "~> 1.2.1"

  s.add_development_dependency "activerecord-jdbcmysql-adapter"
  s.add_development_dependency "foreman"
  s.add_development_dependency "rspec-rails"
  s.add_development_dependency "factory_girl_rails"
  s.add_development_dependency "pry"
  s.add_development_dependency "shoulda"
  s.add_development_dependency "mocha"
  s.add_development_dependency "timecop"
end

Note that you’ll still be using bundler to manage these dependencies. Take a look at your projects Gemfile and you’ll see that bundler reads dependencies from your gemspec.

Engine Models

Models in your engine should be copied over from your original project. However, they should be name-spaced now. Models will now go in app/models/coyote, and each class would be inside the Coyote module. For example:

1
2
3
4
module Coyote
  class CylinderHead < ActiveRecord::Base
  end
end

Correspondingly, all tables for your models will be prefixed with coyote. So the table name above would be coyote_cylinder_head. Before copying all your models, I encourage you to see how files are laid out by running the rails generator:

% bin/rails generate model cylinder_head
Migrations

Over the lifetime of the application you are converting into an engine, it’s likely you have collected a fair number of migrations. I would recommend rolling these migrations up into one base migration, which is the approach I took with rearview. You can do this be going to your original project and dumping the schema:

% rake db:schema:dump

Then create a new migration in your engine:

% bin/rails generate migration BaseSchema

Copy your schema dump into the change method of the migration. When your engine is included in a host, a rake task is automatically made available that will copy migrations from the engine into the host. The host would then run migrations normally.

Engine Controllers

In the same way you copied over your models, you will also need to copy over your controllers. They too must be name-spaced. Controllers will be put in app/controllers/coyote, and each class would also be inside the Coyote module. For example:

1
2
3
4
5
6
require_dependency "coyote/application_controller"

module Coyote
  class PistonsController < ApplicationController
  end
end

Again, you might want to test this out by running a rails generator:

% bin/rails generate resource pistons
Engine Helpers, Views, and the Kitchen-sink

At this point I hope you have gathered enough knowledge to guess how helpers will be laid out:

1
2
3
4
module Coyote
  module PistonsHelper
  end
end

Views should be placed in sub-directories under app/views/coyote, following the examples above. Views of course are not names-paced with modules.

The rest of your application might be in lib, in app/concerns, tests/specs, etc. These files will also need to be moved over. It can be a bit tedious to do, but is not as bad as you would think. Make sure all your other classes and modules are also put in the proper namespace.

The Host

Once you have the engine setup correctly and your tests passing, your almost there. Create a new rails project that will host your engine:

% rails new coyote-web

During development you can add a gem dependency to your Gemfile that points to your local copy:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
source 'http://rubygems.org'

# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'
gem 'rails', '4.0.2'

# Use jdbcsqlite3 as the database for Active Record
gem 'activerecord-jdbcsqlite3-adapter'

# Use SCSS for stylesheets
gem 'sass-rails', '~> 4.0.0'

# Use Uglifier as compressor for JavaScript assets
gem 'uglifier', '>= 1.3.0'

# Use CoffeeScript for .js.coffee assets and views
gem 'coffee-rails', '~> 4.0.0'

# See https://github.com/sstephenson/execjs#readme for more supported runtimes
gem 'therubyrhino'

# Use jquery as the JavaScript library
gem 'jquery-rails'

# Turbolinks makes following links in your web application faster. Read more: https://github.com/rails/turbolinks
gem 'turbolinks'

# Build JSON APIs with ease. Read more: https://github.com/rails/jbuilder
gem 'jbuilder', '~> 1.2'

gem 'coyote', path: '~/workspaces/tmp/coyote'

group :doc do
  # bundle exec rake doc:rails generates the API under doc/api.
  gem 'sdoc', require: false
end

# Use ActiveModel has_secure_password
# gem 'bcrypt-ruby', '~> 3.1.2'

# Use unicorn as the app server
# gem 'unicorn'

# Use Capistrano for deployment
# gem 'capistrano', group: :development

Now add a mount point in your hosts config/routes.rb file:

1
2
3
CoyoteWeb::Application.routes.draw do
  mount Coyote::Engine => "/"
end

To setup your database in the host (after editing config/database.yml):

1
2
3
% rake coyote:install:migrations

% rake db:create db:migrate

When you are ready to publish your application, you will need to create a gem from your engine’s gemspec, publish the gem, and then modify your hosts Gemfile dependency.

Wrapping Up

At the onset there were a few problems we wanted to solve by using engines.

How am I going to separate configuration from code?

In the walk-through above we created two separate projects, an engine and a host. The host is a full rails application independent of the engine. The host only needs to add the engine as a dependency, mount it in the routes, and then run a couple of rake tasks. Take config/database.yml for example. You can see that your users can choose to use Postgresql, while you may be using mysql. This decision is separate from your engine, because the configuration is in the host.

What about gems that may not be appropriate for some users, such as NewRelic?

You don’t want to impose instrumentation or other features that may be important to you but not appropriate for all your users. Users of your engine can add whatever gems they deem necessary for the application in the hosts Gemfile.

How can I create well-defined extension points for my application?

By using engines your users can override views, extend your models, and add new features in the host. In a follow-up post I’ll provide some examples for how users can extend your engine in these areas.

You can learn more about Rails engines by checking out some of the references I have provided below.

Pro Tips

  • Unless there’s a good reason not to, isolate your engine
  • You might be tempted to start refactoring your code while you re-write it as an engine. Avoid this pitfall. I tried it and found it too distracting and eventually backed out and just focused on porting my application to an engine.
  • Make no assumptions about your users environment – be as agnostic as possible
  • Provide solid, well documented configuration for your engine
  • Strong validation for your configuration will make your users happy and reduce the size of your issues queue on GitHub

Further Reading

Rails Guides: Engines

Rocky Mountain Ruby 2013 How I architected my big Rails app for success! by Ben Smith

This post is cross-posted from Trent Albright’s blog.

Comments