Thoughts on the Github Hack

Over the weekend, a young coder demonstrated a security vulnerability in github.com - one with wide-reaching implications. An early demonstration is at: https://github.com/rails/rails/issues/5239.

Our friend went on to make several updates to github as he experimented/demonstrated the vulnerabilities, got his account suspended, reinstated and set off a firestorm of criticism every which way.

I was ready to put it all into the "someone is wrong" pile, until I ran across this pull request on the Rails core: https://github.com/rails/rails/pull/4062.  That's some mid-90's Microsoft style arrogance right there, and on the off chance that anyone is having trouble seeing it, I figured I might add my breath to the maelstrom.

First of all: it was not right to hack github.  It's not okay to ignore the intent of security, no matter how weak the enforcement.  Much better to have pointed out the vulnerability to github, although for sure the fame wouldn't have been as bright (which is why I'm pointedly not referring to him by name in this post.)  Given github's track record, I think there's a pretty good chance they would have come clean, admitted the fault, as well as crediting its reporter.  But that's complete supposition.

That said, I don't think it's legitimate to consider Github a blameless victim.  The flaw in Rails that was exploited is well known, and well reported, and easy, if irritating, to fix.

The technical aside here is pretty simple.  In a file in config/initializers add:

ActiveRecord::Base.__send__(:attr_accessible, nil)

Then you need to white-list mass-assignable attributes in your models:

attr_accessible :name, :body, :whatever

And keep an eye on your logs for

WARNING: Can't mass-assign protected attributes: :blah

Which is a sign that you might need to add an entry for :blah into the respective model.

All pretty simple.  There are a couple of other notes, like "don't allow reference fields (e.g. :person_id) to be mass assigned" but that's the meat of it.  Put the initializer in your generator (that's much harder) and you never have to think about it again.

So, Github didn't put a simple, well-reported fix into their code.  Is that so bad?  I think so.  Github not only invited the developer community to trust them with the products of their labor, pretty much ousting SourceForge from that position in the process and firming up a development environment choice for open source work (i.e. "use git for version control"), it also invites developers to trust them with secrets.  Specifically, the secret contents of client repositories.  Heck, they get you to pay for the privilege.   So, in short, Github is taking money to keep secrets.  And by not covering a known security hole in a default Rails deploy, they were failing to uphold the trust of their paying customers.

I think Github was letting us down pretty badly.  I think an overzealous coder did a bad thing to bring that to light, but you can't argue that Github should be surprised or is blameless.  Two bad things, no one is blameless.

But the last straw for me was reading the Rails core teams' replies to a pull request to set the default for whitelisting attributes in Rails 4.0 to 'true.'  (After previous discussion concluded that making the change for 3.2 would be "too disruptive.")  That having to do attr_accessible for every model was "a lot of paperwork" - the final commend is @dhh's "I don't like this. -1"  Which is to say: we would rather put unsanitized data into the database than do the bare minimum of manual review.  And that's pretty lame.

RSpec 2.0 and before/after hooks

As of RSpec 2, the configuration interface for RSpec changed dramatically.  What used to look like:

Spec::Runner.configure do |config|
config.prepend_before(:each, :type => :controller) do
...
end
end

Now looks more like:

RSpec::configure do |config|
  config.before(:each, :line => 153) do
  end
end

One significant and interesting change is the way that before hooks are processed.  Specifically, the #before, #after, and #around methods are now part of the Hooks module, which is included in both ExampleGroup and in Configuration, so you call configure.before in exactly the same way as you do within a describe block.  Normally, you pass :each or :all, which sets the scope under which the hook will be called, but Hooks inspects the arguments for filtering metadata regardless of where you call it - I don't know that you'd want filter within an ExampleGroup, but you could...

Unfortunately, as cool as the metadata filtering capabilities are they aren't, as far as I can tell, very well documented.  The process of extracting the metadata lives in it's own :nodoc: limbo, and the attachment of metadata to a particular example is scattered throughout the RSpec code.  This, then, is an attempt to pick that apart.

Extracting Filters

When you call Hooks#before, for example (#after and #around work fundamentally the same way), the args are examined and two things are extracted:

A scope, which is the :each, :all, or :suite specification.

A metadata filter hash.  Normally, you call #before(:each, {:hash => [:of, :metadata]}), but you can instead do something like before(:all, :symbol) which will result in a metadata filter like {:symbol => true}

Again, probably if you need to add metadata inside of a describe block, you are Doing Something Wrong, but maybe there's a good reason.  The extreme (excessive?) flexibility of RSpec metadata and filtering does open up a lot of interesting possibilities.

Filter Matching

The metadata filter is used to decide if the hook should be run for a particular example block that it might apply to.  As such, it's a remarkably powerful filtering system, although there's a lot of assumptions about it's format that you need to bear in mind.

The actual mechanics of the metadata filtering happen in RSpec::Core::Metadata#apply? and #apply_condition - there's a long chain of delegation and extra-meta-programming that leads there.

The upshot is that your metadata filter will be compared to the metadata on the example key/value pair by pair, like this:

• A regular expression in the filter will match against the appropriate value for the example.
• If you pass :line_number => 17, Rspec will check to see if the example includes line 17, much like running rspec filename_spec.rb:17
• Any other Fixnum will be compared with == to the value in the metadata
• Anything else gets compared with == to the value in the metadata, after both values have been converted to a string.
• A proc like {|value| ... } will get the value of the key, and can return true for a match.

Filters can nest Hashes, which will be compared to nested Hashes in the metadata.  In other words, if you want to be able to match for metadata like

{...,  :example_group => {..., :full_description => "A very long winded example of the group", ...}, ...}

You can do something like:

before(:each, {:example_group =>{:full_description => /long winded/}})

it "should do something useful, someday", :pending => "Not this day, though"

Which is much faster than using the pending method call inside the block, and can be applied to a describe block to make the whole thing pending - especially handy when you have a before block inside that is causing problems.

In the same token, the example given in RSpec 2 documentation and announcement posts has been doing something like:

it "should not be taking this looooong", :slow => true

Since metadata can also be used to filter examples, you could use this to pull out the examples that take forever from your all-the-time specs, and run them only before a push, for instance.

What Metadata Does RSpec Give Us?

Probably the best way to figure that out is this very pragmatic approach.

A Useful Trick

Very useful for experimenting with metadata is that the proc form of the metadata has a special case: if the proc takes two arguments, the whole metadata hash will get passed into the proc, so you can inspect it at leisure.  The snippet looks like:

require 'pp'
before(:each, :bogus => proc{|val, all| pp all}) {}

{
:execution_result=>{:started_at=>Tue Jun 07 14:13:46 -0700 2011},
:type=>:controller,
:full_description=>"UserSessionsController should be authorized",
:description=>"should be authorized",
:example_group=>
  {
  :full_description=>"UserSessionsController",
  :file_path=> "spec/controllers/user_sessions_controller_spec.rb",
  :describes=>UserSessionsController,
  :description=>"UserSessionsController",
  :block=> #
,
  :line_number=>3,
  :caller=> [ ... the whole backtrace of the group ... ]
  },
:caller=> [ ... the backtrace of the example ...]
}

One of the cool-but-problematic things about metadata in RSpec is that it get's added and updated all over the codebase, and constantly over the lifecycle of an example run and extensions (like Rspec-Rails) add their own fields and values, so it's very hard to have formal documentation for what you can match.  Also, somewhat troubling, is that none of these fields are an explicit part of the RSpec API, and so might change with very little notice.  It seems like the best way to manage working with the metadata is with the above pragmatic approach.

Extending form_for in Rails 3 with your own methods

At LRDesign, we have a bunch of internal tools to make laying out Rails views more consistent. I recently upgraded and improved some of ours for Rails 3, and published them as a gem. (The published / open source ones are available at https://github.com/LRDesign/lrd_view_tools, if you're interested). One of the handy techniques we figured out (poring through the Rails code) is how to correctly add a method to FormBuilder so that you can properly use it inside a form_for block.

An example method added to forms:

Since I nearly always want <input> and <label> tags at the same time, I created a labeled_input method that lets me say this (in HAML):

= form_for(@book) do |f|
    = f.labeled_input :title
    = f.labeled_input :author
    = f.labeled_input :price

to get:

<form action="/books/new">
  <div class="labeled_input">
    <label for="book_title">Title:</label><input id="book_title" name="book[title]" type="text" />
  </div>
  <div class="labeled_input">
    <label for="book_author">Author:</label><input id="book_author" name="book[author]" type="text" />
  </div>
  <div class="labeled_input">
    <label for="book_price">Price:</label>
    <input id="book_price" name="book[price]" type="text" />
  </div>
</form>

Combined with some default CSS code in our application template that aligns the <label>s and <input>s in columns, this saves us a couple of hours setting up clean-looking forms on every new project, while significantly shortening and prettifying our view templates. (Markup Haiku, just like HAML intended.)

Implementing the extension in Rails 3

The code that handles form_for in Rails 3 is rather dense and incomprehensible and takes a while to pore through. Here's the short version to understanding it so you can add your own methods to FormBuilder properly. Since we dug through it, hopefully this will save others some time. The only Rails file you care about for this purpose is actionpack-3.0.x/lib/action_view/helpers/form_helper.rb.

• module ActionView::Helpers::FormHelper defines a bunch of helpers, like label, text_field, etc. that define helpers you use outside of a form_for. For example, text_field(@user, :title) calls this version of the helper.
• class ActionView::Helpers::FormBuilder is what's used to define the helpers you run inside a form_for. It works automatically via metaprogramming ... when loaded, it finds each helper in FormHelper (except for a few) and defines a similarly named method in FormBuilder. form_for(@user) { |f| f.text_field(:title) calls this version of the helper, which basically just calls the FormHelper version but passes the FormBuilder's @object_name as an additional first argument. In version 3.0.7, this metaprogramming happens on lines 1131-1141 of form_helper.rb.
• As a result, if you were to write a new helper in ActionView::Helpers::FormHelper that uses the same argument structure as the pre-built ones, you'd automatically get both kinds of helper. However, if you're writing your own plugin or gem and injecting new helpers, this won't happen because by the time you inject your method FormBuilder will have already done its metaprogramming (it happens when the file is loaded).
• The solution to this is that your gem needs to do the second half - defining the FormBuilder version of the helper - itself. I'll put an example below.
• Most of the helper methods work by instantiating InstanceTag, a local one-size-fits-all class to emit a form tag, and then calling the appropriate method for the kind of tag that's wanted, like to_text_field_tag. It's very confusing why the Rails team decided to do one class for InstanceTag and a bunch of different methods, rather than make subclasses of InstanceTag for each kind of tag they want; an odd OOP decision, but that's what we've got.
• InstanceTag itself has only one line: it includes InstanceTagMethods, a model that defines all the methods for the class, and which isn't used elsewhere.

So to implement a FormBuilder method yourself that you can use inside a form_for, the best way is to inject your method inside FormHelper, and then call that from a method you inject into FormBuilder. This gives you both versions of the method, in the same structure that Rails defines them. You could do this either in a helper file directly in your application, or in a gem (like we have) so you can reuse your form helpers in more than one projects.

An example implementation.

Here's a simplified construction of the labeled_input method we use at LRD. This one just emits a label and a text field and wraps them in a <div>.

Start by defining the helper:

module LRD
  module FormHelper
    def labeled_input(object_name, method, options = {})
      input = text_field(object_name, method, options)
      label = label(object_name, method, options)
      content_tag(:div, (label+input), { :class => 'labeled_input' }
    end
  end
end
ActionView::Helpers::FormHelper.send(:include, LRD::FormHelper)

This will successfully define labeled_input that you can use outside of a form_for.

Now add the FormBuilder version:

To get it working inside of a form_for, you need to add a similar method to ActionView::Helpers::FormBuilder. As mentioned above, Rails does this automatically for its own FormHelper methods using a metaprogramming approach. But since that has already happened by the time your code can inject into FormHelper, you have to do it yourself. The solution we used is to make our own FormBuilder module that manually defines the labeled_input method in the same format that FormBuilder would have done, and then auto-include that into FormBuilder when our own FormHelper module gets included. Add this stuff to the above code block:

# Inside LRD::FormHelper, add this method:
def self.included(arg)
  ActionView::Helpers::FormBuilder.send(:include, LRD::FormBuilder)
end
 
module LRD::FormBuilder
  # ActionPack's metaprogramming would have done this for us, if FormHelper#labeled_input 
  # had been defined  at load.   Instead we define it ourselves here.
  def labeled_input(method, options = {})
    @template.labeled_input(@object_name, method, objectify_options(options))
  end
end

In practice, our labeled_input method is much more complex; it handles other input types, can add instructional comments/notes to the field, and can accept a block if you want to put something other than an <input> where the text field normally goes. This guide should get you started to writing your own form_for methods quickly, but if you want to see how to do more complex things, check out the full version.

Adding more input types or other tags.

If you wanted to add an entire different tag or input type (as opposed to combining different ones, the way labeled_input does), you would probably start by building a module that you inserted into InstanceTag or InstanceTagMethods. It should define a method like MyInstanceTagModule#to_some_funky_tag() in parallel with to_input_field_tag().

Testing it with rSpec 2

Another challenge we faced was writing specs for labeled_input's behavior. It's a bit of a trick because we needed to instantiate ActionView and render some templates to check the output, but rspec-rails is written with the assumption that you will be loading an entire rails project and all the rails gems. If you want to spec just a view helper, you need to load a bunch of rspec-rails's files one by one, and then manually include RSpec::Rails::ViewExampleGroup into RSpec's configuration. We may write a separate post on this process in the future, but in the meantime, take a look at lrd_view_tools' spec_helper file and example spec for labeled_input to get the sense of it.

On the road to faster specs

Running large spec or test suites can be a bane of Rails developers. I've certainly stopped using autotest since half of our projects started exceeding 5 minutes of spec runtime. After seeing (three times!) presentations by Aman Gupta, I had spent some time with perftools trying to figure out what the heck was making my specs take so long. It's not just that more specs take more time to run: if you clock individual specs you will see identical examples run slower in a larger project. I'd seen that rspec runs can spend upwards of 60% of their time in the garbage collector, but not pursued it further than that.

A couple days ago, Jamis at the 37Signals blog took this idea further, dug into ActiveSupport::TestCase, and generated this wonderful blog post that explains his findings and how to get a 40% or more speedup in Test::Unit. His solution involves reducing the frequency of garbage collection and forcing ActiveSupport::TestCase to destroy instance variables it doesn't need anymore).

It's great, but if you do exactly what he says it won't quite work in RSpec - and RSpec users should get to enjoy this new development, too! While RSpec makes use of ActiveSupport::TestCase, it has a different set of internal instance variables, and Jamis' code will end up erasing the variables that store your actual examples. If you drop in Jamis' code to spec_helper.rb you'll see this error:

vendor/rails/activesupport/lib/active_support/whiny_nil.rb:52:in `method_missing':
 undefined method `description' for nil:NilClass (NoMethodError)

All that's needed to make RSpec happy is a little tweak to Jamis' code that protects a different set of instance variables from being unset. Just drop this blob of code at the bottom of your spec_helper.rb - I saw a 43% speed increase in one project's spec suite. (Note that if you are still using fixtures, you might need to add @loaded_fixtures and/or @fixture_cache to @@reserved_ivars; at LRD we long since abandoned fixtures in favor of factories, so I haven't tested this on spec suites with fixtures).

class ActiveSupport::TestCase
  setup :begin_gc_deferment
  teardown :reconsider_gc_deferment
  teardown :scrub_instance_variables
 
  @@reserved_ivars = %w(@_implementation @_result @_proxy  @_assigns_hash_proxy @_backtrace)
  DEFERRED_GC_THRESHOLD = (ENV['DEFER_GC'] || 1.0).to_f
 
  @@last_gc_run = Time.now
 
  def begin_gc_deferment
    GC.disable if DEFERRED_GC_THRESHOLD > 0
  end
 
  def reconsider_gc_deferment
    if DEFERRED_GC_THRESHOLD > 0 && Time.now - @@last_gc_run >= DEFERRED_GC_THRESHOLD
      GC.enable
      GC.start
      GC.disable
 
      @@last_gc_run = Time.now
    end
  end
 
  def scrub_instance_variables
    (instance_variables - @@reserved_ivars).each do |ivar|
      instance_variable_set(ivar, nil)
    end
  end
end

(Most of this code is Jamis', and I'm not taking credit for his fantastic work.)

RSpec already does a much better job of handling instance variables than Test::Unit, so the scrubbing didn't produce a big speedup for me (only about 5%). But the GC deferment did indeed give me a 43% speed improvement in the spec suite for my biggest project; run time dropped from 7m38s to 4m23s ... what a difference!

NinjaScript: Javascript so unobtrusive, you’ll never it see coming

We're happy to announce NinjaScript: a jQuery plugin for unobtrusive scripting.
NinjaScript provides:

• CSS-like language for web page behavior
• Define rich behaviors that include both event handlers and transformations.
• Durable behaviors that survive DOM alteration, with performance comparable to jQuery's live() method.
• Handy built-in behaviors for AJAX.

Motivations

Unobtrusive Javascript is one of the coming movements in web design for a reason. Separation of concerns is generally a good thing, and the idea of separating behavior from semantics is pretty obvious once you think about it. If nothing else, it makes it much easier to think about how you structure your site. Just build it out as if this were still 1998 and you couldn't trust a browser to open an alert box, much less submit AJAX, then come back and mark everything up.

On the other hand, one hears a lot about "Unobtrusive Is Hard" and how a graceful degrade takes twice as long, etc etc. At the same time, software exists to encapsulate skills.  Could be you'll be seeing a Rails plugin from LRD soon to convert the big Rails helpers into degrading versions.

Therefore: NinjaScript. Unobtrusive Javascript in a tidy package so that you can get on with your day.

What's it look like?

Here's a very simple example. Suppose you have an existing

$.behavior({
  '#coolness': $.ninja.submits_as_ajax() 
})

$.behavior({
  '.date_entry': { transform: function(elem){ $(elem).datepicker() }},
  '#work_unit_select_all': { click: selectAllWorkUnits },
 
  '#timeclock form.edit_work_unit':    $.ninja.ajax_submission({
    busy_element: function(elem){ return $('#timeclock')}
  }),
 
  '#messages .flash': {
    transform: function(elem) {
      $(elem).delay(10000).slideUp(600, function(){$(elem).remove()})
    }
  },
  '#timeclock input#work_unit_hours': {
    click: function(evnt, elem) {
      $(elem).val(hours_format(task_elapsed))
    }
  }
});

$.behavior({
  '.fun': {
    events: {
      click: function(ev, elem){ $(elem).sing_and_dance(); }
      mouseover: function(ev, elem) { $(elem).shiver_in_anticipation(); }
      //yes: mouseover.
    }
  }
});

Convection: self-hosted secure file exchange in Rails

Introducing Convection, an open-source (MIT License) project of Logical Reality Design. Need to swap files with clients or collaborators, but don't want to (or can't) trust those files to Amazon or sendbigfiles.com? Want fine-grained control over which users can see which files? Try Convection.

Lots of file exchange services exist, for example SendBigFiles.com etc. However, all of these services are hosted on someone else's hardware, and most of them share files by transferring URLs -- usually via email -- without good access control or authorization schemes.

We built Convection because a client needed to transfer files with other companies, but they needed to host the system themselves because the contracts they hold with their own clients require them not to store data on services that they don't control. The specifications Convection was built around were:

1. Hosted on our own server.
2. Downloads require a login, and files cannot be shared by email.
3. Users must log in to download files or see available files.
4. User accounts can be grouped, groups can be managed.
5. Files can be shared with an entire group.
6. Files uploaded by users default to minimal permission - visible only to the uploader and to admins.
7. All communications over SSL. (we made this optional)

Installing and hosting Convection

To run Convection, you will need a webserver capable of running a Ruby on Rails application, and a database. Setting such a thing up is beyond the scope of this post. If you have a Dreamhost account, you can set up a Rails-capable domain with a couple of clicks in their web panel. In addition to the server, you will need to set up a database (we have only tested MySQL, but Convection should work with any SQL database for which Rails/ActiveRecord has a supported adapter, including PostgreSQL and Oracle), and initialize the database with these two commands:

  > rake db:migrate RAILS_ENV=production
  > rake db:seed RAILS_ENV=production

This will generate the tables necessary for Convection to run, and create a pair of initial demo users "admin" and "user", both with password "foobar".

If you are setting up a server yourself, there are plenty of guides to deploying Rails on the web. Much of our own guide to deploying CruiseControl.rb can be used to set up any Rails application on Slicehost or any other Ubuntu Linux hosting provider.

Let me know if you're trying to deploy Convection and having trouble: if we know people are using it we may put effort into making it easier to deploy and install, and write a more thorough guide.

A few other links that may help you with deploying a Rails application, depending on your environment:

1. Using Phusion Passenger to Deploy a Rails Application on Apache
2. Deploying Rails Applications (book)

If you Google around you may find plenty of other links relevant to your particular environment.

Configuring Convection

If you log into your running Convection application as an administrator (initial user "admin", password "foobar"), an Admin Tools utility will appear in the right hand column. From here, you can access tools for creating users, and groups, and the general site configuration.

In general site config, you can set your site name and logo, set whether or not the site requires SSL access (Note: your server must already support SSL!) outgoing email and email notification preferences, add Google analytics, and an assortment of other site configuration operations that are mostly self-explanatory.

Upload progress bar: experimental feature.

If your site hosts large uploads that take a while to transfer, you can try our experimental tools to provide an upload progress bar to the user. This tool will only work if your site is served by Apache, and requires installing and configuring an optional module for Apache.

To enable this tool, follow the instructions in the README file and associated links, and turn on the progress bar setting in site preferences.

Helping us improve Convection

Convection is currently in version 1.1.4 and has been in production in two places (that we know of) for about five months as of June 8, 2010.

Please let us know if you are using Convection and enjoy it (or don't). Feel free to request features or alterations, but Convection is open source, so also please consider contributing if you have ideas!

Transactional Testing for Multiple Databases in ActiveRecord

We've been working on an app that needs to stand astride two databases - one local DB for the app itself, and another with restrictive policies about modifications that is nonetheless authoritative on many subjects. There's a fair amount of tricky interaction between the two, and testing has been a delightful challenge.

We're using the use_db plugin, and all it takes to make testing transactions happen around multiple DBs is:

In: spec/spec_helper.rb

require 'override_test_callbacks'

My concern comes from the fact that this is a direct and unfiltered monkeypatch on ActiveRecord::TestFixtures. So it relies on use_transactional_fixtures (which could certainly be used without using actual fixures, granted), and if the test transaction code moves within Rails, that's another integration to worry about. Or if we add a spec that doesn't wind up making ActiveRecord::TestFixtures load... Or if we decide to use something other than use_db...

So instead I'm using:

Spec::Runner.configure do |config|
  config.prepend_before do
    (UseDbPlugin.all_use_dbs - [ActiveRecord::Base]).each do |db|
      db.connection.increment_open_transactions
      db.connection.transaction_joinable = false
      db.connection.begin_db_transaction
    end
  end
 
  config.append_after do
    (UseDbPlugin.all_use_dbs - [ActiveRecord::Base]).reverse.each do |db|
      db.connection.rollback_db_transaction
      db.connection.decrement_open_transactions
    end
  end
end

If we weren't already using transactional fixtures, I might pull out the - [ActiveRecord::Base]. And if we were to change off of use_db, there's one place to change the transaction code. Finally, there's much less dependence on the innards of ActiveRecord - only it's published API.

Danger: ActiveRecord, param hashes, and symbol keys

Here's a little foible of ActiveRecord that cost me over an hour today. AR accepts both symbol keys and string keys when specifying attributes. Both of these are valid ways of mass assigning attributes to a Rails model:

MyModel.new(:field_1 => 'foo', :field_2 => 'bar')
MyModel.new('field_1' => 'foo', 'field_2' => 'bar')

It's convenient, often, to not have to worry about whether your keys are symbols are strings since they get converted around a bit when you pass parameters. The downside of this, however, is that it will happily accept BOTH without complaining, and will quietly default to the symbol key regardless of the order you specify them in:

>> model = MyModel.new(:field_1 => 'foo', 'field_1' => 'bar'); nil; 
>> mymodel.field_1
=> 'foo'
>> model = MyModel.new('field_1' => 'foo', :field_1 => 'bar'); nil;
>> mymodel.field_1
=> 'bar'

Okay, so that's kinda sloppy. Bad ActiveRecord! No Biscuit!

This can cause serious confusion for the unwary. When ActionController hands us a params hash, it always has String keys, like this:

>> eval params
=>  { 'article' => { 'title' => 'Awesome blog post', 'body' => 'I will make you smart' } }

But most of us, canonically, specify params and default AR values with symbols, like this:

   post :article => {:title => 'Awesome blog post', :body => 'I will make you smart'}

So we get used to thinking about them as symbols.

This means we can make mistakes like this one I made recently. Consider this block of code for a shopping cart model that pre-fills some fields for an associated Payment by pulling the address from the user's profile, to save the user re-typing their address:

class ShoppingCart < ActiveRecord::Base
  has_one :payment
 
  def build_default_payment(options = {}) 
    #prepopulate the billing address from the profile and merge
    #with params passed into options
    build_payment(prepopulated_fields.merge!(options)    
  end
 
  def prepopulated_fields
    if (addr = self.person.address)
      {
        :billing_address_1 => addr.line_1,
        :billing_address_2 => addr.line_2,
        :city => addr.city,
        :state => addr.state,
        :zip => addr.zipcode
      }
    else
      {}
    end
  end
end

Looks great, right? And if the user's address has a nil field (like no city, or no line_1), it will get overwritten by the hash merge.

Except not. I specified symbol keys in prepopulated_fields, but the hash getting passed to build_default_payment's 'options' argument has string keys, because it's coming from params. So the merge doesn't overwrite the value for :line_1, it simply adds a new key 'line_1'. So, if a user has a profile address but hadn't entered a line_1 (just city and state), and then manually entered line_1 in the payment form to submit, the Payment build during the create action was getting this hash:

build_payment({
   :line_1 => nil,
   :city => 'Pasadena',
   :state =>'CA',
   :zipcode => '91106'
   'line_1' => '100 Main St.'.
})

ActiveRecord was respecting the :line_1 => nil from the profile, and not the 'line_1' => '100 Main St.' from params. This meant that the user couldn't make payment! The payment had validates_inclusion_of line_1, and even though it was typed into the form it was getting ignored because of the nil from his profile address. Very frustrating for a user to manually type in a billing address and get back "Address Line 1 can't be blank." on every submit!

Nasty ... this one took a while to figure out. Beware of this little foible of ActiveRecord!

HOWTO: Setting up CruiseControl.rb on Slicehost

Continuous Integration is a key tool for collborative development, and CruiseControl.rb is the tool of choice for many Ruby and Rails teams, including us at Logical Reality.

Unfortunately, setting up CC.rb for a team can be a relatively frustrating experience: this guide (the first of a series of HOWTOs by LRD) will walk you through every step of setting up a team instance of CruiseControl.rb on a low-cost server from Slicehost.

Step 1: Lease a Ubuntu Slicehost account

I recommend a 384 slice or a 512 slice, as 256MB or RAM is pretty light for anything involving a Rails application.   Our CI server runs on a 512 slice; if you are running it on a smaller slice please let us know how it performs.

I used Ubuntu 9.10 (Karmic) for this post.

Step 2: Create a working user

Slicehost configures slices with an active root account - definitely a Ubuntu no-no - and no user account. Ick! Let's start by creating a user account with sudo access to do everything from. Log in as root using the information Slicehost sends you, run this (replace 'usename' with whatever name you like) and fill in the information it asks for:

 # adduser username

Then edit /etc/sudoers and add this line to the bottom of the file:

username    ALL=(ALL) ALL

Log out, and log back in as the user you've now configured, to make sure it work.

Step 3: Installing packages and gems

Reset your timezone:

sudo dpkg-reconfigure tzdata

Install a whole bunch of packages you'll want for running Rails applications and hosting CruiseControl:

sudo aptitude install locate emacs git-core ruby build-essential \
libopenssl-ruby ruby1.8-dev irb  apache2 apache2-mpm-prefork \
apache2-prefork-dev sqlite3 rubygems mysql-server mysql-client

Go grab a cup of coffee while those install. The mysql install will ask you to set a root password. Do so, and write it down for later use. When all the installs are done, come back and install the ruby gems you'll be needing:

sudo gem install sqlite3-ruby passenger mysql metric_fu reek roodi

Step 4: Assorted server configuration

Add this line to the bottom of your ~/.profile to put your gems in your path:

PATH="$PATH:/var/lib/gems/1.8/bin/"

And source it:

. ~/.profile

Some assorted config: set up the passenger module for Apache, set your hostname, and make /etc/hosts readable. (For some bizarre reason, /etc/hosts was only readable by root on my slice, and that has a tendency to break things down the road).

sudo /var/lib/gems/1.8/bin/passenger-install-apache2-module
sudo emacs /etc/hostname  # set it to "your.hostname.com"
sudo /bin/hostname -F /etc/hostname
sudo chmod a+r /etc/hosts

Step 5: Configure Passenger and Apache

We'll run CruiseControl.rb with Apache and Passenger. Start by enabling the Passenger module. The command below will walk you through a super-easy configuration:

sudo /var/lib/gems/1.8/bin/passenger-install-apache2-module

When the command completes, it will give you three lines to paste into your apache config, they should look pretty much like these below. Put these lines at the top of /etc/apache2/apache2.conf. I included the hostname I set in the previous step as ServerName.

LoadModule passenger_module /var/lib/gems/1.8/gems/passenger-2.2.8/ext/apache2/mod_passenger.so
PassengerRoot /var/lib/gems/1.8/gems/passenger-2.2.8
PassengerRuby /usr/bin/ruby1.8   
 
ServerName your.hostname.com

To set up the application itself, edit /etc/apache2/sites-available/default to look like this:

<VirtualHost *:80>
        ServerAdmin administrator@your-email-domain.com
        DocumentRoot /u/apps/cruisecontrol/public
        RailsEnv production
        RailsBaseURI /
        ServerName <IP Address from Slicehost>
        ServerAlias your.hostname.com
        SetEnv PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/var/lib/gems/1.8/bin/
 
        ErrorLog /var/log/apache2/error.log
 
        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel warn
 
        CustomLog /var/log/apache2/access.log combined
</VirtualHost>

Make a home for the app. (I use /u/apps/ as a convention for apps in apache. Use whatever you like, but make sure your DocumentRoot in your config file above matches.)

sudo mkdir -p /u/apps

Step 6: Download and install CruiseControl.rb

Download cruisecontrol.rb from RubyForge (Check for the current version first; it was 1.4.0 when I installed), and give ownership to the web user www-data:

cd /u/apps
sudo wget http://rubyforge.org/frs/download.php/59598/cruisecontrol-1.4.0.tgz
sudo tar -zxf cruisecontrol-1.4.0.tgz     
sudo mv cruisecontrol-1.4.0 cruisecontrol   
sudo chown -R :www-data cruisecontrol

Give environment.rb to the web user; this prevents an Errno::EACCES accessing environment.rb from Passenger (see discussion at this forum post).

sudo chown www-data:www-data config/environment.rb

Turn off the built-in htaccess, it will break Passenger:

sudo mv public/.htaccess public/.htaccess-disabled
cd config  
sudo cp site_config.rb_example site_config.rb

Step 7: Setting up the user environment

CruiseControl.rb prefers, by default, to put project builds in the running user's ~/.cruise directory. This is unfortunate because the standard user for running Apache, www-data, doesn't have a user directory! There are ways to override this, but I've found that they cause significant problems down the line.

An example of the problem is letting CC.rb check out your source code. If you authenticate access to GitHub or another code repository with SSH, CC.rb — running as www-data — won't be able access your repo since www-data doesn't have a ~/.ssh directory to put the keys in!

After much hacking, I came to the unhappy conclusion that the best solution is simply to let CruiseControl.rb have its way and give user www-data a home directory. Boo, hiss, but here we go:

sudo /etc/init.d/apache2 stop   
sudo usermod -d /home/www-data www-data      
sudo usermod -s /bin/bash www-data
sudo /etc/init.d/apache2 start

If you give www-data standard config files as well, then you can set the PATH so that user www-data can find your gems, and you can set up ssh keys so that CruiseControl.rb can securely check out projects from GitHub or whatever source code repository you're using:

sudo cp -r /etc/skel /home/www-data 
sudo chown www-data:www-data /home/www-data
sudo su www-data               
cd
mkdir ~/.ssh  
cd ~/.ssh
ssh-keygen -t rsa
cat id_rsa.pub

Add this line to the bottom of ~/www-data/.profile:

PATH="$PATH:/var/lib/gems/1.8/bin/"

Re-start Apache:

sudo /etc/init.d/apache2 restart

At this point, you should be able to load CruiseControl.rb in a web browser at the IP address given to you by Slicehost, or at the domain name if you've set up DNS and it's resolving. Congratulations, you have CC.rb up and running! One last thing to configure.

Running CruiseControl.rb will have created a configuration directory . ~www-data/.cruise. You'll want to edit ~www-data/.cruise/site_config.rb to set two options. Uncomment and set appropriate values for this line:

 Configuration.email_from = 'cruisecontrolrb@mydomain.com'
 Configuration.dashboard_url = 'http://my.cruisecontrolrb.host/'

Okay, it's time to get a project installed!

Step 8: Setting up your first project

I'll use Logical Reality's open-source project, Convection, as an example project for CruiseControl.rb. This works best if you run it as user 'www-data'.

The command for adding a new project is really simple:

cd /u/apps/cruisecontrol 
sudo su www-data
./cruise add Convection -r git://github.com/LRDesign/Convection.git -s git

This will set up the build in ~www-data/.cruise/projects/Convection.

Create a test database for the application. For Convection, I'm going to use mysql, and prefix my database name with 'ci' for Continuous Integration.

mysqladmin -u root -p create ci_convection

We don't want to put a functioning database.yml in our GitHub repository, but at the same time we want CruiseControl.rb to be able to build and test the app without help from the user. For all our Rails projects, we use a custom rake task that generates a database.yml from command-line arguments, then rebuilds the database, run the specs, and generate output with metric_fu. For an example of how to do this, look at our integration.rake and ERB database.yml template from Convection.

To configure CruiseControl.rb to run Convection this, we need to add that task to the configuration file for this project. Edit ~www-data/.cruise/projects/Convection/cruise_config.rb so that it looks like this:

 
Project.configure do |project|
 
  # Send email notifications about broken and fixed builds to email1@your.site, email2@your.site (default: send to nobody)
  project.email_notifier.emails = ['sysadmin@lrdesign.com', 'judson@lrdesign.com']
 
  # Set email 'from' field
  project.email_notifier.from = 'sysadmin@lrdesign.com'
 
  # Build the project by invoking rake task 'custom'
  # project.rake_task = 'custom'
 
  # Build the project by invoking shell script "build_my_app.sh". Keep in mind that when the script is invoked,
  # current working directory is <em>[cruise&nbsp;data]</em>/projects/your_project/work, so if you do not keep build_my_app.sh
  # in version control, it should be '../build_my_app.sh' instead
  # project.build_command = 'build_my_app.sh'
  project.build_command = 'rake ci:run[localhost,root,<YOUR_MYSQL_ROOT_PASSWORD>,ci_convection] --trace RAILS_ENV=test'
 
  # Ping Subversion for new revisions every 5 minutes (default: 30 seconds)
  # project.scheduler.polling_interval = 5.minutes
end

Step 9: There is no step nine!

Okay, so it's not the simplest thing in the world to set up. But if you've done everything above correctly, you should have a running server your team can use for continuous integration. If you've included metric_fu in your build task, you should get both test output and a wealth of useful code metrics.

Did this sequence work for you? Did I omit a step or misspell a command? Let me know in comments, and I'll update/correct the post.

RailsTutorial.org launched

The new Ruby on Rails Tutorial book and website by Michael Hartl has launched at RailsTutorial.org.   Hartl is the author of RailsSpace and cofounder of the Insoshi Ruby on Rails social networking platform.

Logical Reality did the logo and layout design work for Rails Tutorial.