Fixing problems with sphinx search

I’ve been working a lot this week with sphinx and ultrasphinx on a project that’s a fork of Insoshi.    Insoshi is in the process of switching search from ferret to sphinx, and sphinx has been integrated into the Insoshi edge branch.

I’ve had dozens of problems, in fact it’s fair to say I’ve spent upwards of 15 hours just debugging ultrasphinx and getting my tests to pass.   There were several problems; here are the main three and how I fixed each one.

This should be useful to anyone upgrading Insoshi to the sphinx version, or to anyone else trying to get ultrasphinx working in their Rails project. I definitely don’t recommend starting with this post if you’re just starting out with sphinx. Instead, go read this much better introductory tutorial from the guys over at Insoshi. Then if you have problems, come back here and you may find solutions.

Getting search tests (or specs) to pass with sphinx

This one is pretty simple, in retrospect, but it can be frustrating and opaque if you are used to ferret.  Unlike ferret, sphinx (at least via ultrasphinx) runs only via a daemon.   Where acts_as_ferret uses a daemon only for the production environment and just accesses the index files directly in test or development, ultrasphinx can only get to the indexes through the daemon.

So, to run your tests, you just build up the indexes for test and run them.  In this case, I’m running the specs for Insoshi’s searches controller:

From the command line in $RAILS_ROOT:

rake db:test:prepare
rake ultrasphinx:configure RAILS_ENV=test
rake ultrasphinx:index RAILS_ENV=test
rake ultrasphinx:daemon:start RAILS_ENV=test
script/spec spec/controllers/searches_controller_spec.rb

The problem, of course, is that it doesn’t work!   The reason is that db:test:prepare creates the structure of your database, but doesn’t load any of your fixtures as data: the test db is empty..  So when you run the index command, an empty index is built.   You can see this from the output of that first index command, which will look something like this:

collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.078 sec, 0.00 bytes/sec, 0.00 docs/sec

Ultrasphinx has built an empty index.

The solution

The solution, believe it or not, is to run the tests, let them fail, re-index, and run the tests again (Many thanks to Long Nguyen at Insoshi for helping me figure this one out):

rake db:test:prepare
rake ultrasphinx:configure RAILS_ENV=test
rake ultrasphinx:index RAILS_ENV=test
rake ultrasphinx:daemon:start RAILS_ENV=test
script/spec spec/controllers/searches_controller_spec.rb   #FAIL!!
rake ultrasphinx:index RAILS_ENV=test
script/spec spec/controllers/searches_controller_spec.rb   #PASS!!

The first attempt to run the specs loads the fixtures, and leaves them in the database, thus letting the subsequent index command build an actual index.

Running sphinx for both test and development environments at the same time

The next big challenge was enabling behavior-driven development. I like to work with autotest and growl running constantly in the background. But this was tough to do with sphinx, because the daemon needed to be stopped and re-started, and the index re-created for each environment, alternately running all of the above commands either with or without RAILS_ENV=test.

The solution is to set up your ultrasphinx base configuration to completely separate both the test and development indexes and to let the daemons for the two environments listen on different ports. I had tried something like this and come close, but not quite, when Long at Insoshi again bailed me out. You need to change the port (in two places), and the paths of the logs, pidfile, and index directories so that test and development daemons are using entirely separate resources. Here’s a diff of my test.conf and default.conf:

33c33
<   port = 3312
---
>   port = 3322
35c35
<   log = log/searchd.log
---
>   log = log/searchd_test.log
39c39
<   pid_file = log/searchd.pid
---
>   pid_file = log/searchd_test.pid
50c50
<   server_port = 3312
---
>   server_port = 3322
57c57
<   sql_range_step = 5000   
---
>   sql_range_step = 999999999   
64c64
<   path = sphinx
---
>   path = sphinx_test

The sql_range_step is related to the next issue, which is that sphinx does not play well with foxy fixtures. Anyway, make the above changes and you should be able to run test and development sphinx daemons at the same time:

rake db:test:prepare
rake ultrasphinx:configure
rake ultrasphinx:configure RAILS_ENV=test
rake ultrasphinx:index
rake ultrasphinx:index RAILS_ENV=test
rake ultrasphinx:daemon:start
rake ultrasphinx:daemon:start RAILS_ENV=test

If it worked, you should see separate indexes in $RAILS_ROOT/sphinx and $RAILS_ROOT/sphinx_test, and two daemons running, which you can confirm with ps waux | grep searchd:

evan      1339   0.0  0.0    78100    292 s000  S     5:37PM   0:00.52 searchd --config <YOUR_RAILS_ROOT>/config/ultrasphinx/test.conf
evan      1326   0.0  0.0    78100    292 s000  S     5:36PM   0:00.68 searchd --config <YOUR_RAILS_ROOT>/config/ultrasphinx/development.conf

Getting sphinx to play well with foxy fixtures

The next problem I discovered was that on some machines, but not others, running my search specs would result in these weird errors:

1)
ActiveRecord::RecordNotFound in 'SearchesController Person searches should search by name'
Couldn't find Person with ID=328556765
/var/www/domains/unithrive/vendor/plugins/ultrasphinx/lib/ultrasphinx/search/internals.rb:308:in `reify_results'
/var/www/domains/unithrive/vendor/plugins/ultrasphinx/lib/ultrasphinx/search/internals.rb:286:in `each'
/var/www/domains/unithrive/vendor/plugins/ultrasphinx/lib/ultrasphinx/search/internals.rb:286:in `reify_results'
/var/www/domains/unithrive/vendor/plugins/ultrasphinx/lib/ultrasphinx/search.rb:362:in `run'
/var/www/domains/unithrive/vendor/plugins/ultrasphinx/lib/ultrasphinx/search/internals.rb:352:in `perform_action_with_retries'
/var/www/domains/unithrive/vendor/plugins/ultrasphinx/lib/ultrasphinx/search.rb:342:in `run'
/var/www/domains/unithrive/app/controllers/searches_controller.rb:38:in `index'
./spec/controllers/searches_controller_spec.rb:51:
script/spec:4:

When I poked into this “Couldn’t find Person with ID=328556765″ error, it seemed like sphinx was almost working. The index was set up, and the search was finding someone in the index during the test. Ultrasphinx was passing back the id 328556765, which didn’t exist in the database. So why would Sphinx “find” a record in its index but then pass back an ID for a database record that didn’t exist?

And furthermore, why would it work on one machine, but not on another?

The brainstorm came when I checked what the actual database IDs were for this particular record, with Person.find_by_name(”fixtures’ name”).id. On machines where it worked, the id was a huge number (is it generally is with foxy fixtures), but on machines where it didn’t work, the id was an even huger number.

Sphinx tries to make sure that all items that get indexed have a different index in sphinx, and it does this by multiplying all of your id’s by N, where N is the number of models getting indexed, and adding an offset of 0 for the first model, 1 for the second, etc. This guarantees that every record from every table will have a unique id. In the case of this application, all of my Person records were getting indexed by sphinx as (Person#id * 4 + 2).

Danger, Will Robinson: 32-bit int rollover!

The problem is that foxy fixtures generate their own ids from a hash of the fixture label, and those ids can be anywhere in the 32-bit unsigned integer space. But Sphinx also stores ids as 32-bit unsigned integers. This means if you happen to get a large fixture id, and then sphinx multiplies it by 4 (or whatever; it could be higher if you have more indexed models), your id will rollover and come out as (id * N + n) % (2^32). Sphinx will store that result, and then when it finds the record in a search, it will try to recreate the original id by subtracting n and dividing by N … giving you the wrong id. Your test will fail to find the record.

Incidentally, this problem with foxy fixtures is why your test.base file needs the line sql_range_step = 999999999. Sphinx builds indexes by searching a few ids at a time. But the ids generated by foxy fixtures are so big that if sphinx only collects them in ranges of 5000 at a time, it will take forever to find them all.

After some googling, I found that these issues are discussed in a thread over at RubyForge.

The solution

I’m working on a plugin that monkeypatches foxy fixtures to create sequential, low-numbered IDs. In the meantime, you can just compile sphinx to support 64-bit ids, which should give you plenty of headroom to handle foxy fixture ids multiplied by N in sphinx^*:

In your sphinx source directory:

configure --enable-id64
make
sudo make install

That should do it. Let me know in comments if any of this information helped you.

^*At least until you start approaching 2^32 models in your application, that is.

Rails 2.1 gotcha: don’t name an ActiveRecord field ‘changes’ !

In one of my Rails projects, I maintain a user-visible log of updates to the database by recording entries into a table that looks like this (from schema.rb):

create_table "log_entries", :force => true do |t|
    t.integer  "user_id",    :limit => 11
    t.datetime "created_at"
    t.datetime "updated_at"
    t.string   "item_type"
    t.integer  "source_id",  :limit => 11
    t.string   "table"
    t.string   "action"
    t.text     "changes"
  end

Whenever a record is created, updated, or deleted, I create and save an instance of LogEntry, containing for example {:table => 'task', :action => 'update'} and in the ‘changes’ column I save a serialized hash showing which attributes of the task object changed before and after save. In addition, I save which logged-in user made these changes, and when.

This is convenient, it gives my client a log that’s much more user-accessible and allows them to easily back-trace who did what to the database and when, which is important for their process and certain certifications.

When I upgraded the project to Rails 2.1 a couple of weeks ago, most of the code still worked fine, but the hashes showing the object changes were no longer showing up in views of the log. The culprit turned out to be Rails 2.1’s new dirty feature. Why? Because it adds the method ‘changes’ to ActiveRecord::Base. This is a great new feature that lets you know what has changed to an ActiveRecord object since you loaded it from the database, with all kinds of benefits like doing more selective updates, or not writing to the database at all if no changes have occurred, thus reducing system load if you call save! a lot.

Unfortunately, the new method ‘changes’ was obscuring my attribute ‘changes’ in some circumstances, and at the very least confusing the heck out of me, the programmer.

The solution, of course, is not to name your fields anything that corresponds to any Ruby core method or any method of ActiveRecord::Base. I fixed my problem with a migration to rename the column ‘details’:

def self.up
  rename_column  :log_entries, :changes, :details
end

def self.down
  rename_column  :log_entries, :details, :changes
end  

Getting <select> options in the right order

Sometimes you may want to generate a selector with a fixed set of options.   In one recent task of mine, we needed a selector for an integer 1 through 10, but the client wanted them labeled also with text to identify how the numbers mapped to the words “High” “Medium” and “Low”. (We were selecting the priority level of a task in a project management application). Essentially, I want to generate this HTML output:

<select id="task_priority" name="task[priority]"> 
    <option value="1">1 - High</option> 
    <option value="2">2 - High</option> 
    <option value="3">3 - High</option> 
    <option value="4">4 - Med</option> 
    <option selected="selected" value="5">5 - Med</option> 
    <option value="6">6 - Med</option> 
    <option value="7">7 - Med</option> 
    <option value="8">8 - Low</option> 
    <option value="9">9 - Low</option> 
    <option value="10">10 - Low</option>
</select>

These are fixed name/value pairs, so it made sense to me to store them as a constant hash in my Task model:

Hash in Task.rb

PRIORITY_OPTIONS = { "1 - High" => "1", "2 - High" => "2", "3 - High" => "3",
"4 - Med" => "4", "5 - Med" => "5", "6 - Med" => "6", "7 - Med" => "7",
"8 - Low" => "8", "9 - Low" => "9", "10 - Low" => "10"  }

However, I was rather dismayed to discover that this didn’t produce the results I wanted, because Ruby hashes (in Ruby 1.8.6) do not preserve order. This is what came out:

That’s not what I wanted! I knew that select will also take arrays, but of course I needed separate name/value pairs, which I can’t get with just straight arrays. I spent a while playing around with OrderedHash, which exists in Rails but is essentially undocumented and, as it turns out, does not support any useful functions of Hash like merge! and insert! that might make it easy to construct my list of options.

The Fix: Array of Arrays, or Array of Hashes

The documentation is not entirely clear on this, but if you send an array of 2-element arrays to select, rails will use the two elements of each inner array as if they were key and value pairs, and because the entire structure is an array it will preserve order. So, to get the results I wanted, I just need to change my constant to this:

PRIORITY_OPTIONS = [ ["1 - High", 1], ["2 - High", 2], ["3 - High", 3],
["4 - Med", 4], ["5 - Med", 5], ["6 - Med", 6], ["7 - Med", 7],
["8 - Low", 8], ["9 - Low", 9], ["10 - Low", 10] ]

And I get the result I was looking for :

As it turns out, the way ActionView processes the options is fairly general: if you pass it any enumerable object, it will iterate that object, and for each element will check to see if that element supports the methods :first and :last (and isn’t a string). If so, it will generate an option with the text set to element.first and the value set to element.last. If it was a string, or didn’t support first and last, both the text and value of the option are set to the element itself.

Testing it

Here’s a handy function I use for testing it the presence of a selector. You pass it the name of your selector, the name of hash or array of options (in the any format supported by select), and optionally the value of an item that should be pre-selected, and it will assert the existence of each of those things. If you need to handle selectors with multiple selections, you can just wrap the last assertion in a loop.

Drop this in test/test_helper.rb

  # Assert existence of form select input
  # the <option> tags can be passed either as 
  # a hash like this: {'foo'=>'bar', 'baz'=>'qux'} 
  # or an array of arrays like this: [ ['foo', 'bar'], ['baz', 'qux'] ]  
  # either will be asserted to match:
  #         <option value='bar'>foo</option>
  #         <option value='qux'>baz</option>
  # or as an array like this:
  #   [ 'foo', 'bar' ] will be asserted to match
  #         <option value='foo'>foo</option>
  #         <option value='bar'>bar</option>
  def assert_select_input(name, option_values, options={})
    attributes = { :name => name }

    # first assert that the select tag exists
    selector = { :tag => "select", :attributes => attributes }
    assert_tag selector   
        
    option_values.each do | opt |
      if !opt.is_a?(String) and opt.respond_to?(:first) and opt.respond_to?(:last)
        assert_tag({:tag => "option", :attributes => { :value => opt.last },      
          :parent => selector, :content => opt.first })
      else
        assert_tag({:tag => "option", :attributes => { :value => opt },      
          :parent => selector, :content => opt })               
      end
    end        
    
    # check for the pre-selected option, if any
    if options[:selected] 
      assert_tag :tag => "option", :attributes => {:selected => 'selected', 
        :value => options[:selected] }
    end    
  end  

As an example of how to use it, here’s my method for testing the task priority selector pictured above:

  # asserts a <SELECT> field, targeting task[priority], containing ten
  # appropriately formatted <OPTIONS>, and with selected_num (if any) the
  # pre-selected option
  def assert_priority_select(selected_num = nil)
    options = selected_num ? { :selected => selected_num } : Hash.new
    assert_select_input "task[priority]", Task::PRIORITY_OPTIONS, options  
  end

Where the array specified in the Task model looks like this:

  PRIORITY_OPTIONS = [ ["1 - High", 1], ["2 - High", 2], ["3 - High", 3], 
    ["4 - Med", 4], ["5 - Med", 5], ["6 - Med", 6], ["7 - Med", 7],
    ["8 - Low", 8], ["9 - Low", 9], ["10 - Low", 10] ]

Using redirect_to :back to improve user interfaces

It’s a scourge that inflicts nearly every website: the dreaded empty page stating “Your search returned no results.” Or even worse, the search results page with a header “Search results” and zero content, the footer immediately following the header. This is not only useless, but also confusing to the user, because it takes the reader several seconds to even realize that the search returned no results.

For example, I just got this one on the otherwise lovely workingwithrails.com when I searched for RoR developers in Pasadena, CA:

This is lame, lame, lame. A search results page with no results is guaranteed not to be useful to the user! From a user interface perspective, it’s akin to redirecting a user to a login page after they try to access some resource, and then sending them back to the front page or a super-lame “thank you for logging in” page afterwards.    The user should always see a useful page, and should never have to waste clicks getting back to what they wanted.

It’s not only bad UI, it’s also entirely unnecessary, because in Rails you can fix this for good with two lines of code.

Avoid wasted pages and annoyed users with redirect_to: back

If your user’s search doesn’t find anything, don’t make them waste their time at a whole page that tells them only that.  Instead just send them straight back to the search page.  Better yet, redirect them back to wherever they came from, since they might well be able to invoke search from any number of different contexts in your application. You don’t need a whole page to tell them the search didn’t find anything - a flash notice is more than sufficient:

Redirecting back with a flash after an empty search:

def search
  @results = YourModel.execute_some_search(params[:some_parameter]);
  if @results.size == 0
    flash[:notice] = "No items found matching '#{params[:some_parameter]}'"
    redirect_to :back
  end
end

Testing redirect_to :back

Testing redirect_to :back isn’t completely trivial, because of course there’s no previous page when you render a page in a test context. So, when you try to run a controller that redirects back, you’ll see an error. The easiest thing to do is to artificially set a previous page. I have these two methods in test/test_helper.rb:

def fake_referer
    @request.env['HTTP_REFERER'] = 'http://previous_page'
  end
  
  def assert_redirected_back
    assert_redirected_to 'http://previous_page'
  end

Then anytime I am working on a controller that involves back redirects, I drop fake_referrer in my setup method and then use assert_redirected_back. For example, in a search controller test (this is from a controller that has an action called ‘contents’ for searching items by text content - the search itself uses ferret):

def setup
    super
    fake_referer
  end

  def test_contents_search_fails
    get :contents, :contents => "some&junk*that(will@never)match$anything"
    assert_not_nil flash[:notice]
    assert_redirected_back
  end

Installing Insoshi on a Dreamhost Account

With the Insoshi social networking platform rapidly gaining in popularity, I thought it might be useful to folks to know how to install it on the ever-popular Dreamhost shared account. If you need a Dreamhost account, please consider using the promo code “LRDESIGN” when you sign up. It will save you $50 on your first year of membership, and will help me with my site hosting expenses so I can keep this blog going.

So, there are still some possible drawbacks to this approach, but I was able to get a running install of Insoshi on my Dreamhost account with this sequence. You might want to read to the bottom of this post to learn about the difficulties with acts_as_ferret and Dreamhost before you commit to running your insoshi site on DH. Hopefully these problems will have a solution soon, and I’ll update this post if/when they do.

If you use this method and it works (or doesn’t!) please let me know in comments.

1) Set up a domain

Use the DH control panel to create a new fully hosted domain for your Insoshi site, for example yourdomain.com or insoshi.yourdomain.com. I will use “insoshi.yourdomain.com” through the rest of this post to indicate the domain that you want to use to run insoshi. I set up http://insoshi.lrdesign.com/ in the process of writing this post, but I can’t guarantee that it will stay up.

When you set up the domain:

• Make sure that fastcgi support is selected.
• Set “specify your web directory” to point to /home/username/insoshi.yourdomain.com/public/

2) Set up mysql databases

1. In the dreamhost panel, select “Goodies -> Manage MySQL”
2. Scroll down to “create a new mysql database”

I used these example settings:

• database name: insoshi
• use hostname: mysql.yourdomain.com (use “new hostname” to create this if you do not already have it)
• New user: insoshi
• New password:

If you want to run tests, you should create a second database called insoshi_test; you can leave the other settings the same.

3) Download the tarball of the current insoshi distribution:

cd
wget http://insoshi.com/home/tarball

You’ll get a tarball with a name like “insoshi-insoshi-e1fd8b8e440c9f3ab34161d4e87de78e956c1012.tar.gz”. Unzip the tarball and copy the contents to the directory you want the website to appear in:

tar xzf insoshi-insoshi-e1fd8b8e440c9f3ab34161d4e87de78e956c1012.tar.gz
cp -r insoshi-insoshi-e1fd8b8e440c9f3ab34161d4e87de78e956c1012/* insoshi.yourdomain.com/

4) Set up your database.yml file

cd ~/insoshi.yourdomain.com/config
cp database.example database.yml

Edit ~/insoshi.yourdomain.com/database.yml and make it look like the following, where <password> is the password you chose in the previous step:

development:
  adapter: mysql
  database: insoshi
  username: insoshi
  password: <password>
  host: mysql.lrdesign.com
  port: 3306

# Warning: The database defined as 'test' will be erased and
# re-generated from your development database when you run 'rake'.
# Do not set this db to the same as development or production.
test:
  adapter: mysql
  database: insoshi_test
  username: insoshi
  password: <password>
  host: mysql.lrdesign.com
  port: 3306

production:
  adapter: mysql
  database: insoshi
  username: insoshi
  password: <password>
  host: mysql.lrdesign.com
  port: 3306

5) Run the insoshi install script.

This will migrate the database and do some insoshi-specific setup. It’s also an excellent way to check that you’ve configured your database.yml correctly.

cd ~/insoshi.yourdomain.com
rake install

If it works, you should see a bunch of migrations (22 as of the current version of insoshi). If not, go back and figure out what’s wrong with your database.yml file.

6) Get rails working

These instructions are an adaptation of the instructions at the Dreamhost Wiki page about Rails.

Generate a dummy rails app and copy the dispatch scripts to your insoshi install:

cd ~
rails dummy
cp dummy/public/dispatch.* insoshi.yourdomain.com/public/

Edit ~/insoshi.yourdomain.com/public/.htaccess file to enable fastCGI. Change the line with dispatch.cgi to look like this:

RewriteRule ^(.*)$ dispatch.fcgi [QSA,L]

change your permissions on the public and dispatch files:

cd ~/insoshi.yourdomai.com
chmod 755 public
chmod 755 public/dispatch.*  

I also had to make my log files writeable:

chmod a+w log/
chmod a+w log/*

7) Start the ferret server

I was not able to get insoshi to run in production mode on Dreamhost at first
because in production mode it needs the ferret server (text search) to be
running or it will refuse to load some of the models. (In test and development
mode, acts_as_ferret will access the ferret databases directly, so this problem only appears in production). Based on this post, I found I could get it working by running:

script/ferret_server start -e production

At this point, I was able to load Insoshi in my browser.

Unfortunately, there are still some problems

I am pretty confident that the approach to running ferret_server above will not be a long term solution, because Dreamhost kills any processes that you leave running for more than a few hours. So ferret_server will go down after a while, and with it your site, so this will probably only get your insoshi site up for a few hours before you have to restart ferret_server. This basically means there’s no good way to use the rails plugin acts_as_ferret on a DH shared account, and unfortunately Insoshi depends on AAF.

Possible Workarounds

You could try putting the startup command in a cron script, to restart ferret_server when Dreamhost kills it, for example:

0,15,30,45 * * * * cd ~/insoshi.yourdomain.com; script/ferret_server start -e production

would attempt to start the ferret server every fifteen minutes. But that might run afoul of Dreamhost server policies (does anyone know for sure?), and in any case your site would still be down in between the time DH killed the ferret_server process and your cron job started again.

You can also alter config/ferret_server.yml to have ferret treat production mode the same as development, directly accessing the ferret database and bypassing ferret_server entirely. However, if you get concurrent access with multiple users, you are very likely to get a corrupted ferret database with that approach.

Hopefully, some permanent solutions?

The Insoshi guys are working on replacing ferret with Sphinx, and that may be a permanent solution to this problem.

You also may want to consider lobbying Dreamhost to allow users run persistent processes like ferret_server. If you are a Dreamhost subscriber you can vote for this feature by following this link to Dreamhost’s Policies Suggestions and voting for “Be able to run simple, persistent scripts!”.

Fixing the test setup bug in Rails 2.0.2

Help! My setup method is not running!

Much to my annoyance (and seemingly, very few other developers), Rails 2.0.2 introduced a bug in ActionController::TestCase — setup methods for tests are not run. It worked in Rails 2.0.1, and broke in Rails 2.0.2. This was fixed in the very next changeset, Revision 8442, but mysteriously it was never rolled out in a 2.0.3. That was five months ago, and given that this basically breaks all tests it’s pretty mysterious that there is still no released version of Rails with this bug fixed.

(NOTE: This was true when written. Now, Rails 2.1 is out, and it includes the fixes. You can use the workarounds below if your project depends on Rails 2.0.2, otherwise I recommend that you simply upgrade to 2.1!)

I’m pretty sure I’m not the only Rails developer who writes tests! So I’m surprised there hasn’t been more of an outcry about this bug. But the Rails dev process has always been rather mysterious to me. So: we need a workaround. I know of two.

Option one: subclass Test::Unit::TestCase

This is what I’ve been doing till now: just going back and writing my tests Rails 1.x style. It’s annoying and clunky: the whole point of ActionController::TestCase in Rails 2.0 was to eliminate all the redundant and non-DRY junk you had to put in your test setup to establish the instance variables @request, @controller, etc. In any case, it looks like this:

class FooControllerTest < Test::Unit::TestCase
  fixtures :tasks, :users, :people
  
  def setup
    super
    @controller = FooController.new
    @request    = ActionController::TestRequest.new
    @response   = ActionController::TestResponse.new
    # ... your setup code here ...
  end

Option two: use a revision from edge with the bug fixed

Normally, I abhor using unreleased software in a production environment. I definitely don’t use EDGE rails in my released software for clients in general … the hassle is too much to maintain with the constant changesets. I’d rather understand and work with stable software, even if that means I’m not one of the cool Rails edgerati: stable software for my customers is the higher priority.

However, in this case I’m making an exception, and I’ve started using revision 8442 of Rails as if it were a released 2.0.3, because this bug just annoys me too much. Installing it is simple. If you’re using frozen Rails in your application (and you should be!), it’s just this at the command-line, in your application root:

$rake rails:freeze:edge REVISION=8442

That should give you a “2.0.3″ that fixes this irritating bug and lets you write your tests and setup the way God intended:

class FooControllerTest < ActionController::TestCase
  fixtures :tasks, :users, :people
  
  def setup
    super
    #... your setup code here ...
  end

Don’t forget the call to super, otherwise you’ll clobber the setup method in ActionController::TestCase and your tests won’t work.

Important note for subversion users!!

If you are maintaining your project in subversion, re-freezing your rails code with the above command will hose all of your .svn/ directories under vendor/rails. This will give you a few minutes of headaches as you get things straightened back out. I recommend deleting your old rails freeze first, committing, and then re-freezing:

$svn rm vendor/rails
$svn commit -m "removed old rails version"
$rake rails:freeze:edge REVISION=8442
$svn add vendor/rails
$svn commit -m "froze rails to revision 8442"

I maintain most of my projects in Git, now, so this wasn’t a problem, but I wasn’t thinking when I converted one of my older projects that’s still in svn. It was definitely a headache when the working repository broke, so avoid that if you can.

So there you go. I never thought I’d recommend using unreleased software in production, but just hold your nose and pretend that Rev 8442 is “Rails 2.0.3″ and you’ll probably be fine. I haven’t found any problems with it, myself.

Tabbed Panels in Rails, Part 1

The tabbed panel is one of the most powerful GUI tools: an easy, intuitive way to make more information quickly accessible on a screen. This post is a practical walkthrough of implementing a tabbed panel solution in Rails using a GPL’ed plugin. Part 1 will demonstrate a basic tabbed panel implementation.

I’m developing a custom project management application in Rails for one of my clients. They need to organize large number of tasks for each project, and those tasks are divided into three categories: Action Items, Problem Reports, and Work Orders. The three types are pretty parallel in structure, and in fact I implemented them all in one table called Tasks with Single Table Inheritance.

Here’s an example of tables of the three types of tasks.

The goal is to collapse these three tables into a single tabbed panel, with tabs for Action Items, Work Orders, and Problems reports to quickly switch between the three tables.

Why I’m not using AJAX for this

Many Rails developers would implement this by drawing the tabs and using tab clicks to trigger a reload of the task listing, replacing the old listing with the right kind of tasks using AJAX and link_to_remote. This would work nicely, but would fail to satisfy one of my clients other desires: when a project’s page is printed, they would like all three categories of tasks to display on the page, each in its own table. If we actually render all three tables in a tabbed panel, then we can turn all three on with CSS in a stylesheet set to media=”print”, and solve both problems without having to generate a separate page!

The railstabbedpanel plugin

Several plugins exist that can do tabbed panels in Rails. After glancing at a couple, I picked railstabbedpanel, which is simple and looked like it would suit my needs nicely. The documentation is a bit sparse, but the “simple example” was enough to put together the basic task I needed. Download the archive and drop it in as vendor/plugins/railstabbedpanel.

As it turns out, this plugin needs a little hackery to implement a more advanced feature I want, but we’ll get to that in Part 2.

On to the implementation!

I start by loading the tasks I want to display into three instance variables:

in app/controllers/project.rb:

  def show
      @project = Project.find(params[:id])
      @action_items=@project.action_items
      @work_orders=@project.work_orders
      @problem_reports=@project.problem_reports
  end

To render a set of panels with railstabbedpanel, you pass a block to the tabbed_panel function, which creates a “tab context” object for you to use. The tab context object (which I called “tabctx” as per the example docs for railstabbedpanel) has a function, panel, which itself takes a block - the content you want inside the panel, and a parameter, which is the title you want to appear on your tab. There are plenty of other options you can pass, but we won’t need any of them in part one.

Since I’m going to be making three panels, I’ll put that in a partial called _task_panel, to which I’ll pass the tab context object and the list of tasks to display.

in app/views/project/show.html.erb

  <% tabbed_panel do |tabctx| %>
    <%= render :partial => "task_panel", :locals => {
      :tabctx => tabctx,
      :tasks => @action_items,
      :title => "Action Items"} %>    
    <%= render :partial => "task_panel", :locals => {
      :tabctx => tabctx,
      :tasks => @work_orders,
      :title => "Work Orders"} %>    
    <%= render :partial => "task_panel", :locals => {
      :tabctx => tabctx,
      :tasks => @problem_reports,
      :title => "Problem Reports"} %>    
  <% end %>

in app/views/project/_task_panel.html.erb:

  <% tabctx.panel(title) do %>
    <table class='listing'>
      <%= render :partial => "task_header_row" %>
      <%= render :partial => "task", :collection => tasks %>
    </table>
  <% end %>    

The two partials above render a fairly simple array of s (_task_header_row.html.erb) or s (_task.html.erb) to fill out the table of information about the task: creation date, due date, responsible party, summary, etc. I’ll spare you the code for those; they’re pretty straightforward.

We also need a little help from CSS to get the display right, as indicated by the documentation for railstabbedpanel:

in public/stylesheets/application.css:

.panel_selected {
    display: block;
}
.panel_unselected {
    display: none;
}
.tab_selected {
    background-color: gray;
}
.tab_unselected {
    background-color: white;
}

This works — sort of — we get our tab names, and clicking on them will activate the appropriate panels. But it certainly doesn’t look too good@ With just the suggested formatting, railstabbedpanel outputs our tabs as a stack of list items:

Obviously we’d like to improve on this. Let’s take a look at the HTML generated by the railstabbedpanel plugin:

<div class='tabbed_panel' id='tabbed_panel_124'>
<ul class='tab_container' id='tabbed_panel_124_tabs'>
  <li class='panel_tab tab_selected ' id='tabbed_panel_124_126_tab'>
    <a href="#" onclick="tabbed_panel_124_125('tabbed_panel_124_126'); 
                    return false;">Action Items</a>
  </li>
  <li class='panel_tab tab_unselected ' id='tabbed_panel_124_127_tab'>
    <a href="#" onclick="tabbed_panel_124_125('tabbed_panel_124_127'); 
                    return false;">Work Orders</a>
  </li>
  <li class='panel_tab tab_unselected ' id='tabbed_panel_124_128_tab'>
    <a href="#" onclick="tabbed_panel_124_125('tabbed_panel_124_128'); 
                    return false;">Problem Reports</a>
  </li>
</ul>
<ul class='panel_panels' id='tabbed_panel_124_panels'>
<li class='panel_panel panel_selected ' id='tabbed_panel_124_126_panel'> 
    <table class='listing' id='action_items'>  
    ... etc ....

So what we have is a ul with the class tab_container, with one li for each tab, each of which has class panel_tab and an additional class for either selected or unselected. The panels themselves are in a second ul of class panel_panels, with list items of class panel_panel. That’s fairly clean semantic markup, and in fact this is one of the reasons I chose railstabbedpanel in the first place.

So, let’s add a couple of basic rules to li.panel_tab to float the tabs next to each other, and some extra space at the top of ul.panel_panels. These few extra rules get us much closer to what we want:

li.panel_tab {
    list-style: none;
    float: left;
    padding: 1em;
    margin-right: .5em; 
}
ul.panel_panels {
    clear: both;
}

The result looks like this, which is a nice little functional tabbed panel:

We could play this game with CSS all day trying to get the tab styling just right and testing it in all the relevant web browsers. But as with everything else, some other developer has already struggled through this before, and with Google I dug up this lovely little reference at Adobe. I applied those styles to the containers determined previously by inspecting the HTML generated by the tabbed panel plugin, and tweaked the colors and fonts a bit to match my client’s application. The resulting CSS looks like this:

/*------------ TABBED PANELS -------------------*/
/* many styles from http://labs.adobe.com/technologies/spry/articles/tabbed_panel/ */
.tabbed_panel {   
    padding: 0px;
    clear: both;
    width: 100%; /* IE Hack (hasLayout Bug)*/    
}
.panel_panel {
    padding: 0.5em 0;
}
.panel_selected {
    display: block;
}
.panel_unselected {
    display: none;
}
.tab_container {
    margin: 0px;
    padding: 0px;
}
ul.panel_panels {
    clear: both;
    border-left: solid 1px #CCC;
    border-bottom: solid 1px #CCC;
    border-top: solid 1px #999;
    border-right: solid 1px #999;
    background-color: white;
    padding: 10px;
}
li.panel_tab {
    position: relative;
    top: 1px;
    float: left;
    padding: 4px 10px;
    margin: 0px 3px 0px 0px;
    font: bold 1.2em sans-serif;
    list-style: none;
    border-left: solid 1px #CCC;
    border-bottom: solid 1px #999;
    border-top: solid 1px #999;
    border-right: solid 1px #999;
    -moz-user-select: none;
    -khtml-user-select: none;
    cursor: pointer;        
}
li.tab_selected {
    background-color: white;
    border-bottom: 1px solid white;
}
li.tab_unselected {
    background-color: #aac;
}
li.tab_unselected:hover {
    background-color: #A77;
}
li.panel_tab a {
    color: black;
    text-decoration: none;
}
li.panel_tab a:hover {
    color: #009;    
}

Which results in this much improved tabbed panel:

Testing It

As I mentioned in a previous post, I think it’s important for tutorials to explain not just how to write the code, but how to test it as well, so let’s take a look at some tests that make this happen.

To start with, I need to add one small bit to the HTML to make it easily testable. If the tables that contain the task information each have an id specifying which group of tasks they contain, it will be easier to target them with assert_tag. So, I added such an ID to each table, using the name of the title, converted to lowercase with underscores for spaces, making them ‘action_items’ and so forth:

In app/views/project/_task_panel.html.erb

 <table class='listing' id='<%= title.gsub(/ /,'_').downcase %>'>   

Now in the test case for the show action on a project, I created a method that can look for a table whose id matches such a title. Inside the same function, I look for rows with an id constructed to match that task’s ID number (this is generated by the _task.html.erb partial, which I didn’t show above). And I look for a cell in that row whose text matches the “summary” field of the task. That should be sufficient to determine that each task of the appropriate type has a row in that table, though you can of course test for plenty more tags and attributes if you like.

In test/functional/project_controller_test.rb:

  private
  # Test that the table containing rows for all registered tasks are correctly
  # rendered.  The task type should be sent in underscore format, like 
  # "action_item" or "work_order"
  def test_tasks_table(title,tasks)     
    id = title.gsub(/ /,'_').downcase
    table_tag = {:tag=>'table',  :attributes=>{:id=>id}}
    assert_tag table_tag
    
    #li containing the appropriate title for the tab
    assert_tag( { :tag => 'li', 
        :attributes => { :class => /panel_tab/ },
        :content => /#{Regexp.escape(title)}/ } )
    
    # check for all the task rows
    tasks.each do |task|
      row_tag = { :tag => 'tr', :ancestor => table_tag, 
        :attributes => { :id => "task_row_#{task.id}" } }
      assert_tag row_tag
      assert_tag( { :tag => 'td', :parent => row_tag, 
          :content => /#{task.summary}/ } ) 
      #...test any other important aspects of the tasks' TR row here...      
    end
  end

And then I simply call this for all three task types in the method that tests the “show” action for a project. In this case, @proj_one contains an instance of the project model pulled from fixtures:

In test/functional/project_controller_test.rb:

  def test_show
    get :show, :id => @proj_one.id

    assert_response :success
    assert_template 'show'
    
    #assert the project object
    assert_not_nil assigns(:project)
    assert assigns(:project).valid?    

   #confirm that tasks row summaries are listed correctly
    test_tasks_table("Action Items",@proj_one.action_items)
    test_tasks_table("Work Orders",@proj_one.work_orders)
    test_tasks_table("Problem Reports",@proj_one.problem_reports)
  end

Okay, that works great, now we’ve confirmed that the tables are output correctly. Let’s move on and test the tabs themselves. As we saw above when examining the HTML, the tabs are a set of li.panel_tabs inside a ul.tab_container. The whole thing is contained inside a div.tabbed_panel. We want to confirm that each tab contains text matching the appropriate title. So, one quick helper method an three lines added to our test_show method will test these for us:

Added to test/functional/project_controller_test.rb

   def test_show
      ... 
    #confirm that the three tabs exist
    assert_panel_tab "Action Items"
    assert_panel_tab "Work Orders"
    assert_panel_tab "Problem Reports"
  end
  
  def assert_panel_tab title 
    ancestor = {:tag => 'div', :attributes => {:class => "tabbed_panel" }}
    parent = {:tag => 'ul', :attributes => {:class => "tab_container" }}
    assert_tag({:tag => 'li', 
        :attributes => {:class => /panel_tab/},
        :parent => parent,
        :ancestor => ancestor,
        :content => /#{title}/  
        }) 
  end
  

Note that I’m not testing :class => "panel_tab" for the tabs, but using match instead of a string. This is because the actual full class is going to be either class="panel_tab selected" or class="panel_tab unselected", and we want to match the tag in either case.

What’s next?

That’s it for this installment. In Part 2, I’ll explain how to endow the tabs with memory, so that if you click on, say, Problem Reports, and then leave the page, that same tab will still be in front when you come back. In Part 3, I’ll look at some more advanced styling, including making all three tables show nicely on the printout.