Fixing problems with sphinx search

I’ve been working a lot this week with sphinx and ultrasphinx on a project that’s a fork of Insoshi.    Insoshi is in the process of switching search from ferret to sphinx, and sphinx has been integrated into the Insoshi edge branch.

I’ve had dozens of problems, in fact it’s fair to say I’ve spent upwards of 15 hours just debugging ultrasphinx and getting my tests to pass.   There were several problems; here are the main three and how I fixed each one.

This should be useful to anyone upgrading Insoshi to the sphinx version, or to anyone else trying to get ultrasphinx working in their Rails project. I definitely don’t recommend starting with this post if you’re just starting out with sphinx. Instead, go read this much better introductory tutorial from the guys over at Insoshi. Then if you have problems, come back here and you may find solutions.

Getting search tests (or specs) to pass with sphinx

This one is pretty simple, in retrospect, but it can be frustrating and opaque if you are used to ferret.  Unlike ferret, sphinx (at least via ultrasphinx) runs only via a daemon.   Where acts_as_ferret uses a daemon only for the production environment and just accesses the index files directly in test or development, ultrasphinx can only get to the indexes through the daemon.

So, to run your tests, you just build up the indexes for test and run them.  In this case, I’m running the specs for Insoshi’s searches controller:

From the command line in $RAILS_ROOT:

rake db:test:prepare
rake ultrasphinx:configure RAILS_ENV=test
rake ultrasphinx:index RAILS_ENV=test
rake ultrasphinx:daemon:start RAILS_ENV=test
script/spec spec/controllers/searches_controller_spec.rb

The problem, of course, is that it doesn’t work!   The reason is that db:test:prepare creates the structure of your database, but doesn’t load any of your fixtures as data: the test db is empty..  So when you run the index command, an empty index is built.   You can see this from the output of that first index command, which will look something like this:

collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.078 sec, 0.00 bytes/sec, 0.00 docs/sec

Ultrasphinx has built an empty index.

The solution

The solution, believe it or not, is to run the tests, let them fail, re-index, and run the tests again (Many thanks to Long Nguyen at Insoshi for helping me figure this one out):

rake db:test:prepare
rake ultrasphinx:configure RAILS_ENV=test
rake ultrasphinx:index RAILS_ENV=test
rake ultrasphinx:daemon:start RAILS_ENV=test
script/spec spec/controllers/searches_controller_spec.rb   #FAIL!!
rake ultrasphinx:index RAILS_ENV=test
script/spec spec/controllers/searches_controller_spec.rb   #PASS!!

The first attempt to run the specs loads the fixtures, and leaves them in the database, thus letting the subsequent index command build an actual index.

Running sphinx for both test and development environments at the same time

The next big challenge was enabling behavior-driven development. I like to work with autotest and growl running constantly in the background. But this was tough to do with sphinx, because the daemon needed to be stopped and re-started, and the index re-created for each environment, alternately running all of the above commands either with or without RAILS_ENV=test.

The solution is to set up your ultrasphinx base configuration to completely separate both the test and development indexes and to let the daemons for the two environments listen on different ports. I had tried something like this and come close, but not quite, when Long at Insoshi again bailed me out. You need to change the port (in two places), and the paths of the logs, pidfile, and index directories so that test and development daemons are using entirely separate resources. Here’s a diff of my test.conf and default.conf:

33c33
<   port = 3312
---
>   port = 3322
35c35
<   log = log/searchd.log
---
>   log = log/searchd_test.log
39c39
<   pid_file = log/searchd.pid
---
>   pid_file = log/searchd_test.pid
50c50
<   server_port = 3312
---
>   server_port = 3322
57c57
<   sql_range_step = 5000   
---
>   sql_range_step = 999999999   
64c64
<   path = sphinx
---
>   path = sphinx_test

The sql_range_step is related to the next issue, which is that sphinx does not play well with foxy fixtures. Anyway, make the above changes and you should be able to run test and development sphinx daemons at the same time:

rake db:test:prepare
rake ultrasphinx:configure
rake ultrasphinx:configure RAILS_ENV=test
rake ultrasphinx:index
rake ultrasphinx:index RAILS_ENV=test
rake ultrasphinx:daemon:start
rake ultrasphinx:daemon:start RAILS_ENV=test

If it worked, you should see separate indexes in $RAILS_ROOT/sphinx and $RAILS_ROOT/sphinx_test, and two daemons running, which you can confirm with ps waux | grep searchd:

evan      1339   0.0  0.0    78100    292 s000  S     5:37PM   0:00.52 searchd --config <YOUR_RAILS_ROOT>/config/ultrasphinx/test.conf
evan      1326   0.0  0.0    78100    292 s000  S     5:36PM   0:00.68 searchd --config <YOUR_RAILS_ROOT>/config/ultrasphinx/development.conf

Getting sphinx to play well with foxy fixtures

The next problem I discovered was that on some machines, but not others, running my search specs would result in these weird errors:

1)
ActiveRecord::RecordNotFound in 'SearchesController Person searches should search by name'
Couldn't find Person with ID=328556765
/var/www/domains/unithrive/vendor/plugins/ultrasphinx/lib/ultrasphinx/search/internals.rb:308:in `reify_results'
/var/www/domains/unithrive/vendor/plugins/ultrasphinx/lib/ultrasphinx/search/internals.rb:286:in `each'
/var/www/domains/unithrive/vendor/plugins/ultrasphinx/lib/ultrasphinx/search/internals.rb:286:in `reify_results'
/var/www/domains/unithrive/vendor/plugins/ultrasphinx/lib/ultrasphinx/search.rb:362:in `run'
/var/www/domains/unithrive/vendor/plugins/ultrasphinx/lib/ultrasphinx/search/internals.rb:352:in `perform_action_with_retries'
/var/www/domains/unithrive/vendor/plugins/ultrasphinx/lib/ultrasphinx/search.rb:342:in `run'
/var/www/domains/unithrive/app/controllers/searches_controller.rb:38:in `index'
./spec/controllers/searches_controller_spec.rb:51:
script/spec:4:

When I poked into this “Couldn’t find Person with ID=328556765″ error, it seemed like sphinx was almost working. The index was set up, and the search was finding someone in the index during the test. Ultrasphinx was passing back the id 328556765, which didn’t exist in the database. So why would Sphinx “find” a record in its index but then pass back an ID for a database record that didn’t exist?

And furthermore, why would it work on one machine, but not on another?

The brainstorm came when I checked what the actual database IDs were for this particular record, with Person.find_by_name(”fixtures’ name”).id. On machines where it worked, the id was a huge number (is it generally is with foxy fixtures), but on machines where it didn’t work, the id was an even huger number.

Sphinx tries to make sure that all items that get indexed have a different index in sphinx, and it does this by multiplying all of your id’s by N, where N is the number of models getting indexed, and adding an offset of 0 for the first model, 1 for the second, etc. This guarantees that every record from every table will have a unique id. In the case of this application, all of my Person records were getting indexed by sphinx as (Person#id * 4 + 2).

Danger, Will Robinson: 32-bit int rollover!

The problem is that foxy fixtures generate their own ids from a hash of the fixture label, and those ids can be anywhere in the 32-bit unsigned integer space. But Sphinx also stores ids as 32-bit unsigned integers. This means if you happen to get a large fixture id, and then sphinx multiplies it by 4 (or whatever; it could be higher if you have more indexed models), your id will rollover and come out as (id * N + n) % (2^32). Sphinx will store that result, and then when it finds the record in a search, it will try to recreate the original id by subtracting n and dividing by N … giving you the wrong id. Your test will fail to find the record.

Incidentally, this problem with foxy fixtures is why your test.base file needs the line sql_range_step = 999999999. Sphinx builds indexes by searching a few ids at a time. But the ids generated by foxy fixtures are so big that if sphinx only collects them in ranges of 5000 at a time, it will take forever to find them all.

After some googling, I found that these issues are discussed in a thread over at RubyForge.

The solution

I’m working on a plugin that monkeypatches foxy fixtures to create sequential, low-numbered IDs. In the meantime, you can just compile sphinx to support 64-bit ids, which should give you plenty of headroom to handle foxy fixture ids multiplied by N in sphinx^*:

In your sphinx source directory:

configure --enable-id64
make
sudo make install

That should do it. Let me know in comments if any of this information helped you.

^*At least until you start approaching 2^32 models in your application, that is.

Changing the origin of your git repository

This is pretty basic and there’s plenty of documentation on the web, but this morning I found it difficult to locate with a google search. Maybe I just didn’t know the right terms. (searching “git repository change origin” seems obvious to me, but the results were useless.) Hopefully this post will help people find the documentation they are looking for if they are searching the same way I was.

Anyway, suppose you have a local git repository that you cloned from some remote repository at, say git://user@dev.foo.com/git/repos/path, and you’ve been happily pulling, pushing and rebaseing to and from that remote repository. But now you or your team leader have moved that remote repository to a new location, and you’d like to point your repository to the new URL without having to do a full checkout.

What you need to do is change your ‘origin’ setting. Unfortunately, there’s no trivially easy way to do this from the command line in current versions of git. Instead, you edit .git/config in your project root, which may look something like this:

...
[remote "origin"]
        url = git://user@dev.foo.com/git/repos/path
        fetch = +refs/heads/*:refs/remotes/origin/*
...

or if your ‘remote’ is actually local:

...
[remote "origin"]
        url = /path/to/repos/on/this/machine
        fetch = +refs/heads/*:refs/remotes/origin/*
...

All you need to do is edit that file with your favorite editor and change the url = setting to your new location. Assuming the new repository is correctly set up and you have your URL right, you’ll be happily pushing and pulling to and from your new remote location.

Rails 2.1 gotcha: don’t name an ActiveRecord field ‘changes’ !

In one of my Rails projects, I maintain a user-visible log of updates to the database by recording entries into a table that looks like this (from schema.rb):

create_table "log_entries", :force => true do |t|
    t.integer  "user_id",    :limit => 11
    t.datetime "created_at"
    t.datetime "updated_at"
    t.string   "item_type"
    t.integer  "source_id",  :limit => 11
    t.string   "table"
    t.string   "action"
    t.text     "changes"
  end

Whenever a record is created, updated, or deleted, I create and save an instance of LogEntry, containing for example {:table => 'task', :action => 'update'} and in the ‘changes’ column I save a serialized hash showing which attributes of the task object changed before and after save. In addition, I save which logged-in user made these changes, and when.

This is convenient, it gives my client a log that’s much more user-accessible and allows them to easily back-trace who did what to the database and when, which is important for their process and certain certifications.

When I upgraded the project to Rails 2.1 a couple of weeks ago, most of the code still worked fine, but the hashes showing the object changes were no longer showing up in views of the log. The culprit turned out to be Rails 2.1’s new dirty feature. Why? Because it adds the method ‘changes’ to ActiveRecord::Base. This is a great new feature that lets you know what has changed to an ActiveRecord object since you loaded it from the database, with all kinds of benefits like doing more selective updates, or not writing to the database at all if no changes have occurred, thus reducing system load if you call save! a lot.

Unfortunately, the new method ‘changes’ was obscuring my attribute ‘changes’ in some circumstances, and at the very least confusing the heck out of me, the programmer.

The solution, of course, is not to name your fields anything that corresponds to any Ruby core method or any method of ActiveRecord::Base. I fixed my problem with a migration to rename the column ‘details’:

def self.up
  rename_column  :log_entries, :changes, :details
end

def self.down
  rename_column  :log_entries, :details, :changes
end  

Getting <select> options in the right order

Sometimes you may want to generate a selector with a fixed set of options.   In one recent task of mine, we needed a selector for an integer 1 through 10, but the client wanted them labeled also with text to identify how the numbers mapped to the words “High” “Medium” and “Low”. (We were selecting the priority level of a task in a project management application). Essentially, I want to generate this HTML output:

<select id="task_priority" name="task[priority]"> 
    <option value="1">1 - High</option> 
    <option value="2">2 - High</option> 
    <option value="3">3 - High</option> 
    <option value="4">4 - Med</option> 
    <option selected="selected" value="5">5 - Med</option> 
    <option value="6">6 - Med</option> 
    <option value="7">7 - Med</option> 
    <option value="8">8 - Low</option> 
    <option value="9">9 - Low</option> 
    <option value="10">10 - Low</option>
</select>

These are fixed name/value pairs, so it made sense to me to store them as a constant hash in my Task model:

Hash in Task.rb

PRIORITY_OPTIONS = { "1 - High" => "1", "2 - High" => "2", "3 - High" => "3",
"4 - Med" => "4", "5 - Med" => "5", "6 - Med" => "6", "7 - Med" => "7",
"8 - Low" => "8", "9 - Low" => "9", "10 - Low" => "10"  }

However, I was rather dismayed to discover that this didn’t produce the results I wanted, because Ruby hashes (in Ruby 1.8.6) do not preserve order. This is what came out:

That’s not what I wanted! I knew that select will also take arrays, but of course I needed separate name/value pairs, which I can’t get with just straight arrays. I spent a while playing around with OrderedHash, which exists in Rails but is essentially undocumented and, as it turns out, does not support any useful functions of Hash like merge! and insert! that might make it easy to construct my list of options.

The Fix: Array of Arrays, or Array of Hashes

The documentation is not entirely clear on this, but if you send an array of 2-element arrays to select, rails will use the two elements of each inner array as if they were key and value pairs, and because the entire structure is an array it will preserve order. So, to get the results I wanted, I just need to change my constant to this:

PRIORITY_OPTIONS = [ ["1 - High", 1], ["2 - High", 2], ["3 - High", 3],
["4 - Med", 4], ["5 - Med", 5], ["6 - Med", 6], ["7 - Med", 7],
["8 - Low", 8], ["9 - Low", 9], ["10 - Low", 10] ]

And I get the result I was looking for :

As it turns out, the way ActionView processes the options is fairly general: if you pass it any enumerable object, it will iterate that object, and for each element will check to see if that element supports the methods :first and :last (and isn’t a string). If so, it will generate an option with the text set to element.first and the value set to element.last. If it was a string, or didn’t support first and last, both the text and value of the option are set to the element itself.

Testing it

Here’s a handy function I use for testing it the presence of a selector. You pass it the name of your selector, the name of hash or array of options (in the any format supported by select), and optionally the value of an item that should be pre-selected, and it will assert the existence of each of those things. If you need to handle selectors with multiple selections, you can just wrap the last assertion in a loop.

Drop this in test/test_helper.rb

  # Assert existence of form select input
  # the <option> tags can be passed either as 
  # a hash like this: {'foo'=>'bar', 'baz'=>'qux'} 
  # or an array of arrays like this: [ ['foo', 'bar'], ['baz', 'qux'] ]  
  # either will be asserted to match:
  #         <option value='bar'>foo</option>
  #         <option value='qux'>baz</option>
  # or as an array like this:
  #   [ 'foo', 'bar' ] will be asserted to match
  #         <option value='foo'>foo</option>
  #         <option value='bar'>bar</option>
  def assert_select_input(name, option_values, options={})
    attributes = { :name => name }

    # first assert that the select tag exists
    selector = { :tag => "select", :attributes => attributes }
    assert_tag selector   
        
    option_values.each do | opt |
      if !opt.is_a?(String) and opt.respond_to?(:first) and opt.respond_to?(:last)
        assert_tag({:tag => "option", :attributes => { :value => opt.last },      
          :parent => selector, :content => opt.first })
      else
        assert_tag({:tag => "option", :attributes => { :value => opt },      
          :parent => selector, :content => opt })               
      end
    end        
    
    # check for the pre-selected option, if any
    if options[:selected] 
      assert_tag :tag => "option", :attributes => {:selected => 'selected', 
        :value => options[:selected] }
    end    
  end  

As an example of how to use it, here’s my method for testing the task priority selector pictured above:

  # asserts a <SELECT> field, targeting task[priority], containing ten
  # appropriately formatted <OPTIONS>, and with selected_num (if any) the
  # pre-selected option
  def assert_priority_select(selected_num = nil)
    options = selected_num ? { :selected => selected_num } : Hash.new
    assert_select_input "task[priority]", Task::PRIORITY_OPTIONS, options  
  end

Where the array specified in the Task model looks like this:

  PRIORITY_OPTIONS = [ ["1 - High", 1], ["2 - High", 2], ["3 - High", 3], 
    ["4 - Med", 4], ["5 - Med", 5], ["6 - Med", 6], ["7 - Med", 7],
    ["8 - Low", 8], ["9 - Low", 9], ["10 - Low", 10] ]

Using redirect_to :back to improve user interfaces

It’s a scourge that inflicts nearly every website: the dreaded empty page stating “Your search returned no results.” Or even worse, the search results page with a header “Search results” and zero content, the footer immediately following the header. This is not only useless, but also confusing to the user, because it takes the reader several seconds to even realize that the search returned no results.

For example, I just got this one on the otherwise lovely workingwithrails.com when I searched for RoR developers in Pasadena, CA:

This is lame, lame, lame. A search results page with no results is guaranteed not to be useful to the user! From a user interface perspective, it’s akin to redirecting a user to a login page after they try to access some resource, and then sending them back to the front page or a super-lame “thank you for logging in” page afterwards.    The user should always see a useful page, and should never have to waste clicks getting back to what they wanted.

It’s not only bad UI, it’s also entirely unnecessary, because in Rails you can fix this for good with two lines of code.

Avoid wasted pages and annoyed users with redirect_to: back

If your user’s search doesn’t find anything, don’t make them waste their time at a whole page that tells them only that.  Instead just send them straight back to the search page.  Better yet, redirect them back to wherever they came from, since they might well be able to invoke search from any number of different contexts in your application. You don’t need a whole page to tell them the search didn’t find anything - a flash notice is more than sufficient:

Redirecting back with a flash after an empty search:

def search
  @results = YourModel.execute_some_search(params[:some_parameter]);
  if @results.size == 0
    flash[:notice] = "No items found matching '#{params[:some_parameter]}'"
    redirect_to :back
  end
end

Testing redirect_to :back

Testing redirect_to :back isn’t completely trivial, because of course there’s no previous page when you render a page in a test context. So, when you try to run a controller that redirects back, you’ll see an error. The easiest thing to do is to artificially set a previous page. I have these two methods in test/test_helper.rb:

def fake_referer
    @request.env['HTTP_REFERER'] = 'http://previous_page'
  end
  
  def assert_redirected_back
    assert_redirected_to 'http://previous_page'
  end

Then anytime I am working on a controller that involves back redirects, I drop fake_referrer in my setup method and then use assert_redirected_back. For example, in a search controller test (this is from a controller that has an action called ‘contents’ for searching items by text content - the search itself uses ferret):

def setup
    super
    fake_referer
  end

  def test_contents_search_fails
    get :contents, :contents => "some&junk*that(will@never)match$anything"
    assert_not_nil flash[:notice]
    assert_redirected_back
  end

Firefox Download Day (= display: inline-block day!)

It’s Firefox 3 Download Day!   Go get it, please.

I am excited about this, but perhaps not for the same reason most other people were. I am excited because Firefox 3 is the first version to implement the CSS rule “display: inline-block”. Inline-block allows you to generate chunks of content of rectangular shape that will flow inline in your content (like text, images, or tables) but whose contents act like a block, and whose borders and margins are respected in the flow.

Let’s make it clear what I mean.

The beauty and versatility of display: inline-block

Setting a CSS element’s display: property to inline-block tells the browser to treat the element as a rectangular block, with internal flow just like any other block, and to respect the padding, border, and margins, but to flow that rectangle with other inline content - exactly the way the browser would flow an image. In addition, you can specify the width and height of inline-block items. This is a crazy useful concept from a layout perspective: it would allow you to make buttons out of text that lay out horizontally, or to layer images on top of each other inside a small span or div.

Here’s an example of a series of list items set with border, padding, and a width and height. The code first:

<style>
      ul {
        width: 400px;
        border: 2px solid black;
        background: #aaf;
      }
      li {
        border: 2px dotted blue;
        padding: 1em;      
        width: 100px;
        height: 35px;
      }
      ul.inline li {
        display: inline;
      }
      ul.inline-block li {
        display: inline-block;
      }
    </style>
<h2>Padded words wrapped with display: inline;</h2>
  <p>
    <ul class="inline">
    <li>Lorem</li>
    <li>ipsum</li>
    <li>dolor</li>
    <li>sit</li>    
    <li>amet</li>
    <li>consectetur</li>
    <li>adipisicing</li>
    <li>elit</li>    
    <li>sed</li>
    <lido</li>
    <li>eiusmod</li>
    </ul>
  </p>

  <h2>Padded words wrapped with display: inline-block;</h2>
  <p>
    <ul class="inline-block">
    <li>Lorem</li>
    <li>ipsum</li>
    <li>dolor</li>
    <li>sit</li>    
    <li>amet</li>
    <li>consectetur</li>
    <li>adipisicing</li>
    <li>elit</li>    
    <li>sed</li>
    <li>do</li>
    <li>eiusmod</li>
    </ul>
  </p>

Here’s how Safari renders the above code (correctly):

And here’s how Firefox 2 renders it (incorrectly):

Horrific - FF2 simply ignores inline-block, causing the list elements to render in their default display mode, block. What’s galling about this is that for several years Firefox has been the only browser that gets it wrong! That’s right, even Internet Explorer 6 gets this right … at least on some elements, specifically those that are natively display: inline, like spans and anchors.

The Firefox bug report was filed way back in the days of Firefox 1 nine years ago. Yes, you read that right. It sat there, unfixed, while every other browser in the world implemented it.

Anyway, today web developers everywhere can rejoice. Now, at last, the released version of every major browser supports display: inline block. So, with only another year or two waiting while FF2 users upgrade to FF3, designers will be able to start using this singularly useful CSS rule (at least in cases where IE supports it as well).

A parting example

A parting example of the kind of cool trick you can play with inline-block: text flowing inside a sized box that is itself flowing inside text.

Code for flowing text inside flowing text:

<div style="font-size: 250%; width: 600px;">
  Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor 
    <span style="width: 130px; height: 40px; font-size: 9px; display: inline-block;
      padding: 1em; border: 1px solid black; overflow: hidden;">
      Duis aute irure dolor in reprehenderit in voluptate velit esse cillum 
      dolore eu fugiat nulla pariatur.
    </span>  
  incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam.   
  </div>

How it renders in a compliant browser … including IE6!*

*Well, more or less. IE6 gets it reasonably close.

Another new design for LRDesign.com

Apparently I can’t stop playing, because today I built and uploaded a third design for the LRDesign main site: “Blue Art”. This one required the IE PNG fix from Twin Helix for IE 6.0 compatibility.

I think there are still a few annoying bugs in the stylesheet switcher on IE; once I clean those up I plan to write a post explaining how to easily set up this kind of stylesheet switcher in a Rails project.

New theme added to LRDesign.com

For quite a while I’ve wanted to decorate the main Logical Reality site with multiple, switchable themes, implementing each with pure CSS on top of semantic HTML markup, a la the CSS Zen Garden. Last night I finally got a start on that, implementing a simple stylesheet switcher and a grunge theme with textures from Urban Dirty.

To check it out, load the main site and click the “Change Skin” link in the upper left.

Installing Insoshi on a Dreamhost Account

With the Insoshi social networking platform rapidly gaining in popularity, I thought it might be useful to folks to know how to install it on the ever-popular Dreamhost shared account. If you need a Dreamhost account, please consider using the promo code “LRDESIGN” when you sign up. It will save you $50 on your first year of membership, and will help me with my site hosting expenses so I can keep this blog going.

So, there are still some possible drawbacks to this approach, but I was able to get a running install of Insoshi on my Dreamhost account with this sequence. You might want to read to the bottom of this post to learn about the difficulties with acts_as_ferret and Dreamhost before you commit to running your insoshi site on DH. Hopefully these problems will have a solution soon, and I’ll update this post if/when they do.

If you use this method and it works (or doesn’t!) please let me know in comments.

1) Set up a domain

Use the DH control panel to create a new fully hosted domain for your Insoshi site, for example yourdomain.com or insoshi.yourdomain.com. I will use “insoshi.yourdomain.com” through the rest of this post to indicate the domain that you want to use to run insoshi. I set up http://insoshi.lrdesign.com/ in the process of writing this post, but I can’t guarantee that it will stay up.

When you set up the domain:

• Make sure that fastcgi support is selected.
• Set “specify your web directory” to point to /home/username/insoshi.yourdomain.com/public/

2) Set up mysql databases

1. In the dreamhost panel, select “Goodies -> Manage MySQL”
2. Scroll down to “create a new mysql database”

I used these example settings:

• database name: insoshi
• use hostname: mysql.yourdomain.com (use “new hostname” to create this if you do not already have it)
• New user: insoshi
• New password:

If you want to run tests, you should create a second database called insoshi_test; you can leave the other settings the same.

3) Download the tarball of the current insoshi distribution:

cd
wget http://insoshi.com/home/tarball

You’ll get a tarball with a name like “insoshi-insoshi-e1fd8b8e440c9f3ab34161d4e87de78e956c1012.tar.gz”. Unzip the tarball and copy the contents to the directory you want the website to appear in:

tar xzf insoshi-insoshi-e1fd8b8e440c9f3ab34161d4e87de78e956c1012.tar.gz
cp -r insoshi-insoshi-e1fd8b8e440c9f3ab34161d4e87de78e956c1012/* insoshi.yourdomain.com/

4) Set up your database.yml file

cd ~/insoshi.yourdomain.com/config
cp database.example database.yml

Edit ~/insoshi.yourdomain.com/database.yml and make it look like the following, where <password> is the password you chose in the previous step:

development:
  adapter: mysql
  database: insoshi
  username: insoshi
  password: <password>
  host: mysql.lrdesign.com
  port: 3306

# Warning: The database defined as 'test' will be erased and
# re-generated from your development database when you run 'rake'.
# Do not set this db to the same as development or production.
test:
  adapter: mysql
  database: insoshi_test
  username: insoshi
  password: <password>
  host: mysql.lrdesign.com
  port: 3306

production:
  adapter: mysql
  database: insoshi
  username: insoshi
  password: <password>
  host: mysql.lrdesign.com
  port: 3306

5) Run the insoshi install script.

This will migrate the database and do some insoshi-specific setup. It’s also an excellent way to check that you’ve configured your database.yml correctly.

cd ~/insoshi.yourdomain.com
rake install

If it works, you should see a bunch of migrations (22 as of the current version of insoshi). If not, go back and figure out what’s wrong with your database.yml file.

6) Get rails working

These instructions are an adaptation of the instructions at the Dreamhost Wiki page about Rails.

Generate a dummy rails app and copy the dispatch scripts to your insoshi install:

cd ~
rails dummy
cp dummy/public/dispatch.* insoshi.yourdomain.com/public/

Edit ~/insoshi.yourdomain.com/public/.htaccess file to enable fastCGI. Change the line with dispatch.cgi to look like this:

RewriteRule ^(.*)$ dispatch.fcgi [QSA,L]

change your permissions on the public and dispatch files:

cd ~/insoshi.yourdomai.com
chmod 755 public
chmod 755 public/dispatch.*  

I also had to make my log files writeable:

chmod a+w log/
chmod a+w log/*

7) Start the ferret server

I was not able to get insoshi to run in production mode on Dreamhost at first
because in production mode it needs the ferret server (text search) to be
running or it will refuse to load some of the models. (In test and development
mode, acts_as_ferret will access the ferret databases directly, so this problem only appears in production). Based on this post, I found I could get it working by running:

script/ferret_server start -e production

At this point, I was able to load Insoshi in my browser.

Unfortunately, there are still some problems

I am pretty confident that the approach to running ferret_server above will not be a long term solution, because Dreamhost kills any processes that you leave running for more than a few hours. So ferret_server will go down after a while, and with it your site, so this will probably only get your insoshi site up for a few hours before you have to restart ferret_server. This basically means there’s no good way to use the rails plugin acts_as_ferret on a DH shared account, and unfortunately Insoshi depends on AAF.

Possible Workarounds

You could try putting the startup command in a cron script, to restart ferret_server when Dreamhost kills it, for example:

0,15,30,45 * * * * cd ~/insoshi.yourdomain.com; script/ferret_server start -e production

would attempt to start the ferret server every fifteen minutes. But that might run afoul of Dreamhost server policies (does anyone know for sure?), and in any case your site would still be down in between the time DH killed the ferret_server process and your cron job started again.

You can also alter config/ferret_server.yml to have ferret treat production mode the same as development, directly accessing the ferret database and bypassing ferret_server entirely. However, if you get concurrent access with multiple users, you are very likely to get a corrupted ferret database with that approach.

Hopefully, some permanent solutions?

The Insoshi guys are working on replacing ferret with Sphinx, and that may be a permanent solution to this problem.

You also may want to consider lobbying Dreamhost to allow users run persistent processes like ferret_server. If you are a Dreamhost subscriber you can vote for this feature by following this link to Dreamhost’s Policies Suggestions and voting for “Be able to run simple, persistent scripts!”.

Getting mysql and sphinx to work on OS X 10.4 on a G5

After some significant problems using acts_as_ferret (for some unknown reason, it absolutely will not run on my deployment server; the models will load in test but not in development or production if they use acts_as_ferret), I decided to try sphinx on the recommendation of Michael Hartl. Unfortunately, I had some problems there too: sphinx would not compile on my development workstation, a Dual G5 mac running OS X 10.4.

I did manage to solve it, though, by switching from the package-installed version of MySQL to a compiled-from-source copy. In case anyone else runs into the problem, here’s a record of how I solved it.

Symptoms of the problem

Sphinx would not compile on my mac. The error seemed to be a complaint about the header libraries, certain MySQL symbols were not getting correctly defined:

[20:05:10] sphinx-0.9.8-rc2$ make
Making all in src
g++ -o indexer indexer.o libsphinx.a -L/usr/local/mysql/lib -lmysqlclient -lz
/usr/libexec/gcc/powerpc-apple-darwin8/4.0.1/ld: Undefined symbols:
_mysql_close
_mysql_errno
_mysql_error
_mysql_fetch_fields
_mysql_fetch_row
_mysql_free_result
_mysql_init
_mysql_num_fields
_mysql_query
_mysql_real_connect
_mysql_use_result
collect2: ld returned 1 exit status
make[1]: * [indexer] Error 1
make: * [all-recursive] Error 1

It also fail to build with MacPorts (port install sphinx), with similar errors. So, after some digging I decided to try swapping out MySQL, which I’d installed from the .dmg package installer, to a copy compiled from source, hoping that this would get the header libraries set up correctly. It worked, and in the process I upgraded from 5.0.45 to the current 5.0.51b. Here’s the sequence:

First, upgrade mysql to a hand-built one.

These instructions are straight from the HiveLogic instructions on building MySQL on OS X. All I’ve added is the backup and restore of databases plus a newer URLs for the most recent MySQL 5.0 release (as of May 16, 2008).

Back up and turn off the old server:

First, back up all your existing databases using mysqldump. I did it to a bzipped file in my home directory.

[11:39:03] evan$ mysqldump --all-databases -p | bzip2 -c > databasebackup.sql.bz2

This could take a while, depending on how many databases you have, and how large. About ten minutes, for me. Now, shut down the server and remove the symlink /usr/local/mysql:

[11:49:13] evan$ sudo mysqladmin shutdown
[11:51:56] bin$ cd /usr/local
[11:52:14] local$ sudo rm mysql

Now download and build the new mysql:

[11:47:10] evan$ cd ~/src
[11:47:23] src$ wget http://dev.mysql.com/get/Downloads/MySQL-5.0/mysql-5.0.51b.tar.gz/from/http://mysql.he.net/
[11:48:09] src$ tar xzf mysql-5.0.51b.tar.gz
[11:48:25] src$ cd mysql-5.0.51b

Configure and compile:

[11:49:10] mysql-5.0.51b$ CC=gcc CFLAGS="-O3 -fno-omit-frame-pointer" CXX=gcc 
CXXFLAGS="-O3 -fno-omit-frame-pointer -felide-constructors 
-fno-exceptions -fno-rtti" 
[11:49:32] mysql-5.0.51b$ ./configure --prefix=/usr/local/mysql 
--with-extra-charsets=complex --enable-thread-safe-client 
--enable-local-infile --enable-shared
[11:49:32] mysql-5.0.51b$ make; sudo make install;

This takes a while. I went to take a shower. Fortunately, for me, it worked the first time and required no hackery, which is a pleasant surprise when building OSS software. Now setup the initial directories and databases:

[12:10:17] mysql-5.0.51b$  cd /usr/local/mysql
[12:10:21] mysql-5.0.51b$ sudo ./bin/mysql_install_db --user=mysql
[12:10:28] mysql-5.0.51b$ sudo chown -R mysql ./var

And finally start the new server and import the backed-up databases:

cd /usr/local/mysql/bin
sudo ./mysqld_safe &
bzcat ~/databasebackup.sql.bz2 | mysql

After another ten-minute import, I’m back up and running with a fresh build of MySQL.

Installing Sphinx

After that, sphinx (0.9.8-rc2) compiled normally:

[13:20:51] src$ wget http://www.sphinxsearch.com/downloads/sphinx-0.9.8-rc2.tar.gz
[13:20:56] src$ tar xzf sphinx-0.9.8-rc2.tar.gz 
[13:21:09] src$ cd sphinx-0.9.8-rc2
[13:21:12] sphinx-0.9.8-rc2$ ./configure; make
[13:23:30] sphinx-0.9.8-rc2$ make install 
... success ...