LRBlog

Logical Reality Design: Web Design and Software Development

Convection: self-hosted secure file exchange in Rails

June 8, 2010

Introducing Convection, an open-source (MIT License) project of Logical Reality Design. Need to swap files with clients or collaborators, but don't want to (or can't) trust those files to Amazon or sendbigfiles.com? Want fine-grained control over which users can see which files? Try Convection.

Lots of file exchange services exist, for example SendBigFiles.com etc. However, all of these services are hosted on someone else's hardware, and most of them share files by transferring URLs -- usually via email -- without good access control or authorization schemes.

We built Convection because a client needed to transfer files with other companies, but they needed to host the system themselves because the contracts they hold with their own clients require them not to store data on services that they don't control. The specifications Convection was built around were:

  1. Hosted on our own server.
  2. Downloads require a login, and files cannot be shared by email.
  3. Users must log in to download files or see available files.
  4. User accounts can be grouped, groups can be managed.
  5. Files can be shared with an entire group.
  6. Files uploaded by users default to minimal permission - visible only to the uploader and to admins.
  7. All communications over SSL. (we made this optional)

Installing and hosting Convection

To run Convection, you will need a webserver capable of running a Ruby on Rails application, and a database. Setting such a thing up is beyond the scope of this post. If you have a Dreamhost account, you can set up a Rails-capable domain with a couple of clicks in their web panel. In addition to the server, you will need to set up a database (we have only tested MySQL, but Convection should work with any SQL database for which Rails/ActiveRecord has a supported adapter, including PostgreSQL and Oracle), and initialize the database with these two commands:

  > rake db:migrate RAILS_ENV=production
  > rake db:seed RAILS_ENV=production

This will generate the tables necessary for Convection to run, and create a pair of initial demo users "admin" and "user", both with password "foobar".

If you are setting up a server yourself, there are plenty of guides to deploying Rails on the web. Much of our own guide to deploying CruiseControl.rb can be used to set up any Rails application on Slicehost or any other Ubuntu Linux hosting provider.

Let me know if you're trying to deploy Convection and having trouble: if we know people are using it we may put effort into making it easier to deploy and install, and write a more thorough guide.

A few other links that may help you with deploying a Rails application, depending on your environment:

  1. Using Phusion Passenger to Deploy a Rails Application on Apache
  2. Deploying Rails Applications (book)

If you Google around you may find plenty of other links relevant to your particular environment.

Configuring Convection

If you log into your running Convection application as an administrator (initial user "admin", password "foobar"), an Admin Tools utility will appear in the right hand column. From here, you can access tools for creating users, and groups, and the general site configuration.

In general site config, you can set your site name and logo, set whether or not the site requires SSL access (Note: your server must already support SSL!) outgoing email and email notification preferences, add Google analytics, and an assortment of other site configuration operations that are mostly self-explanatory.

Upload progress bar: experimental feature.

If your site hosts large uploads that take a while to transfer, you can try our experimental tools to provide an upload progress bar to the user. This tool will only work if your site is served by Apache, and requires installing and configuring an optional module for Apache.

To enable this tool, follow the instructions in the README file and associated links, and turn on the progress bar setting in site preferences.

Helping us improve Convection

Convection is currently in version 1.1.4 and has been in production in two places (that we know of) for about five months as of June 8, 2010.

Please let us know if you are using Convection and enjoy it (or don't). Feel free to request features or alterations, but Convection is open source, so also please consider contributing if you have ideas!

Using link_to (or other helper methods) in a controller

May 6, 2009

This one was a big aggravator to me lately. I have one controller that needs to call link_to and url_for, which are normally helper methods you'd call from a view. However, in this case during certain modifications to a record, I actually need to append user-visible HTML links to a block of HTML stored in that object, or possibly another one.

Specifically, I needed to put annotations in the description of a work order object that said, for example "this work order was escalated from Problem Report 293. This was done in a create action that redirected at the end and never rendered a view, so I really did need to generate that link in the controller. And for consistency with the rest of the application, I wanted to generate the link with link_to(@task).

Now, ActionView::Helpers::UrlHelper is not loaded in a Rails controller, even if you've put helper :all in application.rb (application_controller.rb in newer versions). So, when I tried to use link_to in the controller, I got an error:

NoMethodError: undefined method `link_to' for #
/Users/evan/Development/Ruby/eclipticdb/app/helpers/tasks_helper.rb:64:in `task_link'
/Users/evan/Development/Ruby/eclipticdb/app/controllers/tasks_controller.rb:103:in `escalate'
... etc ...

The first fix - but with a problem

A year ago, I fixed this just by adding include ActionView::Helpers::UrlHelper at the top of that controller. This worked great ... for a while.

Lately, I've been rewriting this application into a RESTful style - it had previously been a controller/action style application. In the process, I started linking things with resource paths and polymorphic paths ... a lot of link_to @task and edit_polymorphic_path(@task) sorts of bits. And these started breaking. I began seeing this mysterious error:

Error:

You have a nil object when you didn't expect it!
The error occurred while evaluating nil.url_for

... some code here that calls a link_to ...

Trace of template inclusion: /tasks/_task_panel.html.erb, /tasks/_task_tabbed_panel.html.erb, /tasks/index.html.erb

RAILS_ROOT: /Users/evan/Development/Ruby/eclipticdb
Application Trace | Framework Trace | Full Trace

vendor/rails/actionpack/lib/action_view/helpers/url_helper.rb:71:in `send'
vendor/rails/actionpack/lib/action_view/helpers/url_helper.rb:71:in `url_for'

This one was a real bitch to debug, I have to say. The line in question that was failing in url_helper.rb said this: url = @controller.send(:url_for, options). Clearly, @controller was nil ... which was very bizarre, because I never interact with that instance variable anywhere.

I thrashed around trying to find the cause of this error for quite some time. Eventually I realized that the link_to method was only failing when called from a view in TasksController, and not from any other controller. And then I realized that TasksController was the one where, a year ago, I'd put include ActionView::Helpers::UrlHelper at the top. Somehow, including that helper in the controller was nullifying @controller when those helper method we called from within the view. I removed the include and my polymorphic and resource links all started working again.

Now back to the original problem!

Of course, that then left me back with the problem I'd had a year ago ... needing to use link_to from within the controller and having no way to do it. After a fair bit of googling around I found this post from Neeraj, which had an interesting approach -- but a commenter had suggested a much easier solution:

[sourcecode language='ror']self.class.helpers.link_to[/sourcecode]

I'm not certain where one would find this in the docs, but it does seem to have solved my problem for now. Onward and upward!

Bypassing mass assignment for update_attributes

March 14, 2009

I've been following this excellent post by M. Hartl and this post by E. Chapweske banishing mass assignment from one of my Rails applications due to launch soon.

I'm following Chapweske's approach of blocking mass assignment by default in all models, by putting this line in an initializer:

ActiveRecord::Base.send(:attr_accessible, nil)

This had the expected side effect of breaking several zillion tests, because tests frequently use things like Model.build() and Model.create!() to generate on-demand fixtures during testing. Hartl has a great bit of code that creates unsafe_build() and unsafe_create() methods in ActiveRecord. You can use these methods instead of build() and create() to function as expected in your tests.

This works great, except that I also use the mass-assignment method update_attributes! in my tests and specs frequently, particularly when I want to spec the effect a change on one model has on an associated models' methods. So, I expanded on Hartl's helper code a bit, to give myself the necessary methods. In case it helps anyone else:

/lib/initializers/unsafe_build_and_create.rb

class ActiveRecord::Base

# Build and create records unsafely, bypassing attr_accessible.
# These methods are especially useful in tests and in the console.

def self.unsafe_build(attrs)
record = new
record.unsafe_attributes = attrs
record
end

def self.unsafe_create(attrs)
record = unsafe_build(attrs)
record.save
record
end

def self.unsafe_create!(attrs)
unsafe_build(attrs).save!
end

def unsafe_update_attributes!(attrs)
self.unsafe_attributes = attrs
self.save!
end

def unsafe_update_attributes(attrs)
self.unsafe_attributes = attrs
self.save
end

def unsafe_attributes=(attrs)
attrs.each do |k, v|
send("#{k}=", v)
end
end
end

Fixing problems with sphinx search

July 24, 2008

I've been working a lot this week with sphinx and ultrasphinx on a project that's a fork of Insoshi.    Insoshi is in the process of switching search from ferret to sphinx, and sphinx has been integrated into the Insoshi edge branch.

I've had dozens of problems, in fact it's fair to say I've spent upwards of 15 hours just debugging ultrasphinx and getting my tests to pass.   There were several problems; here are the main three and how I fixed each one.

This should be useful to anyone upgrading Insoshi to the sphinx version, or to anyone else trying to get ultrasphinx working in their Rails project. I definitely don't recommend starting with this post if you're just starting out with sphinx. Instead, go read this much better introductory tutorial from the guys over at Insoshi. Then if you have problems, come back here and you may find solutions.

Getting search tests (or specs) to pass with sphinx

This one is pretty simple, in retrospect, but it can be frustrating and opaque if you are used to ferret.  Unlike ferret, sphinx (at least via ultrasphinx) runs only via a daemon.   Where acts_as_ferret uses a daemon only for the production environment and just accesses the index files directly in test or development, ultrasphinx can only get to the indexes through the daemon.

So, to run your tests, you just build up the indexes for test and run them.  In this case, I'm running the specs for Insoshi's searches controller:

From the command line in $RAILS_ROOT:

rake db:test:prepare
rake ultrasphinx:configure RAILS_ENV=test
rake ultrasphinx:index RAILS_ENV=test
rake ultrasphinx:daemon:start RAILS_ENV=test
script/spec spec/controllers/searches_controller_spec.rb

The problem, of course, is that it doesn't work!   The reason is that db:test:prepare creates the structure of your database, but doesn't load any of your fixtures as data: the test db is empty..  So when you run the index command, an empty index is built.   You can see this from the output of that first index command, which will look something like this:

collected 0 docs, 0.0 MB
total 0 docs, 0 bytes
total 0.078 sec, 0.00 bytes/sec, 0.00 docs/sec

Ultrasphinx has built an empty index.

The solution

The solution, believe it or not, is to run the tests, let them fail, re-index, and run the tests again (Many thanks to Long Nguyen at Insoshi for helping me figure this one out):

rake db:test:prepare
rake ultrasphinx:configure RAILS_ENV=test
rake ultrasphinx:index RAILS_ENV=test
rake ultrasphinx:daemon:start RAILS_ENV=test
script/spec spec/controllers/searches_controller_spec.rb #FAIL!!
rake ultrasphinx:index RAILS_ENV=test
script/spec spec/controllers/searches_controller_spec.rb #PASS!!

The first attempt to run the specs loads the fixtures, and leaves them in the database, thus letting the subsequent index command build an actual index.

Running sphinx for both test and development environments at the same time

The next big challenge was enabling behavior-driven development. I like to work with autotest and growl running constantly in the background. But this was tough to do with sphinx, because the daemon needed to be stopped and re-started, and the index re-created for each environment, alternately running all of the above commands either with or without RAILS_ENV=test.

The solution is to set up your ultrasphinx base configuration to completely separate both the test and development indexes and to let the daemons for the two environments listen on different ports. I had tried something like this and come close, but not quite, when Long at Insoshi again bailed me out. You need to change the port (in two places), and the paths of the logs, pidfile, and index directories so that test and development daemons are using entirely separate resources. Here's a diff of my test.conf and default.conf:

33c33
< port = 3312
---
> port = 3322
35c35
< log = log/searchd.log
---
> log = log/searchd_test.log
39c39
< pid_file = log/searchd.pid
---
> pid_file = log/searchd_test.pid
50c50
< server_port = 3312
---
> server_port = 3322
57c57
< sql_range_step = 5000
---
> sql_range_step = 999999999
64c64
< path = sphinx
---
> path = sphinx_test

The sql_range_step is related to the next issue, which is that sphinx does not play well with foxy fixtures. Anyway, make the above changes and you should be able to run test and development sphinx daemons at the same time:

rake db:test:prepare
rake ultrasphinx:configure
rake ultrasphinx:configure RAILS_ENV=test
rake ultrasphinx:index
rake ultrasphinx:index RAILS_ENV=test
rake ultrasphinx:daemon:start
rake ultrasphinx:daemon:start RAILS_ENV=test

If it worked, you should see separate indexes in $RAILS_ROOT/sphinx and $RAILS_ROOT/sphinx_test, and two daemons running, which you can confirm with ps waux | grep searchd:
evan 1339 0.0 0.0 78100 292 s000 S 5:37PM 0:00.52 searchd --config /config/ultrasphinx/test.conf
evan 1326 0.0 0.0 78100 292 s000 S 5:36PM 0:00.68 searchd --config /config/ultrasphinx/development.conf

Getting sphinx to play well with foxy fixtures

The next problem I discovered was that on some machines, but not others, running my search specs would result in these weird errors:
1)
ActiveRecord::RecordNotFound in 'SearchesController Person searches should search by name'
Couldn't find Person with ID=328556765
/var/www/domains/unithrive/vendor/plugins/ultrasphinx/lib/ultrasphinx/search/internals.rb:308:in `reify_results'
/var/www/domains/unithrive/vendor/plugins/ultrasphinx/lib/ultrasphinx/search/internals.rb:286:in `each'
/var/www/domains/unithrive/vendor/plugins/ultrasphinx/lib/ultrasphinx/search/internals.rb:286:in `reify_results'
/var/www/domains/unithrive/vendor/plugins/ultrasphinx/lib/ultrasphinx/search.rb:362:in `run'
/var/www/domains/unithrive/vendor/plugins/ultrasphinx/lib/ultrasphinx/search/internals.rb:352:in `perform_action_with_retries'
/var/www/domains/unithrive/vendor/plugins/ultrasphinx/lib/ultrasphinx/search.rb:342:in `run'
/var/www/domains/unithrive/app/controllers/searches_controller.rb:38:in `index'
./spec/controllers/searches_controller_spec.rb:51:
script/spec:4:

When I poked into this "Couldn't find Person with ID=328556765" error, it seemed like sphinx was almost working. The index was set up, and the search was finding someone in the index during the test. Ultrasphinx was passing back the id 328556765, which didn't exist in the database. So why would Sphinx "find" a record in its index but then pass back an ID for a database record that didn't exist?

And furthermore, why would it work on one machine, but not on another?

The brainstorm came when I checked what the actual database IDs were for this particular record, with Person.find_by_name("fixtures' name").id. On machines where it worked, the id was a huge number (is it generally is with foxy fixtures), but on machines where it didn't work, the id was an even huger number.

Sphinx tries to make sure that all items that get indexed have a different index in sphinx, and it does this by multiplying all of your id's by N, where N is the number of models getting indexed, and adding an offset of 0 for the first model, 1 for the second, etc. This guarantees that every record from every table will have a unique id. In the case of this application, all of my Person records were getting indexed by sphinx as (Person#id * 4 + 2).

Danger, Will Robinson: 32-bit int rollover!

The problem is that foxy fixtures generate their own ids from a hash of the fixture label, and those ids can be anywhere in the 32-bit unsigned integer space. But Sphinx also stores ids as 32-bit unsigned integers. This means if you happen to get a large fixture id, and then sphinx multiplies it by 4 (or whatever; it could be higher if you have more indexed models), your id will rollover and come out as (id * N + n) % (2^32). Sphinx will store that result, and then when it finds the record in a search, it will try to recreate the original id by subtracting n and dividing by N ... giving you the wrong id. Your test will fail to find the record.

Incidentally, this problem with foxy fixtures is why your test.base file needs the line sql_range_step = 999999999. Sphinx builds indexes by searching a few ids at a time. But the ids generated by foxy fixtures are so big that if sphinx only collects them in ranges of 5000 at a time, it will take forever to find them all.

After some googling, I found that these issues are discussed in a thread over at RubyForge.

The solution

I'm working on a plugin that monkeypatches foxy fixtures to create sequential, low-numbered IDs. In the meantime, you can just compile sphinx to support 64-bit ids, which should give you plenty of headroom to handle foxy fixture ids multiplied by N in sphinx*:

In your sphinx source directory:

configure --enable-id64
make
sudo make install

That should do it. Let me know in comments if any of this information helped you.

*At least until you start approaching 2^32 models in your application, that is.

Rails 2.1 gotcha: don’t name an ActiveRecord field ‘changes’ !

July 10, 2008

In one of my Rails projects, I maintain a user-visible log of updates to the database by recording entries into a table that looks like this (from schema.rb):

create_table "log_entries", :force => true do |t|
t.integer "user_id", :limit => 11
t.datetime "created_at"
t.datetime "updated_at"
t.string "item_type"
t.integer "source_id", :limit => 11
t.string "table"
t.string "action"
t.text "changes"
end

Whenever a record is created, updated, or deleted, I create and save an instance of LogEntry, containing for example {:table => 'task', :action => 'update'} and in the 'changes' column I save a serialized hash showing which attributes of the task object changed before and after save. In addition, I save which logged-in user made these changes, and when.

This is convenient, it gives my client a log that's much more user-accessible and allows them to easily back-trace who did what to the database and when, which is important for their process and certain certifications.

When I upgraded the project to Rails 2.1 a couple of weeks ago, most of the code still worked fine, but the hashes showing the object changes were no longer showing up in views of the log. The culprit turned out to be Rails 2.1's new dirty feature. Why? Because it adds the method 'changes' to ActiveRecord::Base. This is a great new feature that lets you know what has changed to an ActiveRecord object since you loaded it from the database, with all kinds of benefits like doing more selective updates, or not writing to the database at all if no changes have occurred, thus reducing system load if you call save! a lot.

Unfortunately, the new method 'changes' was obscuring my attribute 'changes' in some circumstances, and at the very least confusing the heck out of me, the programmer.

The solution, of course, is not to name your fields anything that corresponds to any Ruby core method or any method of ActiveRecord::Base. I fixed my problem with a migration to rename the column 'details':

def self.up
rename_column :log_entries, :changes, :details
end

def self.down
rename_column :log_entries, :details, :changes
end

Getting <select> options in the right order

June 23, 2008

Sometimes you may want to generate a selector with a fixed set of options.   In one recent task of mine, we needed a selector for an integer 1 through 10, but the client wanted them labeled also with text to identify how the numbers mapped to the words "High" "Medium" and "Low". (We were selecting the priority level of a task in a project management application). Essentially, I want to generate this HTML output:

These are fixed name/value pairs, so it made sense to me to store them as a constant hash in my Task model:

Hash in Task.rb


PRIORITY_OPTIONS = { "1 - High" => "1", "2 - High" => "2", "3 - High" => "3",
"4 - Med" => "4", "5 - Med" => "5", "6 - Med" => "6", "7 - Med" => "7",
"8 - Low" => "8", "9 - Low" => "9", "10 - Low" => "10" }

However, I was rather dismayed to discover that this didn't produce the results I wanted, because Ruby hashes (in Ruby 1.8.6) do not preserve order. This is what came out:

Badly ordered priority selector

That's not what I wanted! I knew that select will also take arrays, but of course I needed separate name/value pairs, which I can't get with just straight arrays. I spent a while playing around with OrderedHash, which exists in Rails but is essentially undocumented and, as it turns out, does not support any useful functions of Hash like merge! and insert! that might make it easy to construct my list of options.

The Fix: Array of Arrays, or Array of Hashes

The documentation is not entirely clear on this, but if you send an array of 2-element arrays to select, rails will use the two elements of each inner array as if they were key and value pairs, and because the entire structure is an array it will preserve order. So, to get the results I wanted, I just need to change my constant to this:

PRIORITY_OPTIONS = [ ["1 - High", 1], ["2 - High", 2], ["3 - High", 3],
["4 - Med", 4], ["5 - Med", 5], ["6 - Med", 6], ["7 - Med", 7],
["8 - Low", 8], ["9 - Low", 9], ["10 - Low", 10] ]

And I get the result I was looking for :

Correctly ordered select options.

As it turns out, the way ActionView processes the options is fairly general: if you pass it any enumerable object, it will iterate that object, and for each element will check to see if that element supports the methods :first and :last (and isn't a string). If so, it will generate an option with the text set to element.first and the value set to element.last. If it was a string, or didn't support first and last, both the text and value of the option are set to the element itself.

Testing it

Here's a handy function I use for testing it the presence of a selector. You pass it the name of your selector, the name of hash or array of options (in the any format supported by select), and optionally the value of an item that should be pre-selected, and it will assert the existence of each of those things. If you need to handle selectors with multiple selections, you can just wrap the last assertion in a loop.

Drop this in test/test_helper.rb

# Assert existence of form select input
# the

#

# or as an array like this:
# [ 'foo', 'bar' ] will be asserted to match
#

#

def assert_select_input(name, option_values, options={})
attributes = { :name => name }

# first assert that the select tag exists
selector = { :tag => "select", :attributes => attributes }
assert_tag selector

option_values.each do | opt |
if !opt.is_a?(String) and opt.respond_to?(:first) and opt.respond_to?(:last)
assert_tag({:tag => "option", :attributes => { :value => opt.last },
:parent => selector, :content => opt.first })
else
assert_tag({:tag => "option", :attributes => { :value => opt },
:parent => selector, :content => opt })
end
end

# check for the pre-selected option, if any
if options[:selected]
assert_tag :tag => "option", :attributes => {:selected => 'selected',
:value => options[:selected] }
end
end

As an example of how to use it, here's my method for testing the task priority selector pictured above:

# asserts a