Headache.new(Unicorn + Capistrano + Bundler + USR2)

If you're running Unicorn with Bundler and deploying with Capistrano like we are at Flipstone, you may run into trouble if you implement USR2 based restarts with Unicorn. It may appear as if Unicorn is ignoring your USR2 signal and never launching a new process. That may not be what's happening at all. You may be being bit by an insidious and hard to track down interaction between the three libraries.

The answer, it turns out, lies here in the Unicorn documentation. Look for the heading about BUNDLE_GEMFILE when using Capistrano. What's actually happening is that Unicorn is attempting to re-exec a new binary on receiving the USR2 signal, but that new binary is almost immediately failing with a bundler complaint about a missing Gemfile. Unless you're watching the process list closely, you'll probably miss it and think the signal was ignored. You can verify this is the problem by looking in the unicorn.log file for a missing Gemfile message.

Bundler sets BUNDLE_GEMFILE in the environment when it starts up. If you're using Capistrano, the path to the Gemfile is stored with release timestamp in the path, rather than the current symlink. That will allow unicorn to restart just fine using the old Gemfile as long as you don't add any gems and the Gemfile remains in place. If you're cleaning up old releases (as you should be) with Capistrano, you'll end up with Unicorn failing to restart after the release that BUNDLE_GEMFILE is pointing to is deleted. In our case, that turned out to be after 5 deployments, since we keep 5 old releases.

The solution, which I found here, is to reset BUNDLE_GEMFILE in your unicorn configuration file before any workers are forked, like so:

before_exec do |server|
  ENV["BUNDLE_GEMFILE"] = "/path/to/app/current/Gemfile"

Once you do that, your application will always pick up the Gemfile from the current Capistrano deployment instead of any old ones.