David Vollbracht - Home

Next AFPUG Meeting is April 6th

2010-03-09T14:29:00Z

The next meeting of the Atlanta Function Programming Users Group is going to be Monday, April 6th. All the details are available here.

As I mentioned in my last post, the agenda is to compare implementations of the first 10 Project Euler problems in different functional languages. I look forward to seeing everyone’s problem solutions!

First AFPUG Meeting!

2010-03-02T16:36:00Z

Last night we held the first Atlanta Functional Programming Users Group meeting at the ThoughtWorks office in Atlanta. I gave a short introduction Haskell, which generated a lot of good questions. You can check out the presentation at the group’s slideshare page.

While the group’s next meetup date wasn’t specifically decided, it should sometime in April. The agenda for the next meeting is to compare solutions to the first 10 Project Euler problems in multiple functional languages. Our door is always open to new members, so come and join us!

The Dark Side of Elastic IPs

2010-02-19T23:10:00Z

Did I inherit thothle.ca?

Google “David Vollbracht”. What domain name comes up? Right now the domain is thothle.ca. In fact, you may even be viewing this page on thothle.ca. This is quite awkward, as I do NOT own thothle.ca. I’m not even Canadian. Nonetheless, I’m writing this post right now on thothle.ca, because it seemed appropriate.

whois thothle.ca

No, I won’t post all the contact info here, but suffice to say the nameservers for thothle.ca are from mydyndns.org. My conclusion: the owner of thothcle.ca was using my Amazon elastic IP and released it. He did not, however, update his dns to reflect that the IP is no longer his. Consequently his domain now points to my website, and has become the #1 page on google when searching for my name.

Conclusion

Obviously this isn’t really Amazon’s fault. They can’t exactly follow up after every IP is released and check where it is pointing now. The best I can do is contact the domain’s owner and hope he’ll change the nameserver config to not point to the IP. Funnily enough, this kind of reminds me of the poor chap who has my old cell phone number… I can empathize now.

Come to the first AFPUG Meeting!

2010-02-16T11:38:00Z

I’m pleased to announce that ThoughtWorks is sponsoring the first Atlanta Function Programming Users Group meeting. All the info is here at our google group page. I’ll be giving a short introduction to the Haskell programming language and then we’ll open up the floor to discussion about the group’s future. If you’re in Atlanta (or can drive there) and you’re interested in Functional Programming, this is the place for you!

Now Hosted on EC2

2010-02-03T16:23:00Z

I’ve moved the hosting for this blog from my old VPS into Amazon EC2. My calculations indicate this should save me around $9-$10 a month when using a single reserved m1.small instance. This is a pretty good deal since an m1.small instance has over twice as much memory and about 60% more cpu horsepower. The EC2 setup isn’t very complicated, so I’ll give you an overview right here.

I’m running a single elastic ip attached to a single running m1.small instance. I started with the public Fedora Ruby on Rails Web Starter AMI (ami-22b0534b) and lightly customized it to fit my needs. That included installing a few rubygems and mucking about in init.d and chkconfig to get the services starting the way I wanted.

The most complicated part of the setup was separating out some parts of the machine to an EBS volume for persistent storage. In particular, the mysql database needs to be on persistent storage so it is safe if the instance goes down. Also, I wanted any rails apps to be on persistent storage so I don’t have to create a new ami for every single configure change, upgrade, or deployment. All this is achieved simply by putting symlinks to the normal places where these would live in the filesystem and pointing them to the mount point for the EBS volume.

With that done and working the only remaining issue was to ensure that the EBS volume gets attached and mounted. A bit of googling confirmed that creating an init.d script was the generally accepted approach for this, and I found a workable starting point here. That solution relied on the EC2 api tools, which in turn rely on Java which I didn’t relish the idea of installing. Instead I installed the very nice amazon-ec rubygem and rewrote the mountec2vol script on top of it. Voila! Now bringing up a new instance attaches the EBS, mounts it, and then starts any services that needed the volume once it is available.

If you’re interested in the init.d script, you can get it here. It’s limited currently by having the volume id specified in the script, so you have to rebuild the ami to change the volume id. I’ll probably extract that in the future, but it’s doing what I need for now. If you find it useful, let me know!

DynamicTeardown

2009-07-09T03:13:00Z

I ran into a situation while working on DeepTest recently where I wanted an object to start a temporary drb server in the middle of the test and have it be stopped automatically at the end of the test. Ideally, I wanted to encapsulate inside the class itself, which is a fake used for testing and so can be aware of the test framework. Here’s the snippet I ended up with:


module DynamicTeardown
  class <<self
    def dynamic_teardowns
      @dynamic_teardowns ||= []
    end

    def on_teardown(&block)
      dynamic_teardowns << block
    end

    def run_dynamic_teardowns
      while td = dynamic_teardowns.shift
        td.call rescue nil
      end
    end
  end
end

module DRbTestHelp
  def drb_server_for(server)
    drb_server = DRb::DRbServer.new "druby://localhost:0", server
    DynamicTeardown.on_teardown { drb_server.stop_service }
    return DRbObject.new_with_uri(drb_server.uri)
  end
end

class Test::Unit::TestCase
  def teardown
    DynamicTeardown.run_dynamic_teardowns
  end
end

This nice little implementation allows the class to know that it needs to get some teardown done while allowing the test framework to determine when the teardowns are called. So far it’s working great for me.

DeepTest 2.0 in the works

2009-06-24T10:53:00Z

I’ve started on a significant overhaul of DeepTest which will be released as version 2.0 when it’s finished. I thought I’d share some of my plans for what 2.0 will include and some thoughts about what might come versions after. I don’t have a timetable for yet, but work so far is going fairly quickly.

Plans for DeepTest 2.0

A revamped distributed DeepTest architecture that will help ease setup and maintenance.
A new console user interface that will provide much more information about what is going on during the test run
Much better error handling & reporting (no more “Maybe an error was printed above?”!)
RSpec integration that doesn’t break with every new release of rspec.
Automatic configuration for Rails projects

What comes after 2.0? The next big feature I’d like to implement is support for cloud testing with deep test on EC2. Other than that, the features coming after 2.0 will likely be driven be requests. Is there a feature you’d like to see in DeepTest? Comment here or message me (qxjit) on github and we’ll get it rolling!

30 Days of Tech: Day 30 - Haskell Decompose

2008-07-01T02:48:00Z

Some time ago Tom, a friend, presented a math programming challenge to a me and a few others. The challenge was to create a function that when given a positive number would return a set of all the sets of positive numbers whose sum was the original number. He wasn’t so much interested in the result itself as he was in what sort of techniques were used in the implementation—memoization of course being a choice candidate for this problem. I was toying a lot with Haskell at the time, and since this was fundamentally a math problem, Haskell seemed well suited to the problem too.

Here’s what I came up with:


  module Decompose
    where

  import Data.List

  decompose = ((map decompose' [0..]) !!) where
    decompose' n = nub (map sort (decompositions n (floor ((toRational n)/2))))

  decompositions n 0 = [[n]]
  decompositions n m = nmCombinations ++ decompositions n (m - 1) where
    nmCombinations = [a ++ b | a <- (decompose (n - m)), b <- (decompose m)]

I was surprised at how short the Haskell solution was. If you run this through Hugs, you’ll see something like this:


  Hugs> :load "Decompose" 
  Decompose> decompose 5
  [[1,1,1,1,1],[1,1,1,2],[1,2,2],[1,1,3],[2,3],[1,4],[5]]
  Decompose> decompose 10
  [[1,1,1,1,1,1,1,1,1,1],[1,1,1,1,1,1,1,1,2],[1,1,1,1,1,1,2,2],[1,1,1,1,1,1,1,3],[1,1,1,1,1,2,3],[1,1,1,1,1,1,4],[1,1,1,1,1,5],[1,1,1,1,2,2,2],[1,1,1,2,2,3],[1,1,1,1,2,4],[1,1,1,2,5],[1,1,2,2,2,2],[1,2,2,2,3],[1,1,2,2,4],[1,2,2,5],[1,1,1,1,3,3],[1,1,2,3,3],[1,1,1,3,4],[1,1,3,5],[2,2,3,3],[1,2,3,4],[2,3,5],[1,1,4,4],[1,4,5],[5,5],[1,3,3,3],[3,3,4],[2,2,2,2,2],[2,2,2,4],[2,4,4],[1,1,1,1,6],[1,1,2,6],[2,2,6],[1,3,6],[4,6],[1,1,1,7],[1,2,7],[3,7],[1,1,8],[2,8],[1,9],[10]]

The only performance trick in here is indeed memoization, though that may not be obvious at first. It lies in the definition of decompose. The decompose function is really just a list of decomposition results for all positive numbers, indexed by the number being decomposed. Obviously this would be an infinitely long lost, but Haskell has no problem handling that—it just calculates elements of the list on demand. Since decomposing a number always involves decomposing all the numbers below it, the fact that you have to calculate the list out to the length of the original number being decomposed is not a hindrance, it’s actually an advantage!

30 Days of Tech: Day 29 - C, Ruby, & Threads, Oh My!

2008-06-30T03:07:00Z

When Philippe and I worked on SystemTimer together we went back and forth about how much to write in Ruby and how much to write in C. Writing purely at the C level gave us more control over what was happening in the Ruby process as we installed and cleared signal handlers and such, but working in Ruby produced far more readable code.

Ultimately we ended up writing a hybrid—Ruby code that calls C code that calls back into Ruby code for a few interesting bits. When I go back and look at the result now, I’m pretty happy with it. There was one trick we had to pull in our situation, however, to ensure everything went according to plan. Since we entered C code to perform low level operations with timers we felt we didn’t want our code to be interrupted by Ruby’s thread scheduling. We wanted to ensure the timer got set up properly without the chance of another thread being scheduled. At first this led is to try to write as much in C as possible, since we wouldn’t ever hit the Ruby thread scheduler. Eventually we realized we could clean up a bunch of C code by extracting Ruby methods. To prevent other threads from being scheduled, we simply extracted C functions to wrap the Ruby methods, like this one.


  static void install_ruby_sigalrm_handler(VALUE self) {
    rb_thread_critical = 1;
    rb_funcall(self, rb_intern("install_ruby_sigalrm_handler"), 0);
    rb_thread_critical = 0;
  }

The C function install_ruby_sigalrm_handler is called by another C function, and is quite readable in that context. If you’re reading the C code, you don’t even have to know it’s calling back to Ruby. All the C function does, however, is set rb_thread_critical and then call back into Ruby code. rb_thread_critical is the equivalent of Thread.critical in the Ruby API. When it is set, Ruby will not schedule another thread (except in a few cases documented with Thread.critical). Using this simple tool we were able to effectively blend Ruby and C code without worrying about our operations being preempted. Arigatoo, Matsumoto-san.

30 Days of Tech: Day 28 - lambda args

2008-06-29T03:08:00Z

The Ruby documentation for the lambda method (a.k.a. proc) says “Equivalent to Proc.new, except the resulting Proc objects check the number of parameters passed when called.” This is almost exactly right. There are, however, at least two cases where this turns out to be false.

The first is when the block passed to lambda does not have an argument list specified. The resulting Proc object will not check arguments in this case.


  def test_lambda_checks_arguments
    l = lambda {||}
    assert_raises(ArgumentError) {l.call(1)}
  end

  def test_lambda_doesnt_check_arguments_if_no_arguments_specified
    l = lambda {}
    assert_nothing_raised {l.call(1)}
  end

The two tests above prove that Proc#call doesn’t check arguments for a lambda proc with no arguments unless you specify that the proc takes no arguments by including an empty parameter list. This is not the only case where argument checking is not done for lambda procs, however.

If you pass a lambda proc to a method is a block (using &), this circumvents the argument checking that would normally take place, even when you specify a blank argument list.


  def test_lambda_doesnt_check_arguments_when_yielded_to_even_if_arguments_specified
    l = lambda {||}
    def yield_1_to; yield 1; end
    assert_nothing_raised {yield_1_to &l}
  end

The above test passes, even though one argument is yielded to a lambda proc that explicitly takes no args. I haven’t read the source yet, but my guess about what’s happening here is that Ruby converts the proc back to a normal block. Since blocks normally don’t check arguments, there’s no reason to expect the one here to either. Nonetheless, the result surprised me until I thought about it longer.

I categorize this as one of those Ruby oddities that for the most part doesn’t hurt you in practice. I don’t recall ever having run into a bug or debugging issue where not knowing about this hindered me. I do think that building knowledge about oddities like these is good practice though. It deepens your knowledge of Ruby in general, and every so often one small bit of knowledge will save you a chunk of time.

30 Days of Tech: Day 27 - Short Circuited Timeout

2008-06-28T03:30:00Z

Ruby’s timeout method operates by raising a exception, Timeout::Error. Timeout::Error inherits from Interrupt, SignalException, and finally Exception. In particular, it doesn’t inherit from StandardError, which means it isn’t caught by a default rescue clause with no exception class specified. When I first encountered this, I was annoyed that I had to explicitly catch timeout errors anytime I wanted to recover from errors in general, whether they were timeout errors or not. I realized something the other day that changed my opinion on this however.

Here’s the timeout method for Ruby 1.8.6.


  def timeout(sec, exception=Error)
    return yield if sec == nil or sec.zero?
    raise ThreadError, "timeout within critical session" if Thread.critical
    begin
      x = Thread.current
      y = Thread.start {
        sleep sec
        x.raise exception, "execution expired" if x.alive?
      }
      yield sec
      #    return true
    ensure
      y.kill if y and y.alive?
    end
  end

It operates by starting a thread, y that will raise an exception after sleeping. It raises the exception by calling x.raise, causing the exception be raised in thread x, which is the thread that called timeout—the one doing the work that needs to be timed out. That exception terminates the execution of the thread wherever it happens to be and starts propagating an exception up the stack. This exception is subject to the same handling as every other exception that might happen. It can caught be a rescue clause. In particular, it can be caught by a rescue clause that is within the block being subjected to the timeout


  require 'timeout'

  begin
    Timeout.timeout(1) do
      begin
        sleep 2
      rescue Exception => e
        puts "Caught #{e.inspect} within timeout" 
      end
    end
  rescue Exception => e
    puts "Caught #{e.inspect} at top level" 
  end

This produces the output Caught #<Timeout::Error: execution expired> within timeout. If Timeout::Error extended from StandardError, it would be caught by any generic rescue expression that would handle exceptions occurring at the point where the timeout error happened to be raised. This rescue clause may not even be in your application code—it could be in a library or the Rails framework, which could result in very hard to track down bugs. We can verify this is the case by specifying what type of error timeout should raise.


  require 'timeout'

  begin
    Timeout.timeout(1, StandardError) do
      begin
        sleep 2
      rescue => e
        puts "Caught #{e.inspect} within timeout" 
      end
    end
  rescue => e
    puts "Caught #{e.inspect} at top level" 
  end

The only differences here are that Timeout.timeout is passed the type of exception to raise, StandardError, and that the rescue clauses have be changed to omit the exception type. The output is essentially the same as before, Caught #<StandardError: execution expired> within timeout.

Incidentally, SystemTimer terminates the offending thread by raising an exception as well. If there is a rescue clause that handles Timeout::Error within the block given for either Timeout or SystemTimer, it could prevent the error from reaching handling code outside the timeout call. Fortunately the likelihood of encountering this scenario is minimized because Timeout::Error doesn’t inherit from StandardError. I haven’t hit a bug caused by this in practice yet. I’m sure if I had before I had thought through the situation it would have taken me a lot longer to figure out what was going on.

30 Days of Tech: Day 26 - Fixnum#object_id

2008-06-27T03:25:00Z

In Ruby every object has an object_id that uniquely identifies that object in that particular instance of the Ruby process. Ruby also treats integers as objects—they are instances of Fixnum. For efficiency reasons, though, Ruby implements integers internally as simple values. That is, it doesn’t allocate heap memory for every integer you use, it just passes the value around instead. So how does Ruby generate object ids for all these integers?

You may have never noticed, but all the object ids that you usually end up seeing are even.


  irb(main):011:0> Object.object_id
  => 110230
  irb(main):012:0> Object.new.object_id
  => 191490
  irb(main):013:0> nil.object_id
  => 4
  irb(main):014:0> "a_string".object_id
  => 177360

This is because Ruby is reserving all the odd numbers to be object_ids for Fixnums.


  irb(main):015:0> 1.object_id
  => 3
  irb(main):016:0> 2.object_id
  => 5
  irb(main):017:0> 3.object_id
  => 7
  irb(main):018:0> 4.object_id
  => 9

Not only are they reserved, they’re calculated using a simple formula! This allows Ruby to easily get back to the value from the id if you should ever ask for it.


  irb(main):022:0> ObjectSpace._id2ref(9)  
  => 4

And now if you ever see an odd (as in n % 2 = 1, not strange) object_id in some Ruby output somewhere, you’ll know immediately what the value of that object is!

30 Days of Tech: Day 25 - Gem dependency version

2008-06-26T01:27:00Z

RubyGems and I had a fight. It was about what version of net-ssh we wanted to have installed on my machine. RubyGems wanted the latest (2.0.2) and I wanted a 1.x version (1.1.4). Apparently net-ssh 2 has an issue with sshing into Fedora Core 4, which I needed it to do. Specifically I needed to use capistrano 1.4.1 to deploy an app to Fedora Core 4

This is something I’ve been doing regularly for some time, but ever since the MacBrick incident I haven’t needed these specific gems installed, so I was starting from scratch. Once I figured out what version of the gems I wanted, here was my first attempt: (I included --no-rdoc and --no-ri just to save time writing this post).


  $ sudo gem install net-ssh --version=1.1.4 --no-rdoc --no-ri
    Bulk updating Gem source index for: http://gems.rubyforge.org/
    Successfully installed net-ssh-1.1.4
    1 gem installed
  $ sudo gem install capistrano --version=1.4.1 --no-rdoc --no-ri
    Bulk updating Gem source index for: http://gems.rubyforge.org/
    Successfully installed net-ssh-2.0.2
    Successfully installed net-sftp-2.0.1
    Successfully installed capistrano-1.4.1
    3 gems installed

Clearly this didn’t work so well. Installing capistrano installed the latest version of net-ssh. But why? Capistrano’s gemspec for 1.4.1 says this


  ...
  s.add_dependency(%q<net-ssh>, [">= 1.0.10"])
  s.add_dependency(%q<net-sftp>, [">= 1.1.0"])
  ...

But net-sftp 2.0.1’s gemspec says it needs net-ssh 1.99.1. So we need to install specific version of both. So, attempt number two.


  $ sudo gem install net-ssh --version=1.1.4 --no-rdoc --no-ri
    Bulk updating Gem source index for: http://gems.rubyforge.org/
    Successfully installed net-ssh-1.1.4
    1 gem installed
  $ sudo gem install net-sftp --version=1.1.0 --no-rdoc --no-ri
    Bulk updating Gem source index for: http://gems.rubyforge.org/
    Successfully installed net-ssh-2.0.2
    Successfully installed net-sftp-1.1.0
    2 gems installed

Rats. Even though net-sftp 1.1.0 only specifies that it depends on net-ssh >= 1.0.0, RubyGems went ahead and installed the latest version anyway. It assumed that even though the dependency was satisfied, I want the latest version of net-ssh— exactly what I’m trying to avoid.

Armed with this information, I made one last ditch attempt.


  $ sudo gem install capistrano --version=1.4.1 --no-rdoc --no-ri
    Bulk updating Gem source index for: http://gems.rubyforge.org/
    Successfully installed net-ssh-2.0.2
    Successfully installed net-sftp-2.0.1
    Successfully installed capistrano-1.4.1
    3 gems installed
  $ sudo gem install net-ssh --version=1.1.4 --no-rdoc --no-ri
    Bulk updating Gem source index for: http://gems.rubyforge.org/
    Successfully installed net-ssh-1.1.4
    1 gem installed
  $ sudo gem install net-sftp --version=1.1.0 --no-rdoc --no-ri
    Bulk updating Gem source index for: http://gems.rubyforge.org/
    Successfully installed net-sftp-1.1.0
    1 gem installed
  $ sudo gem uninstall net-sftp net-ssh

    Select gem to uninstall:
     1. net-sftp-1.1.0
     2. net-sftp-2.0.1
     3. All versions
    > 2
    Successfully uninstalled net-sftp-2.0.1

    Select gem to uninstall:
     1. net-ssh-1.1.4
     2. net-ssh-2.0.2
     3. All versions
    > 2
    Successfully uninstalled net-ssh-2.0.2

Finally, by letting RubyGems install the latest version of net-ssh and net-sftp, then installing the versions I wanted, and the uninstalling the offending versions, I was able to get into a working state to deploy. Hopefully I’ll never run into this problem again. If you should be so unlucky, however, I hope this knowledge will help you.

30 Days of Tech: Day 24 - drubyall

2008-06-25T03:18:00Z

When we first started trying to use distributed DeepTest, we quickly ran into an issue with DRb servers not binding to the correct addresses. DRb does provide a way to bind to 0.0.0.0 to listen to all addresses, but there’s a downside to the way it does it. The url you provide to DRb to accomplish this must omit the hostname (e.g. druby://:0/). DRb then determines what hostname to report to clients by calling getaddrinfo with the empty hostname. Unfortunately for us getaddrinfo didn’t provide a hostname that could be used by our developer machines to connect to the DeepTest servers. For a brief moment we thought we were stuck.

When we looked into how DRb handles protocols, however, we realized we could simply add a protocol that would accept a hostname as part of the url and use it to report the DRb url to clients when needed. For listening on the network, however, the protocol always binds to 0.0.0.0. By registering this protocol (which we called drubyall since it listens on all interfaces) you can use it in any DRb url, which is pretty handy. Although it wasn’t nice that we ran into the problem, it was very nice that DRb gave us an easy out to extending its communications layer.


  module DeepTest
    class DRbBindAllTCPSocket < DRb::DRbTCPSocket
      def self.parse_uri(uri)
        if uri =~ /^drubyall:\/\/(.*?):(\d+)(\?(.*))?$/
          host = $1
          port = $2.to_i
          option = $4
          [host, port, option]
        else
          raise(DRb::DRbBadScheme, uri) unless uri =~ /^drubyall:/
          raise(DRb::DRbBadURI, 'can\'t parse uri:' + uri)
        end
      end

      # Open a server listening for connections at +uri+ using 
      # configuration +config+.
      def self.open_server(uri, config)
        uri = 'drubyall://:0' unless uri
        host, port, opt = parse_uri(uri)

        if host.size == 0
          host = getservername
        end

        DeepTest.logger.debug("Listening on port #{port}, all addresses.")
        soc = TCPServer.open('0.0.0.0', port)          
        port = soc.addr[1] if port == 0
        uri = "druby://#{host}:#{port}" 
        self.new(uri, soc, config)
      end
    end
  end

  DRb::DRbProtocol.add_protocol DeepTest::DRbBindAllTCPSocket

30 Days of Tech: Day 23 - DearSoul

2008-06-24T02:11:00Z


  class DearSoul
    def self.new(first, last)
      ObjectSpace.define_finalizer(allocate, proc {|id|
        puts "Goodbye, #{first}.  We'll miss you." 
      })
    end
  end

  DearSoul.new("George", "Carlin")