Kittensoft - rubinius

Phase Three

2008-11-20T04:41:00Z

Evan broke the news earlier about Engine Yard reducing funding for Rubinius with the unfortunate effect that four of the original six, myself included, will no longer be employed there. My three truly excellent colleagues will have absolutely no problems finding employment, nor do I foresee any stalling for Rubinius—I will most certainly continue work at the usual pace after taking a couple days to arrange some things on my end.

And, really, the story is that Engine Yard continues to sponsor Rubinius development, even in the current climate. I cannot but be glad that they gave this level of support for any period of time, let alone a full year. It has been a superb opportunity.

I must also express my gratitude to the community for the frankly astounding wave of support in the last couple days. You all rock.

This leaves me looking for new challenges too, if you will. A few offers have come in, but if you have interesting work for me, please write me under “projects” in the base domain here. (Either the U.S. or Finland—or telecommuting anywhere, naturally.)

Guy Decoux Dies

2008-09-24T23:39:00Z

Jean-François Trân informed Ruby-Core that ‘ts’, Guy Decoux, has died.

Extremely sad :/

Defensive Programming

2008-09-12T09:57:00Z

In the process of studying the current codebase, I ran into some functions whose existence I knew about: as() and try_as(). Both are essentially cast functions, explicit in the spirit of static_cast() but slightly tailored for our purposes (in the VM, each class contains its type as data explicitly, e.g. ArrayType and so on.) All Ruby classes are subclasses of Object, and the instances are passed around as Object*; the functions take the type to attempt to cast into and the object. as() will raise a TypeError if the cast is not possible, i.e. the real type of the object is not kind_of? the target type. try_as() is meant to be used in conditions and returns NULL for an invalid cast for those cases where the object's type is not always clear. Sounds good, right?

Well, one thing caught my attention. Both of the functions were treating the object already being NULL as a normal, ignorable case. I thought about it for a moment, and could not come up with any valid situation where one would have an unknown completely uninitialised object. In some cases a local object pointer might be NULL until it gets initialised (to Qnil or Qundef if nothing else), but in those cases you also know what the type is already. So this seemed like something that could mask problems.. I asked for opinions on #rubinius, since it is always possible I was just not understanding the valid use case, but no-one else could think of any either so I changed both functions to raise an Assertion if given a NULL.

Aside from fixing some shortcuts in the tests exposed by this, I did actually find a fairly nifty bug! LookupTable, which is used as an extra-lightweight hash, was raising Assertions left and right. Somehow NULLs were getting in the hash buckets. After some research, it turned out that the offender was this line in LookupTable::remove (pseudo for clarity):

link = try_as<Tuple>(bucket.value_or_another_hash_bucket);

The intention is to check if the bucket has a link to another one and (later) recurse further into it if so. The problem is that if the element was not another bucket, i.e. of type Tuple, the link became NULL instead of the value object contained in this bucket. The rest of the code incorrectly assumed that link would always be the Tuple or the value so it was storing the NULLs. A quick fix to correctly handle the value, too, made sure we were not going to lose methods or instance variables and such.

Lesson in this? 1. I should really condense my writing; but also 2. always be as strict as possible. It helps to ensure you weed out stuff that you had not considered when writing the code, whether it turns out to be correct or incorrect behaviour.

The Return of the Son of the Thing from Planet Rubinius' Mother III

2008-09-11T11:44:00Z

I am currently working directly with Rubinius with the rest of the guys, getting the C++ rewrite flying: the milestone is set at being able to run mspec and the spec suite again. Specifically, I am reimplementing the C extension API from Shotgun called Subtend (though I hope we can just call it RNI..) and in the process I have had to familiarise myself with the new codebase, since I sadly did not have the time to do that earlier.

It is nice. Really nice. The OO aspect has made it a virtual pleasure to read. The logic is clearer and responsibilities are separated much better. There certainly is room for improvement, but then there always is. Granted, it _is_ still a VM implementation, but it is far more approachable than it was before. I will try to do my part to make sure there is enough documentation for the tricky and overarching parts: so far I have written a description of the method dispatch system (which is ultimately fairly simple, but being the central function of the VM it involves most of the codebase in one way or another) and just added class and method documentations in the headers in (doxygen)[http://doxygen.org] syntax so you might actually see the generated API docs online at some point too.

ucontext Broken on OS X 10.5 Leopard

2008-09-08T22:20:00Z

Apparently due to an issue between headers and the libc implementation on Leopard, ucontext_t are not allocated enough space which will cause bus errors or segmentation faults, typically when returning from the function to which the execution has setcontext() -ed to a context previously grabbed using getcontext() . This affects both x86 and PPC, and a bug has been filed although not addressed yet.

The workaround is to manually size the types correctly. So if you have, say, a struct like this:

#include <ucontext.h>
struct  thready {
  :
  ucontext_t  some_context;
  ucontext_t  some_other_context;
  :
};

The fixed version inserts the padding:

#include <ucontext.h>

#define HAS_UCONTEXT      // Or something

struct thready {
  :
  #if defined(__APPLE__) && defined(HAS_UCONTEXT)
    ucontext_t some_context;
    _STRUCT_MCONTEXT  __some_context_mc;
    ucontext_t some_other_context;
    _STRUCT_MCONTEXT __some_other_context_mc;
  #else
    ucontext_t some_context;
    ucontext_t some_other_context;
  #endif
};

Naturally, you can also use the conditional separately for each ucontext_t if it is easier or more logical in your case and/or refine the conditional itself.

I had a terrible time trying to search for information, and most stuff I found seemed to indicate PPC issues only. In the interest of hopefully making the search shorter for someone else, I will gratuitously list the terms I was looking for: ucontext, ucontext_t, getcontext, setcontext, makecontext, swapcontext, segfault, sigbus, bus error, stack corruption, returning from, os x, 10.5, leopard, x86, ppc.

Hopefully they will get it fixed soon!

Conditional Compilation of Ruby Code

2008-04-01T23:31:00Z

It was one of those silly ideas-nights and I decided to add the ability to have blocks of code that can be omitted from the bytecode depending on a condition as a proof-of-concept. Once the parser is in, we can start working on ‘real,’ more generic sexp and AST manipulators towards something approaching Lisp’s facilities.

I figured it would be a fairly short task and for a change I was correct&emdash;and a good portion of the thanks goes to Rubinius’ already-excellent facilities: I was able to modify the sexp with the full power of Ruby. It took me a bit to figure out the best way to set this up but in the end there are literally about 11 lines of implementation code plus the specs. The end result is a block construct for a special form that allows conditionalising. Currently I only set it up to process global variables but it is trivial to add support for the rest—what may not be trivial is for the user to get other types of conditions set up without confusion over compile-time versus runtime. The invocation is pretty simple:


  # ...

  Rubinius.compile_if($DEBUG) { Logger.debug "Blah blah: #{some_variable}" } 

  # ...

If $DEBUG is false at the time this section goes through the compiler, that entire line disappears from the first R to the last }. If it evaluates to true, the block contents are processed normally&emdash;but we still remove the now-extra wrapper block around it both to avoid the overhead as well as ensure that the variable scope etc. are consistent.

This should be useful for at least removing those generally-unnecessary debug statements etc.; the if $DEBUG does not cost much by itself but it adds up over time. The other easy target are platform/library/etc. specific sections that can be omitted in some cases. It is general-purpose in that the only limiting factor is the condition variable (or possibly expression in the future.)

The commit is here.

Splitting the Specs to a Separate Project with git Submodules?

2008-03-13T01:47:00Z

If we have a separate RubySpec repository which is included to Rubinius by means of git’s submodule support (mounted at spec/ruby/) there are two types of committers: those who work with RubySpec only and those who work with Rubinius only. The workflow for a RubySpec-only committer is very simple:

Clone the main RubySpec to a local working copy.
Make changes.
Push changes to RubySpec mainline or make a patch if no commit bit.

That is the end for that committer. Eventually one of the Rubinius committers will pull in the changes to the Rubinius mainline. For a Rubinius committer wanting to make a change to specs, the process is a bit different. One note to make is that having RubySpecs as a separate project imposes one extra step to the normal workflow even if not making any changes to the specs. When first cloning the rbx repo, one must run `git submodule init && git submodule update` to pull in the specs—which is obviously needed to just RUN the specs. Then, if a `git pull` contains submodule changes, another update is usually required with `git submodule update`.

Rubinius Committers

If a person working with the Rubinius repo wants to make changes or add a new spec or whatever, they can of course choose to just work on a separate RubySpec checkout. If the committer does not have a commit bit to RubySpecs, they MUST use a separate repository. That works exactly as described above.

If, on the other hand, we have an individual who is both a Rubinius and RubySpecs committer and wants to work directly in the Rubinius checkout, then the workflow looks like this:

`cd spec/ruby`
`git checkout master`
Make and commit changes. (Use branches if you like but come back to master.)
Push changes to mainline RubySpecs (just `git push`.) <—THIS IS IMPORTANT
`cd ../..`
`git add spec/ruby` <—NO TRAILING /, that overwrites the submodule
Commit.
Push to mainline Rubinius.

At this point the updated submodule becomes available to others working with the Rubinius repo using `git pull && git submodule update`.

Full Example

Here is a long example of the entire process, covering some ground that TFM did not seem to do. Please send in any corrections and/or improvements.


  # We will work from scratch to illustrate the process
  $ cd ~/code/tmp

  # OK, first set up our submodule
  $ mkdir rubyspec
  $ cd rubyspec
  $ git init
   Initialized empty Git repository in .git/
  $ echo 'rubyspec' > README
  $ git add README
  $ git commit -a -m 'Initial import of rubyspec'
   Created initial commit 4c43e66: Initial import of rubyspec
    1 files changed, 1 insertions(+), 0 deletions(-)
    create mode 100644 README
  $ cd ..

  # OK, now set up the 'public' rubyspec repo (rubyspec.com or whatever)
  $ git clone --bare rubyspec rubyspec.git
   Initialized empty Git repository in /Users/user/code/tmp/rubyspec.git/

  # Next, we create the superproject, obviously the public rbx repo
  # already exists in reality.
  $ mkdir rbx
  $ cd rbx
  $ git init
   Initialized empty Git repository in .git/
  $ echo 'rbx' > README
  $ git add README
  $ git commit -a -m 'RBX import'
   Created initial commit 43b9035: RBX import
    1 files changed, 1 insertions(+), 0 deletions(-)
    create mode 100644 README
  $ cd ..

  # OK, same deal here, make a 'public' repo (git://git.rubini.us in reality)
  $ git clone --bare rbx rbx.git
   Initialized empty Git repository in /Users/user/code/tmp/rbx.git/

  # Now, let us get rid of the originals so we have a more realistic situation
  $ rm -rf rbx rubyspec
  $ ls
   rbx.git            rubyspec.git

  # Next we want to set up the submodule link, and we need a real rbx repo
  $ git clone rbx.git my_rbx
   Initialized empty Git repository in /Users/user/code/tmp/my_rbx/.git/
  $ cd my_rbx
  $ ls
   README
  # Now we associate the submodule (in reality we would use the URI, not local path)
  $ mkdir spec
  # We can 'mount' the submodule anywhere in the repo, in this case it goes in spec/ruby/
  $ git submodule add ~/code/tmp/rubyspec.git spec/ruby
   Initialized empty Git repository in /Users/user/code/tmp/my_rbx/spec/ruby/.git/
  $ ls -la
   total 16
   drwxr-xr-x  4 user  group  204 Mar 12 19:39 .
   drwxr-xr-x  7 user  group  238 Mar 12 19:37 ..
   drwxr-xr-x  7 user  group  374 Mar 12 19:39 .git
   -rw-r--r--  1 user  group   85 Mar 12 19:39 .gitmodules
   -rw-r--r--  1 user  group    4 Mar 12 19:37 README
   drwxr-xr-x  3 user  group  102 Mar 12 19:39 spec

  # .gitmodules has the tracking info
  $ cat .gitmodules
   [submodule "spec/ruby"]
           path = spec/ruby
           url = /Users/user/code/tmp/rubyspec.git
  # Our rubyspec repo should be in place now
  $ ls -la spec/ruby
   total 8
   drwxr-xr-x  3 user  group  136 Mar 12 19:39 .
   drwxr-xr-x  3 user  group  102 Mar 12 19:39 ..
   drwxr-xr-x  7 user  group  374 Mar 12 19:39 .git
   -rw-r--r--  1 user  group    9 Mar 12 19:39 README
  $ cat spec/ruby/README
   rubyspec

  # Now we need to record the submodule in the superproject
  $ git status
   On branch master
   Changes to be committed:
     (use "git reset HEAD <file>..." to unstage)

         new file:   .gitmodules
         new file:   spec/ruby

  $ git commit -m 'Imported rubyspec submodule'
   Created commit 0ae5cd3: Imported rubyspec submodule
    2 files changed, 4 insertions(+), 0 deletions(-)
    create mode 100644 .gitmodules
    create mode 160000 spec/ruby

  # And then record this over at the mainline
  $ git push
   Counting objects: 5, done.
   Compressing objects: 100% (3/3), done.
   Writing objects: 100% (4/4), 439 bytes, done.
   Total 4 (delta 0), reused 0 (delta 0)
   Unpacking objects: 100% (4/4), done.
   To /Users/user/code/tmp/rbx.git
      43b9035..0ae5cd3  master -> master
  $ cd ..

  # Alright. Now, our test situation is that Joe just hacks on rubyspec and
  # Bob hacks on rubyspec as a part of rbx!

  # Let us create Bob's repo quick first:
  $ git clone rbx.git bobs_rbx
  $ cd bobs_rbx
  $ ls -la
   total 8
   drwxr-xr-x  4 user  group  204 Mar 12 19:47 .
   drwxr-xr-x  9 user  group  306 Mar 12 19:44 ..
   drwxr-xr-x  7 user  group  374 Mar 12 19:44 .git
   -rw-r--r--  1 user  group   85 Mar 12 19:44 .gitmodules
   -rw-r--r--  1 user  group    0 Mar 12 19:45 README
   drwxr-xr-x  3 user  group  102 Mar 12 19:44 spec

  # Now, as you can see, a pristine checkout of a superproject will not actually contain the submodule code
  $ ls -la spec/ruby
   total 0
   drwxr-xr-x  2 user  group   68 Mar 12 19:44 .
   drwxr-xr-x  3 user  group  102 Mar 12 19:44 ..

  # We need to first init to properly link the modules and then update to pull the data in
  $ git submodule init
   Submodule 'spec/ruby' (/Users/user/code/tmp/rubyspec.git) registered for path 'spec/ruby'
  $ git submodule update
   Initialized empty Git repository in /Users/user/code/tmp/bobs_rbx/spec/ruby/.git/
   Submodule path 'spec/ruby': checked out '4c43e665381054478aa7c82da1d1d63b5c258e4b'
  $ ls -la spec/ruby
   total 8
   drwxr-xr-x  3 user  group  136 Mar 12 19:48 .
   drwxr-xr-x  3 user  group  102 Mar 12 19:48 ..
   drwxr-xr-x  7 user  group  408 Mar 12 19:48 .git
   -rw-r--r--  1 user  group    9 Mar 12 19:48 README
  # OK, now it is there. Let us hop out of here for a bit.
  $ cd ..

  # Now, Joe is doing some work on his repo which is JUST the rubyspec submodule
  $ git clone rubyspec.git joes_spec
   Initialized empty Git repository in /Users/user/code/tmp/joes_spec/.git/
  $ cd joes_spec
  $ ls
   README
  # OK, Joe makes a change
  $ echo 'new spec' > spec.rb
  $ git add spec.rb
  $ git commit -m 'Added spec.rb'
   Created commit f380258: Added spec.rb
    1 files changed, 1 insertions(+), 0 deletions(-)
    create mode 100644 spec.rb
  # Joe pushes to mainline of rubyspec
  $ git push
   Counting objects: 4, done.
   Compressing objects: 100% (2/2), done.
   Writing objects: 100% (3/3), 290 bytes, done.
   Total 3 (delta 0), reused 0 (delta 0)
   Unpacking objects: 100% (3/3), done.
   To /Users/user/code/tmp/rubyspec.git
      4c43e66..f380258  master -> master

  # Now, let us check in with Bob who has his clone of the rbx repo
  $ cd bobs_rbx

  # Bob does not have Joe's spec yet
  $ ls -la spec/ruby/
   total 8
   drwxr-xr-x  3 user  group  136 Mar 12 19:48 .
   drwxr-xr-x  3 user  group  102 Mar 12 19:48 ..
   drwxr-xr-x  7 user  group  408 Mar 12 19:48 .git
   -rw-r--r--  1 user  group    9 Mar 12 19:48 README

  # Updating does not seem to work either!
  $ git submodule update
  $ ls -la spec/ruby/
   total 8
   drwxr-xr-x  3 user  group  136 Mar 12 19:48 .
   drwxr-xr-x  3 user  group  102 Mar 12 19:48 ..
   drwxr-xr-x  7 user  group  408 Mar 12 19:48 .git
   -rw-r--r--  1 user  group    9 Mar 12 19:48 README

  # Did Joe's change get registered? Let us check..
  $ cd ..
  $ git clone rubyspec.git check_joes_change
   Initialized empty Git repository in /Users/user/code/tmp/check_joes_change/.git/
  $ ls check_joes_change/
   README  spec.rb
  # Yep, it is there! So what is going on?

  # Right, we need to update the mainline rbx. Technically Bob could actually do this
  # too but for now, let us pretend that some access control allows only me to make
  # that update. So back to my repo.
  $ cd my_rbx/
  $
  # Now, we need to grab the change first
  $ cd spec/ruby/
  $ ls
   README
  $ git status
   On branch master
   nothing to commit (working directory clean)
  $ git pull
   remote: Counting objects: 4, done.
   remote: Compressing objects: 100% (2/2)remote: , done.
   remote: Total 3 (delta 0), reused 0 (delta 0)
   Unpacking objects: 100% (3/3), done.
   From /Users/user/code/tmp/rubyspec
      4c43e66..f380258  master     -> origin/master
   Updating 4c43e66..f380258
   Fast forward
    spec.rb |    1 +
    1 files changed, 1 insertions(+), 0 deletions(-)
    create mode 100644 spec.rb
  $ ls -la
   total 16
   drwxr-xr-x  3 user  group  170 Mar 12 19:54 .
   drwxr-xr-x  3 user  group  102 Mar 12 19:39 ..
   drwxr-xr-x  7 user  group  442 Mar 12 19:54 .git
   -rw-r--r--  1 user  group    9 Mar 12 19:39 README
   -rw-r--r--  1 user  group    9 Mar 12 19:54 spec.rb
  # OK, there it is!
  $ cd ../../
  # So, need to record this update in the superproject
  $ git status
   On branch master
   Changed but not updated:
     (use "git add <file>..." to update what will be committed)

         modified:   spec/ruby

   no changes added to commit (use "git add" and/or "git commit -a")
  $ git add spec/ruby
  $ git commit -m 'Updated our rubyspec checkout to newest version'
   Created commit 0d96517: Updated our rubyspec checkout to newest version
    1 files changed, 1 insertions(+), 1 deletions(-)
  # And then push this change to mainline
  $ git push
   Counting objects: 5, done.
   Compressing objects: 100% (2/2), done.
   Writing objects: 100% (3/3), 319 bytes, done.
   Total 3 (delta 1), reused 0 (delta 0)
   Unpacking objects: 100% (3/3), done.
   To /Users/user/code/tmp/rbx.git
      0ae5cd3..0d96517  master -> master
  $ cd ..
  $ cd bobs_rbx/
  # NOW Bob should be able to access it
  $ git submodule update
  # No? How about this:
  $ git pull
   remote: Counting objects: 5, done.
   remote: Compressing objects: 100% (2/2), done.
   remote: Total 3 (delta 1), reused 0 (delta 0)
   Unpacking objects: 100% (3/3), done.
   From /Users/user/code/tmp/rbx
      0ae5cd3..0d96517  master     -> origin/master
   Updating 0ae5cd3..0d96517
   README: needs update
   Fast forward
    spec/ruby |    2 +-
    1 files changed, 1 insertions(+), 1 deletions(-)
  # That by itself does not update yet
  $ ls -la spec/ruby
   total 8
   drwxr-xr-x  3 user  group  136 Mar 12 19:48 .
   drwxr-xr-x  3 user  group  102 Mar 12 19:48 ..
   drwxr-xr-x  7 user  group  408 Mar 12 19:48 .git
   -rw-r--r--  1 user  group    9 Mar 12 19:48 README
  # Now we must update
  $ git submodule update
   Submodule path 'spec/ruby': checked out 'f38025812763dd8ee44beebcc0c8a3997c9aac0f'
  # And now it is there
  $ ls -la spec/ruby
   total 16
   drwxr-xr-x  3 user  group  170 Mar 12 19:57 .
   drwxr-xr-x  3 user  group  102 Mar 12 19:48 ..
   drwxr-xr-x  7 user  group  408 Mar 12 19:57 .git
   -rw-r--r--  1 user  group    9 Mar 12 19:48 README
   -rw-r--r--  1 user  group    9 Mar 12 19:57 spec.rb
  # And there we go, properly updated now

  # Of course, Bob can make a few changes now too. Here is the only potential problem.
  # First, change something in rbx.
  $ echo 'RTFM' > INSTALL
  $ git add INSTALL
  $ git commit -m 'Installation instructions'
   Created commit c0e6667: Installation instructions
    1 files changed, 1 insertions(+), 0 deletions(-)
    create mode 100644 INSTALL
  $ git push
   Counting objects: 4, done.
   Compressing objects: 100% (2/2), done.
   Writing objects: 100% (3/3), 361 bytes, done.
   Total 3 (delta 0), reused 0 (delta 0)
   Unpacking objects: 100% (3/3), done.
   To /Users/user/code/tmp/rbx.git
      0d96517..c0e6667  master -> master
  # That was normal. Now, what if Bob wants to make changes to rubyspec?
  # He can make the changes normally. If he has commit access, he can
  # just hop over and do so.
  $ cd spec/ruby
  $ ls
   README  spec.rb
  # One notable thing is that this is a detached by default to avoid some problems, so
  # Bob must go to a real branch.
  $ git status
   Not currently on any branch.
   nothing to commit (working directory clean)
  $ git checkout master
   Previous HEAD position was b292add... Added a spec
   Switched to branch "master"
  $ echo 'my spec' > spec2.rb
  $ git add spec2.rb
  $ git commit -m 'Added a spec'
   Created commit b1e7bdb: Added a spec
    1 files changed, 1 insertions(+), 0 deletions(-)
    create mode 100644 spec2.rb
  # Now Bob can just push it if he has access
  $ git push
   Counting objects: 4, done.
   Compressing objects: 100% (2/2), done.
   Writing objects: 100% (3/3), 313 bytes, done.
   Total 3 (delta 0), reused 0 (delta 0)
   Unpacking objects: 100% (3/3), done.
   To /Users/user/code/tmp/rubyspec.git
      f380258..b1e7bdb  master -> master
  # As the data shows, we directly updated the rubyspec project
  $
  # Obviously, if Bob did not have commit rights, he would do the normal git-format-patch dance instead
  $
  # Now, the problem
  $ cd ../..
  $ git status
   On branch master
   Changed but not updated:
     (use "git add <file>..." to update what will be committed)

         modified:   spec/ruby

   no changes added to commit (use "git add" and/or "git commit -a")
  # This shows the change we made--and adding it is perfectly safe so long
  # as we do it correctly. This is incorrect:
  $ git add spec/ruby/
  $ git status
   On branch master
   Changes to be committed:
     (use "git reset HEAD <file>..." to unstage)

         deleted:    spec/ruby
         new file:   spec/ruby/README
         new file:   spec/ruby/spec.rb
         new file:   spec/ruby/spec2.rb

  $ git reset --hard HEAD
   HEAD is now at c0e6667... Installation instructions
  # OK, so that is not good. The trailing / has git thinking this is a new
  # path which overwrites the submodule. So, without:
  $ git add spec/ruby
  $ git status
   On branch master
   Changes to be committed:
     (use "git reset HEAD <file>..." to unstage)

         modified:   spec/ruby

  # So that worked fine. Now we can update mainline normally
  $ git commit -m 'New spec in rubyspec, updated here and pushed to rubyspec mainline'
   Created commit 3749f4b: New spec in rubyspec, updated here and pushed to rubyspec mainline
    1 files changed, 1 insertions(+), 1 deletions(-)
  $ git push
   Counting objects: 5, done.
   Compressing objects: 100% (2/2), done.
   Writing objects: 100% (3/3), 331 bytes, done.
   Total 3 (delta 1), reused 0 (delta 0)
   Unpacking objects: 100% (3/3), done.
   To /Users/user/code/tmp/rbx.git
      c0e6667..3749f4b  master -> master
  $

Reinventing the Which Wheel Comes First

2008-02-15T04:49:00Z

Aside from research, the past few days I have been putzing with reimplementing Rubinius' Array#sort and Array#sort! because, well, they sucked. The implementation is an iterative three-way partitioning Quicksort that drops down to Insertion Sort for small Arrays or subpartitions. Two observations:

The new implementation is at least one order of magnitude faster in all areas compared to the old one. Pathological cases are much improved from this, even, three orders of magnitude usually. String Arrays are about two orders of magnitude faster on small-medium Arrays and perform at the same scale with (theoretically) infinitely large ones. The old version ran out of stack space around 2000 or so.
MatzRuby is typically still far faster, but particularly the block forms are actually fairly close, in some cases actually within the same order of magnitude (albeit slower.) And the new version is pure Ruby all the way.

Here are some benchmarking figures. Note the disparity between the Array sizes.

For MatzRuby and new Rubinius: Numeric, 1000 elements, 100 loops 146233 strings, each with 20 characters For old Rubinius: Numeric, 100 elements, 100 loops 1462 strings, each with 20 characters #sort MatzRuby OLD* NEW ----------------------------------------------- sorted 0.007812 2.581782 0.602038 sorted block 0.195312 4.064099 1.782695 reversed 0.007812 4.423980 8.090656 reversed block 0.195312 4.909585 26.117707 random 0.023438 0.920588 1.643777 random block 1.914062 0.967784 3.984373 same 0.007812 0.062574 0.462366 same block 1.906250 0.915978 4.052798 two 0.007812 0.099981 0.426605 two block 1.898438 0.902365 3.756180 three 0.000000 0.118842 0.500360 three block 1.898438 0.907539 4.471767 mixed 0.015625 0.635714 1.161138 mixed block 1.859375 0.809821 3.037062 strings 0.015625 5.512026 3.742163 strings block 0.304688 10.301844 7.038649

Next mini-project will be converting Array to use a more efficient storage mechanism. That should be fun.

Platform Abstraction Musings

2008-02-13T03:25:00Z

The current platform abstraction in Rubinius is a bit hodgepodge--it works, but the responsibilities and hierarchies are not defined in any predictable manner. What I have been thinking about is a basic idea of a simple hierarchical model where each platform is resolved to a class that provides a complete implementation for it by inheriting and selectively overriding the specifications from its superclasses.

General idea

So, we take POSIX to be one of the top-level implementations (others could be WinNT etc.) POSIX implements the default platform that could run any theoretical operating system that did not deviate from or extend the standard in any (meaningful for us) way. Then, POSIX can be subclassed for example as Linux, which allows us to override the parts of POSIX where a Linux machine will differ or provide extra functionality. This can further be subclassed as Debian < Linux, for example, if anything like that is needed. Furthermore, we can use Ruby modules to represent cross-cutting concerns. So while we could split Linux into Linux32 and Linux64, it may be simpler to just implement Linux as a 32-bit and provide Linux64 (or even just a Generic64 for some things) as a module that can be inserted anywhere it is needed. Further, we can even divide to OS versions.


    # A 64-bit MonkeyLinux 1.8 could look like this
        POSIX
          ^
          |
        Linux
          ^
          |
     MonkeyLinux <<< Linux64
          ^
          |
    MonkeyLinux::V18

The beauty of it is, of course, that it is not necessary to go that deep. If Linux Distro X does not need to specify any behaviour deviating from the base Linux provided, then that is what is used.

Config vs. Runtime

This scheme needs to be split into two components: the build-time config such as struct and constant definitions and the runtime that uses those definitions where needed, uses FFI to attach functions etc. Ideally we would implement both of these in the same class/module hierarchy thereby reducing any mismatch problems but parallel hierarchies work too if they turn out to be considerably easier to implement.

Either way, the first step would be the build-time setup of creating platform.conf, types.conf and so on. This is already done with the tools we have (StructGenerator et al.) but we can make it easier to manage by splitting it up into the above-described hierarchy. When the time comes to build, we just resolve the correct class to use (more about this further down) and ask it to generate the config information or provide the data to do so--not sure what will turn out to be easier here yet. It would also probably be a good idea to codify the exact platform class used here so that it does not need to be re-resolved when the runtime starts. The runtime side of course largely corresponds to what you can see in kernel/platform now, with the exception that, for example, the .attach_method call that ultimately gets run is possibly the result of overrides in the descent through the class hierarchy instead of a hardcoded guess.

Resolving the Correct Platform

The only obvious problem remaining is the actual method of determining which path should be taken in the platform implementation hierarchy. Broadly, two methods can be used: registration or resolution. The former keeps some type of a matching table that selects the correct class based on the key (platform info) given. The latter, which at this point seems favourable, would instead take the approach of testing the platform info against its children's filters/resolvers and passing it further down as necessary. In this scheme, let us say that 'amd64-freebsd6.2' is passed in. The top-level Platform uses POSIX.resolve to determine it is a POSIX platform (initially this 'check' is probably platform !~ /winnt/i or something) and then POSIX peddles it to its subclasses until possibly BSD.resolve accepts it and so on until no subclass can further resolve it. Modules, of course, are taken into account in this whenever inserted.

All Rubinius All the Time

2008-02-11T00:50:00Z

The cat is out of the bag, so I can finally publically express a huge "YAY" for the opportunity to work full-time on Rubinius, courtesy of Engine Yard. I will specifically be architecting and implementing the mod_rubinius approach to simplifying Ruby web application deployment while keeping it as (or more) performant (mainly Rails, obviously, but others will benefit equally.)

I will still have plenty of time to do what I have been doing thus far with Rubinius, too. Might even have time to regularly publish the Newsletter again! After all, everything up to now has been on my spare time and I am just trading 10 hours of wasted time at my previous job and commute for 9-10 hours of mod_rubinius work--and I still have the same amount of free time to work on the project as I have had before! I think I need to come up with a way to get out of the house for a bit every day, though, to avoid cabin fever.

The mod_rubinius work itself will of course necessarily involve a lot of Rubinius work, initially at least in the realm of multi-VM (Rubinius can run completely separate interpreters one per native thread), Rubinius' own C interface (as opposed to Subtend) and the basic I/O layer.

You have a say in what happens with mod_rubinius, too! Hop over to Ezra's post to tell us exactly how you deploy your Merb/Ramaze/Rails/Nitro/IOWA/plain CGI/whatever applications in your wildest dreams.

Lastly, a huge thanks to Engine Yard for letting me invest my full energy in this! I cannot overstate how great everyone at EY has been throughout my transition, and honestly I do not think I could hope to do better than to be able to make a living from what started as a hobby.

Compiler Walkthrough Illustration

2007-11-09T19:49:00Z

Note: this is a sort of a draft. See the real document over in the repo.


# Start
[:block, 
  [:newline, 1, "(eval)", 
    [:defn, :foo, 
      [:scope, 
        [:block, 
          [:args, [:a, :b], [], nil, nil], 
          [:newline, 1, "(eval)", 
            [:yield, 
              [:call, [:lvar, :a, 0], :+, [:array, [:lvar, :b, 0]]], false
            ]
          ]
        ], 
        [:a, :b]
      ]
    ]
  ]
]

# into_script
[:script,  
 [:block, 
   [:newline, 1, "(eval)", 
    [:defn, :foo, 
      [:scope, 
        [:block, 
          [:args, [:a, :b], [], nil, nil], 
          [:newline, 1, "(eval)", 
            [:yield, 
              [:call, [:lvar, :a, 0], :+, [:array, [:lvar, :b, 0]]], false
            ]
          ]
        ], 
        [:a, :b]
      ]
    ]
   ]
  ]
]

# Script.create strips :script
[
 [:block, 
   [:newline, 1, "(eval)", 
     [:defn, :foo, 
       [:scope, 
         [:block, 
           [:args, [:a, :b], [], nil, nil], 
           [:newline, 1, "(eval)", 
             [:yield, 
               [:call, [:lvar, :a, 0], :+, [:array, [:lvar, :b, 0]]], false
             ]
           ]
         ], 
         [:a, :b]
       ]
     ]
   ]
 ]
]
# out = [<script node>]


# Script.consume
# convert sexp.first ->
# Block.create comp, sexp
 [:block, 
   [:newline, 1, "(eval)", 
     [:defn, :foo, 
       [:scope, 
         [:block, 
           [:args, [:a, :b], [], nil, nil], 
           [:newline, 1, "(eval)", 
             [:yield, 
               [:call, [:lvar, :a, 0], :+, [:array, [:lvar, :b, 0]]], false
             ]
           ]
         ], 
         [:a, :b]
       ]
     ]
   ]
 ]
# out = [<block node>]

# Block#consume sexp.shift
 [ 
   [:newline, 1, "(eval)", 
     [:defn, :foo, 
       [:scope, 
         [:block, 
           [:args, [:a, :b], [], nil, nil], 
           [:newline, 1, "(eval)", 
             [:yield, 
               [:call, [:lvar, :a, 0], :+, [:array, [:lvar, :b, 0]]], false
             ]
           ]
         ], 
         [:a, :b]
       ]
     ]
   ]
 ]
# out = [<newline node>]

# Compiler#convert_sexp first
# -> Newline.create
# Newline.new.consume sexp.shift
   [1, "(eval)", 
     [:defn, :foo, 
       [:scope, 
         [:block, 
           [:args, [:a, :b], [], nil, nil], 
           [:newline, 1, "(eval)", 
             [:yield, 
               [:call, [:lvar, :a, 0], :+, [:array, [:lvar, :b, 0]]], false
             ]
           ]
         ], 
         [:a, :b]
       ]
     ]
   ]

# << out = [1, "eval", <defn node>]
# compiler.convert <defn node>

# Define.create
# Define.new.consume sexp.shift
# name = sexp[0] (:foo), body = sexp[1..-1]
# scope = super(body)
# ClosedScope#consume body
     [
       [:scope, 
         [:block, 
           [:args, [:a, :b], [], nil, nil], 
           [:newline, 1, "(eval)", 
             [:yield, 
               [:call, [:lvar, :a, 0], :+, [:array, [:lvar, :b, 0]]], false
             ]
           ]
         ], 
         [:a, :b]
       ]
     ]
# out = []

# Scope.new.consume sexp.shift
# convert args.first, 
# out = [<converted>, locals]
       [ 
         [:block, 
           [:args, [:a, :b], [], nil, nil], 
           [:newline, 1, "(eval)", 
             [:yield, 
               [:call, [:lvar, :a, 0], :+, [:array, [:lvar, :b, 0]]], false
             ]
           ]
         ], 
         [:a, :b]
       ]

# Block.new.consume sexp.shift
# :args => convert
# :newline => convert

         [ 
           [:args, [:a, :b], [], nil, nil], 
           [:newline, 1, "(eval)", 
             [:yield, 
               [:call, [:lvar, :a, 0], :+, [:array, [:lvar, :b, 0]]], false
             ]
           ]


# out = [<args node>, <newline node>] 

# Arguments.new.consume sexp.shift
[[:a, :b], [], nil, nil]

# Newline.new.consume sexp.shift
            [1, "(eval)", 
             [:yield, 
               [:call, [:lvar, :a, 0], :+, [:array, [:lvar, :b, 0]]], false
             ]
            ]

IRC Newsletter has moved

2007-07-12T18:27:00Z

Hello!

More recent IRC dispatches can be found on the Rubinius website.

Thanks for stopping by!

The Dailyish IRC Newsletter, 2007.05.03

2007-05-05T01:42:00Z

Stuff from 2007.05.02-03

Libevent woes
Evan has been tracking down a bug in the library used for monitoring files and sockets, Libevent. It is supposed to abstract the best platform-specific library into an event- based one. On BSD this means using kqueue, epoll on Linux and so on (it can fall back on select() or poll() also.)

Aanyway. Apparently epoll is causing problems with regular files for some reason. There is currently a workaround in place but if anyone wants to try to tackle it, go for it.

I/O buffering
Currently no buffering is being done on I/O (except of course whatever the platform may add) which means it is performing less-than optimally. A few ideas were tossed around from writing our own buffering to borrowing ideas from nginx’ probably brilliant implementation.

!=
Defiler was working on implementing != as a method rather than as a simple translation to a not compiler translation. A default implementation in (presumably) Object would still be a not-condition but it might allow some nice tricks where and != might measure different things. This is, of course, wildly incompatible with MRI.

Packaging
brixen was wondering about future packaging needs and whether the rubinius team should be responsible for maintaining those instead of the traditional shift of responsibility to third-party packagers. Discussion on the topic and volunteers most welcome.

Debugging
It is about time to start implementing a debugger to help with some of the trickier VM issues. Implementation is at this point completely open.

Developments
A mysterious coder by the IRC nick of mojambo has been working on Time.

Loading libraries
kevinclark (guess what his real name is.. no fun some of these people not embracing anonymity) found out that the syntax for loading a library (#require, #load etc.) is

out = VM.load_library ‘path/to/file’, ‘file’

where the first argument is the full path and the second is the extension name. In the context of normal Ruby loading these will be the same.

The extension suffix is currently determined statically for the platform (.so, .bundle or .dll) in library.c.

rbconfig
The above discussion lead into a short discussion about implementing rbconfig.

Raw logs
I have the raw IRC logs available uncropped and infrequently updated with few gaps at http://files.kittensoft.org/rubinius.txt

The Dailyish IRC Newsletter, 2007.05.02

2007-05-04T00:21:00Z

Hi!

After a hiatus due to an ill-advised COBOL->Java conversion project at work, I may again have enough time to post these semi-regularly. My apologies. I will start off easy with two topics and then try to work in some older news along with a hopefully-steady stream of, er, new news.

I think this forum might actually be easier for me to work with so I will try it out for a bit. Perhaps brixen could set up a nice subforum just for the Newsletter for ease of navigation and RSS’ing.

Locality of testing platforms
Evan put some collected money to good use and got an Intel Mac Mini. Currently he has managed to cram Solaris and FreeBSD on there as well so potential platform-specific problems and tweaks should be easier to do. He also said that if we ever get an UltraSparc machine, I get to house it :D

Speccing
We (and by we I mean brixen) finally got fed up trying to get RSpec to run natively on Rubinius and the host/target workaround so he wrote a quick little replacement residing at spec/mini_rspec.rb.

I contributed a few small improvements and a stupid bash kludge to run the specs so now
```
bin/specrunner <file>
```
will allow you to run new-style specs with relative ease.

New-style? Well, with the change we can finally get rid of the example do … end syntax for the specs. This means that all current specs have to be restructured to the normal spec-style. I am currently working down spec/core (Enumerable now) and brixen is “cherry-picking” files to convert whatever that means. Any help there is appreciated.

Next step, once all of the specs are converted, is to start running them at least semi-regularly. Watching thousands of specs go by is probably not the easiest way to figure out whether your change is working as intended so the Rakefile has two new tasks, spec:diff:run and spec:diff:save. The first one grabs a previous capture of a full spec run from spec/diffs/base.txt and runs a diff -u against a current spec run. This should make it easier to see what has changed and whether you might have broken something else. The latter task just saves a new base run.

Until later! Watch this space.

The Dailyish IRC Newsletter, 2007.02.21-23

2007-02-24T07:23:00Z

The past few days have been almost entirely chatless on the channel, people being involved in various time-consuming projects. Yours truly has been suffering from the quinquennial case of Being Sick (a good Constitution score keeps me healthy for the intervening periods—definitely not a dump stat for a Code Wizard) and apparently some wild, roaming dog ate everyone else’s homework.

The next installment should actually have something more interesting again.