Stuff from 2007.05.02-03

• Libevent woes

  Evan has been tracking down a bug in the library used for monitoring files and sockets, Libevent. It is supposed to abstract the best platform-specific library into an event- based one. On BSD this means using kqueue, epoll on Linux and so on (it can fall back on select() or poll() also.)

  Aanyway. Apparently epoll is causing problems with regular files for some reason. There is currently a workaround in place but if anyone wants to try to tackle it, go for it.

• I/O buffering

  Currently no buffering is being done on I/O (except of course whatever the platform may add) which means it is performing less-than optimally. A few ideas were tossed around from writing our own buffering to borrowing ideas from nginx’ probably brilliant implementation.

• !=

  Defiler was working on implementing != as a method rather than as a simple translation to a not compiler translation. A default implementation in (presumably) Object would still be a not-condition but it might allow some nice tricks where and != might measure different things. This is, of course, wildly incompatible with MRI.

• Packaging

  brixen was wondering about future packaging needs and whether the rubinius team should be responsible for maintaining those instead of the traditional shift of responsibility to third-party packagers. Discussion on the topic and volunteers most welcome.

• Debugging

  It is about time to start implementing a debugger to help with some of the trickier VM issues. Implementation is at this point completely open.

• Developments

  A mysterious coder by the IRC nick of mojambo has been working on Time.

• Loading libraries

  kevinclark (guess what his real name is.. no fun some of these people not embracing anonymity) found out that the syntax for loading a library (#require, #load etc.) is

  out = VM.load_library ‘path/to/file’, ‘file’

  where the first argument is the full path and the second is the extension name. In the context of normal Ruby loading these will be the same.

  The extension suffix is currently determined statically for the platform (.so, .bundle or .dll) in library.c.

• rbconfig

  The above discussion lead into a short discussion about implementing rbconfig.

• Raw logs

  I have the raw IRC logs available uncropped and infrequently updated with few gaps at http://files.kittensoft.org/rubinius.txt

Hi!

After a hiatus due to an ill-advised COBOL->Java conversion project at work, I may again have enough time to post these semi-regularly. My apologies. I will start off easy with two topics and then try to work in some older news along with a hopefully-steady stream of, er, new news.

I think this forum might actually be easier for me to work with so I will try it out for a bit. Perhaps brixen could set up a nice subforum just for the Newsletter for ease of navigation and RSS’ing.

• Locality of testing platforms

  Evan put some collected money to good use and got an Intel Mac Mini. Currently he has managed to cram Solaris and FreeBSD on there as well so potential platform-specific problems and tweaks should be easier to do. He also said that if we ever get an UltraSparc machine, I get to house it :D

• Speccing

  We (and by we I mean brixen) finally got fed up trying to get RSpec to run natively on Rubinius and the host/target workaround so he wrote a quick little replacement residing at spec/mini_rspec.rb.

  I contributed a few small improvements and a stupid bash kludge to run the specs so now

  bin/specrunner 

  will allow you to run new-style specs with relative ease.

  New-style? Well, with the change we can finally get rid of the example do … end syntax for the specs. This means that all current specs have to be restructured to the normal spec-style. I am currently working down spec/core (Enumerable now) and brixen is “cherry-picking” files to convert whatever that means. Any help there is appreciated.

  Next step, once all of the specs are converted, is to start running them at least semi-regularly. Watching thousands of specs go by is probably not the easiest way to figure out whether your change is working as intended so the Rakefile has two new tasks, spec:diff:run and spec:diff:save. The first one grabs a previous capture of a full spec run from spec/diffs/base.txt and runs a diff -u against a current spec run. This should make it easier to see what has changed and whether you might have broken something else. The latter task just saves a new base run.

Until later! Watch this space.

The past few days have been almost entirely chatless on the channel, people being involved in various time-consuming projects. Yours truly has been suffering from the quinquennial case of Being Sick (a good Constitution score keeps me healthy for the intervening periods—definitely not a dump stat for a Code Wizard) and apparently some wild, roaming dog ate everyone else’s homework.

The next installment should actually have something more interesting again.

Sorry about the drought, busy few days.

• loop is Aki Reijonen

Topics:

• FMC in trunk

  evan pushed in FastMethodContext, the main part at least. Not sure if anyone has done benchmarking on this yet.

• Large object support

  This is something that is to be added (very large arrays for example.) These kinds of objects will probably be handled differently in the VM for efficiency.

• Paths

  Rubinius is not too savvy with paths for various important files yet; make sure you run everything from the top-level rubinius directory to avoid strange problems of not finding stuff.

• Developments

  loop put together a patch to fix a bundle of stuff as his first contribution, thanks!

• Instance variables versus fields

  evan added a piece about this to the wiki, go forth and read!

  HackingGuide

• Benchmarking

  Article

• Local variable scoping

  evan has decided to move to use the method of local scoping that MRI and jruby employ.

• Method calling semantics etc.

  evan and headius had a discussion about the recently proposed changes to MRI method calling (methods vs. functional calls) and method visibilities and so on, also touching on possibly removing stuff from Kernel into a Functions module.

  I need a bit more time to process this so you will have to wait for a longer dissection.

• Building on OSX

  Apparently OOo installs its own broken version of dlfcn.h in /usr/local/include. Symptoms are DL_xx errors. had this problem when trying to build. It can be solved by renaming that file and symlinking to /usr/include/dlfcn.h.

• Trouble tickets

  pate went in and added tickets for pretty much everything under the sun. If you are looking for things to do, head over to the tracker on rubini.us and pick one to work on. Hooray for pate!

New stuff.

• Matters of precedence

  An important part still to be designed and implemented is the Ruby Native Interface or RNI (to be renamed following a contest that closes on Monday over on pate’s blog.) This layer will support writing C extensions—and running old extensions from MRI.

  However, another important step before this is the stabilisation of the shotgun platform. Both the compiler and VM side need some work. I have been working on getting everything to be able to be compiled with shotgun instead of relying on rcompile. The next step would be to actually make the compiled code /work/.

  pate’s blog

• sirb and evaling

  evan pushed in a change that allows local variables to persist between cycles in sirb (essentially by allowing using the same vm state) so sirb should work closer to normal now.

• New developments

  zimbatm is working on some String stuff (one patch is already in) and hornbeck is starting his tour of duty with Time!

• Pointers…?

  mae suggested that we allow (via some primitive-like syntax) access to direct memory pointers. The question is, can you come up with a good use case for this?

• FMC backtrack

  evan decided on another approach for the faster MethodContext merge; he is currently breaking it into pieces that can be individually merged to trunk. The fmc branch does still exist if you want to look at it.

• There is Marshalling and there is Marshalling

  Sidenote-type thing: the marshal format used by rubinius is not the same that MRI uses. There will be a compatibility or conversion library.

• Website changes

  http://rubini.us is moving to use devalot instead of collaboa. You can head over there and create accounts yourself.

• Notoriety

  Someone posted this, ah, ‘insightful’ commentary.

  http://jesusphreak.infogami.com/blog/rubinius_fights_on

  It was discussed in some detail—too much, in fact. They are clearly trying to distract us by their suckiness.

• Virtues of atomic commits

  Discussion was had about atomic commits—any commit made should be self-contained and usually minimal set of changes strictly related to one thing (i.e. not putting vm fixes and a new method implementation in the same commit, for example.) This helps focus and clear the intention of a changeset as well as make it easier to roll it back if a part does not function as expected.

  Evan has expressed a preference for not getting eight commits in a half-hour so we certainly try to abide by that—but it does not mean commits cannot be atomic!

• The Injured and Vacationing list

  evan is going to be out vacationing for a week-ish but cabo has returned from his excursion so we all have someone to bother!

• Architecture, self-hosting and bootstrapping

  gcorriga stopped by and asked to verify his impression that rubinius produces C code from Ruby to create the VM and, if so, would it allow interpreting the same code.

  He was informed that, currently, the C is a manual translation from the Ruby VM and the two are not kept in sync.

  The idea is to move to the cuby model of writing the VM in Ruby and translating that code to C that can be compiled for the twofold goal of making the VM easier to develop and having the C speed. The cuby stuff can be bootstrapped using the existing tools and then fully migrate to it instead.

• Heisenbug rears its.. erm, is that a tail?

  With the recent commits, everything that is needed to build the compiler using shotgun actually compiles (does not run yet though.)

  In testing this, the problem nicknamed Heisenbug by cabo manifested itself by causing a segfault in the compilation process—but only on 32-bit. On 64-bit everything works fine.

  You can use the RUBINIUS_OMSIZE env to try to work around this (OM standing for Object Memory.) A value around two million (2000000) or up should be alright.

  Hopefully we will be able to get rid of the problem soon.

These summaries are available on the Web:

• Web
• Atom

The rubinius IRC channel is #rubinius on Freenode. Join us!

As an administrative note, these summaries are now available on the Web, too.

• Web
• Atom

The process is manual so expect some propagation delay.

New people:

• scoopr: Not much is known about this reclusive creature that dwells in the cold regions of Northern Europe.

• imperator: Daniel Berger

On to the topics!

• Future of the parser

  We were just idly speculating about the future of parsing Ruby scripts in Rubinius. As you may know, the Bison grammar used by MRI are actually currently actively hindering some syntax improvements to it (and it is likely going to be a problem for us as well.)

  A few options have been tossed around, most prominently ANTLR for which a C backend exists and the XRuby people have kindly already written a BSD-licenced grammar as a part of the Rubyfront project.

  Alternatives include the currently en-vogue Kelbt as well as variants of the PEG parsing scheme.

  • XRuby
  • Kelbt
  • PEG

• Symbols now complete

  The past weekend there was a lot of chatter from mainly me talking to myself about dynamic Symbol processing in the parser. First with cabo’s kind assistance we got all literal Symbols to function right and when I finally gave up and kludged a bit, dynamics work fine and get compiled right. The only missing part is that some Symbols have an #inspect string that differs from MRI.

• Mathematical developments

  scoopr is diving in to Rubinius development and starting it by reproducing the Math module from Core!

  mae and brixen have, at length, talked out how different aspects of the numerical system should work; Floats in particular. Topics of interest have been various methods, the role of NaN, IEEE and coercion and so on.

• Got irb.

  brixen and a cadre of folks from (I believe) the pdx.rb group got a start on getting sirb (shotgun irb, simple irb, whatever) to work again and have succeeded! It is limited but bin/sirb.rb will let you do a bit of playing-around.

• FastMethodContext!

  evan uploaded his FastMethodContext implementation as a branch. To take a peek at it and help with debugging, head over to

  svn co http://code.fallingsnow.net/svn/rubinius/branches/fmc

  (Or of course just svn up if you already have trunk and braches.)

• Lair of the Test Monster

  My superior interrogation technique of just asking about it got imperator to reveal the location of his secret testing site.

  • http://rubyforge.org/viewvc/ruby_test/?root=shards

  Certainly something to look into.

• Fiscal policies

  As an aside, imperator wondered about rounding up some venture capitalists. I expressed some scepticism and a preference for free money but he seemed fairly confident. Perhaps he is right!

• Writings

  mae started a blog on his http://mattelder.org.

• rubini.us downtime

  The site is back up now.

The Rubinius IRC channel is #rubinius on Freenode. Join us!

New names:

• pate is Pat Eyler!

Topics:

• Philosophy of specifications

  A very prominently ongoing discussion in the past few days has been brixen and mae’s contest over approaches to testing and speccing with yours truly chiming in for the former upon occasion. My super- crystallisation of the two theories (logs available for the curious and brave):

  Top-down YAGNI: start with top-level interfaces, descend recursively but only as deep as is necessary.

  Lateral exhaustive: every method should be tested, in isolation.

  It should be noted that these may best be styled as professional disagreements: neither approach is likely to be vastly superior to the other and can surely coexist peacefully.

  Now, the top-down posse will be wearing Green and the exhaustivist hoodlums Red. We meet behind Lou’s Diner on Tuesday night. No knives, bats and clubs are fine. If the cops get a whiff, the rumble is off!

• Debugging switches

  As you may know, a couple switches help in debugging shotgun. There is really no performance hit either so it might be fun to just keep the switches up and watch the happy flow of sexp.

  -dc   Debug compiler, shows sexps and assembly information.
  -dl   Debug loader, though not much information available.

• Empirical IRC statistics

  The past few days I have seen previously idle or nonexistent people chime up, ask questions etc. Hooray!

  Channel record is still at 50 people.

• Rubinius Bug Days?

  Super-sleuthing by your intrepid reporter has brought to light a fantastic scoop!

  pate, the Ever Motile One, brought up having dedicated days to hack on bugs in large groups of people. His probing of the tendrils of his vast network of programmers has uncovered a great deal of interest in the idea from (currently-)non-rubinius folks so this could well be an excellent two-fer-one for us all! Hooray for pate!

  But we will let him do the real announcements when time draws near.

• What, exactly, is a Primitive?

  Amidst mae’s work on Primitives, evan confirmed something that to a degree defines what one is: each Primitive should have a ‘self’ object of a known class.

  What does that mean? Primitives are tiny pieces of C code that implement functionality that is not possible to achieve in pure Ruby—but at the same time are clearly a part of the core libs as opposed to instructions which deal more with the language itself as well as the VM.

• Future, ahaha, of concurrency

  MenTaLguY popped by which meant it was time for an invigorating discussion about different threading models and about the possi- bility of seeing something called ‘thunk-replacement’ which as far as I understood would mean interpreter-level support for transparent Futures.

  Futures, then, are essentially concurrent operations hiding behind proxy objects that transform into the Real Thing once the computation completes.

  Then talk shifted to even more fantastic applications of various concurrency schemes but at this point my brain’s safety went off and I shut down for about five minutes to avoid overheating.

  Dogs are fuzzy. Dogs.

• Human interest

  In lighter news, we had two rather serious bugs that affected HEAD for a little while but everything is hunky-dory right now and work is ongoing getting the functionality in place.

The Rubinius IRC channel is #rubinius on Freenode. Join us!

Apologies about the delay, for some reason I thought I did not have enough content to ‘publish’ yesterday. Hopefully none of you had to go on a crime spree due to missing your fix.

Additional apologies are offered for including information from the commit stream and any non-biased commentary instead of delivering just cold, hard IRC news. Think of those parts as op-eds. Or the cartoons.

• Class variables vs. class instance variables again

  mae and I talked about having @@x be syntactic sugar for (self.class.) instance_eval {@x} since it would break compatibility with 1.8 (1.9 follows pretty much the same track as proposed change). Decision was to be deferred to evan.

• Get to Know a Macro pt. 2

  REFERENCE_P tells you whether an object reference cleverly holds an immediate value instead of a reference, just like in MRI. The objects that appear as immediate values are as follows: nil, true, false, undef, Symbols and Fixnums. The reference is encoded to make this detection simple.

  In addition, you can use things like STRING_P to figure out whether a reference points to a core class instance.

• Block parameter semantics

  cabo and I put our heads together trying to figure out how exactly block parameters work in some kind of strict terms. The issue is mostly with multiple parameters. Here are a few test cases I wrote: http://pastie.caboo.se/37715.

  And this is what we came up with:

  1. Multiple arguments are always treated as separate.
  2. If the only argument is a single Array, multiple assignment semantics are used.
  3. If the only argument responds to #to_ary, 2) applies.
  4. * behaves normally.
  5. If the block takes only one parameter, a warning is emitted if multiple are passed.
  6. If the block takes multiple parameters, extra ones are silently ignored.
  7. If the block takes multiple parameters and too few are passed in, missing ones are silently assigned nil.
  8. Damnit, I forgot what this one was when I had to rewrite this after my battery died. Damnit all!

  Please add any cases these do not address.

• How does addressing work?

  In my travails through the compilation path I took a look at assembler.rb as well and my grand hypothesis was confirmed by cabo. The bytecode addresses for labels and gotos are simply byte offsets in the encoded instructions. The assembler keeps tabs on how many bytes different instructions consume with @current_op which can be used when labeling is needed.

• ruedbg

  The incredible ruedbg will walk you through the transformation of sexp to assembly by compiler.rb with merely food, sleep and some money! For self-education I was tracing this piece of code through the compiler to see how the processing went. You can follow along on your local copy of compiler.rb!

  [[1, 2]].each {|a, b| p a, b}  # http://pastie.caboo.se/37718

• Know your datatypes: Tuple

  Tuple is somewhat like a very primitive Array. It has two properties: 1) it has a fixed number of fields, 2) any objects can be inserted and removed to/from those fields.

  Rubinius uses Tuples for many internal representations including in argument passing. They are also exposed to the outside world as, essentially, an additional Core class.

• Primitives

  mae and evan have engaged in a back-and-forth all through the weekend regarding changes to primitives. mae is keen on trying to isolate them as much as possible along with some more drastic changes but those are, for now, on hold. One product of this has been the addition of the argument count to send_primitive, enabling mae to better test the primitives.

• Committing changes

  evan stated a preference of receiving slightly larger aggregate changesets instead of minimal one- or two-line changes (barring high-importance stuff of course) very frequently.

• The Big Reveal nighs

  evan is polishing off his addition of among other things a fastmethodcontext to replace the current presumably slower MethodContext which should make things on the VM side snappier.

  He also indicated that in the future he would try to set up branches for these types of large all-encompassive changesets so that we may view the code while he is working at it without compromising stability in the trunk. Hooray!

• Subversion (and svk)

  We rehashed all the old centralised- vs. distributed Version Control System arguments without really getting anywhere but perhaps that was not the point. Fun was had by all.

  In the process, I set up svk on this side to help with the workflow. svk help intro gave me all the information I needed once I actually FOUND it :/

• Issues of importance

  To help get rid of obsolete.rcompile, shotgun needs to get some problems fixed. Run bin/reports/self_hosting_errors.rb and see where help is needed!

1
2

  "a#{@x}b"  => [:dstr, "a", [:evstr, [:ivar, :@x]], [:str, "b", 0]]
  :"a#{@x}b" => [:dsym, [:array, [:evstr, [:ivar, :@x]], [:str, "b", 0]]]

• Compatibility

  brixen and I had a spirited conversation about incompatibility with headius putting in some insightful commentary. Since the IRC Correspondents write the history, here is my take.

  There are two kinds of incompatibility. The first, clearer one is that there is an interface incompatibility with the reference implementation. For example, having String#split return a Tuple instead of an Array or different semantics for the ternary op. Another would be a missing feature, shipping without Hash for example or substituting it with a Dict. These types of problems are less likely to be an issue.

  The second and slightly harder form are additions to the standard (if a portable library is not available.) Again I pick on poor Tuple but if Tuple is available in Rubinius as, essentially, an additional Core class, then what is there to stop Bob Programmer from using it to write his MightyIRCBot? And what if Bob wants to distribute the program, will people need to have multiple Ruby implementations available? For the sake of the argument, assume that no portable Tuple

  My point was not necessarily to assign responsibility to anyone over Bob’s decision but to state that /regardless of whose ‘fault’ it is/, it will be a problem. I am sure all of the long-time programmers remember a time when a meticulously crafted program either completely exploded or had its performance halved due to a platform not offering all of the features of another one.

  My personal ideal would be that /so long as there must be different implementations of Ruby/, Matz defines what the language is and all implementations conform to it so that any Ruby application can run on any of the platforms. These points are of course all moot should Matz decide that Rubinius is where ‘it’ is ‘at’ and decree that all other Ruby implementations must die a horrible death and enforce singularity.

The Rubinius IRC channel is #rubinius on Freenode. Join us!