Progress on 64-bit port

First the good news:
We can now generate, from a single set of sources, four different VMs...

 One that will run a 32-bit image on a 32-bit host
  same old VM
 One that will run a 32-bit image on a 64-bit host
  needed to run on 64-bit machines
 One that will run a 64-bit image on a 32-bit host
  great for testing 64-bit VM on 32-bit machines
 One that will run a 64-bit image on a 64-bit host
  the thing we were after

... and all of them work!

Ian and I worked together on this, with Ian focused on issues of C
translation rules, and me focused on properly generalizing all the
occurrences of 4 (bytes per word), 2 (shift count), and lots of other
things derived therefrom.  I also wrote a tracer subclass that will write
images in a 64-bit format, and Ian also contributed the brilliant
framework that enabled this four-way compatibility.

Now the fine print...
The 64-bit format is the simplest change that could work, not the simplest
format that could work.  For those who understand the object current
format, here is the change:

 All object pointers are 64 bits as you'd expect.

 SmallIntegers are still 31-bits, although stored sign-extended to 63
 bits (plus tag bit).

 32-bit Object header:
 3 bits reserved for gc (mark, old, dirty)
 12 bits object hash (for HashSets)
 5 bits compact class index
 4 bits object format (includes low bits of size in bytes)
 6 bits object size in 32-bit words
 2 bits header type (0: 3-word, 1: 2-word, 2: forbidden, 3: 1-word)

 64-bit Object header:
 32 bits all zero
 3 bits reserved for gc (mark, old, dirty)
 12 bits object hash (for HashSets)
 5 bits compact class index
 4 bits object format (includes low bits of size in bytes)
 5 bits object size in 64-bit words
 1 bit extra 4-bit of size in bytes or 32-bit words
 2 bits header type (0: 3-word, 1: 2-word, 2: forbidden, 3: 1-word)

 The object size field had to be in units of 8 bytes, and therefore three
 low-order bits are required instead of the 2 that are in the current format field.
 I simply used the vacated bit of the word size field for the 3rd low bit.
 It's Byzantine, but it works with few changes.

We have only converted the core interpreter and 3 plugins (BitBlt,
BalloonEngine and FilePlugin).  This is enough to run Squeak and test it.
Moreover we now have a lot of experience with the conversion process, and
we will write down a set of guidelines for converting the remaining
plugins. We may ask for help from the Squeak community in converting the
remaining plugins, and also image segment code.  In some case there is
clearly someone who knows the code much better than us, and for the rest,
there may simply be others who want a complete 64-bit system.

We have not done *any* of the Version 4 changes.  The first reason is that
we need to release this work soon before we get out of phase with other
Squeak progress.  The second is some discussion in our group about the
possibility of making *much* more major changes.  As soon as we get
clearer about this and confirm our intention, we'll have more to say about
it.  If were to go ahead with such a rethink, then any time spent of
Version 4 would have been wasted.

We have not done any testing beyond the point of bringing up a morphic
screen with Squeak mouse, opening a browser and running the benchmarks.
There is lots to test, and surely a number of bugs yet to be found.  The
Squeak community can be of enormous help here.

Next steps...
Ian and I are both pretty busy, but it is our hope to test the build
process some more, and put out a complete source tree with this work in
the course of the next month.  At that point others can help us to convert
the remaining plugins and to test the system as a whole.

 - Dan


>  How much bigger is the 64-bit image compared to
>  32-bit image for, say, the 3.7 gamma ?

Morphic32.image  18.3
Morphic64.image  27.7
This is an 3.7 gamma image plus VMMaker and a number of projects.

You can, of course, predict this for any image with code like...
  | nWords c headerSize nInstVars nIndexableWords |
  nWords := 0.
  Smalltalk systemNavigation allObjectsDo:
    [:x | c := x class.
    headerSize := c indexIfCompact > 0
        ifTrue: [1] ifFalse: [2].  "Very few are 3"
    nInstVars := c instSize.
    nIndexableWords := (c isVariable and: [c isBytes not])
        ifTrue: [x size] ifFalse: [0].
    nWords := nWords + headerSize + nInstVars + nIndexableWords].

  ^ (nWords - Display bits size) * 4   "expected increment in size 
  (Display not saved)"

For the above image, it predicted an increment of 10210684 a while after 
the two images were written.

>  I hope you did test " 3 + 4 " ;-)

For sure.  And it is a pretty good test, since you have to display the 
windows (with gradient fills, outline text and all), run text editing, 
bring up a menu and run the compiler to get there.  Become, GC and process 
switches happen along the way, too.  Full GC and save work also.