Monkeypatch or mixin? -- it's all in the compiler, not the runtime

And C# extension methods are really monkey-patching.

Let's look at this example -- MSFT provided events and a handler mechanism; ElegantCode writes an extension method to do the necessary null handler check, and only when I include the ElegantCode namespace does the compiler see that myEvent.Fire(...) is a valid call. The MyExtensions type has disappeared from sight -- it doesn't appear as a useful type at any point; and the extension method only appears to the compiler when an object is named explioitly -- no implicit this from inside a type named in an extension method.

In contrast, let's consider the following Scala snippet (which parallels the types in my previous post)

If we build this with the (once again abandoned) .net compiler, then decompile, we get this C# code

The underlying mechanism by which the mixin (trait) is rendered -- static methods in a separate class holding the implementation -- is the same as for C# extension methods having the same effect (see previous post). The difference is that the trait type Stringify has appeared as a significant type : it's an interface, and the static methods are on a separate synthetic class.

The difference then continues into the consumers -- a type has to opt in to the trait explicitly

or, when we decompile,

and here is the trade-off. We have a significant type (which other types can consume), and we do get implicit this, but the trait has to be inherited to become available, and delegating methods are injected to provide implicit this. (Parenthetical note -- Scala makes the use of the token Library in the type definition unambiguous; another compiler-level difference)

Extension methods are completely backwards compatible -- you can use LINQ without having to change all the collection types in your code simply by including the namespace of the new extension methods, but you don't get a new orthogonal type that expresses "those classes expressing the extensions" -- the common class is the pre-existing one that the extension method has as its this parameter.

If you want to, you can manually add a marker interface to your C# code, and have your extension methods work on instances of that interface as the this parameter, in the same way as Stringify in the Scala example appears as an interface type. That provides something closer to mix-in behaviour that can be applied across otherwise unrelated types, but that is a separate manual step, and still leaves you with monkeypatched methods -- you can't declare the mix-in methods in that interface, and you still need explicit this, even though the IL is much the same in each case.

Building a stand-alone scalalib.dll for .net convenience

Following up on the previous post -- making a single dll scala runtime goes as follows

1. Download and expand IKVM 7 into folder
2. Copy the contents of the bin subfolder into a folder .\scala-out
3. Copy scalalib.dll and forkjoin.dll into that same .\scala-out folder
4. Install ILMerge if not already available
5. Run this command (assuming powershell prompt)
   

   & 'C:\Program Files\Microsoft\ILMerge\ILMerge.exe' /closed /allowDup /t:library /targetplatform:"v4,c:\windows\Microsoft.NET\Framework\v4.0.30319" /out:scalalib.dll .\scala-out\scalalib.dll .\scala-out\forkjoin.dll  .\scala-out\IKVM.OpenJDK.Charsets.dll .\scala-out\IKVM.Runtime.JNI.dll .\scala-out\IKVM.OpenJDK.Text.dll .\scala-out\IKVM.OpenJDK.Beans.dll .\scala-out\IKVM.OpenJDK.XML.API.dll .\scala-out\IKVM.Reflection.dll .\scala-out\IKVM.Runtime.dll .\scala-out\IKVM.OpenJDK.Management.dll .\scala-out\IKVM.OpenJDK.Corba.dll .\scala-out\IKVM.OpenJDK.Core.dll

   and wait until done (takes ~3GB memory, so 64-bit systems only)
6. Observe that some of those referenced assemblies aren't actually in the IKVM subset that the scalacompiler.exe drop bundles
7. Build helloworld.exe as before and co-locate it with the new 33Mb scalalib.dll that resulted from all the grinding
8. It just works™

Hopefully the real deal will bundle something like this -- ideally also strong-named (but hey, you can do that with Mono.Cecil to rewrite that file and the assemblies you link against it).

P.S. If you have .net 4.5 on your machine, change the target platform as indicated here.

PP.S. The runtime has a dependency on the native code ikvm-native-win32-x64.dll and ikvm-native-win32-x86.dll from the IKVM download, though I'm not sure where from the depths of IKVM you may hit those.

Hello Scala.net

Belatedly spotting the March '12 update to the Scala.net story, a very brief "Hello, World"

Build with

path\to\scala-bin\scalacompiler.exe -d path\to\output -target:exe -Ystruct-dispatch:no-cache -Xassem-name HelloWorld.exe -Xassem-extdirs path\to\scala-bin -Xshow-class HelloWorld -Xassem-path "C:\Windows\Microsoft.NET\Framework\v4.0.30319\System.Windows.Forms.dll" hw.scala

where giving "." as path\to\output will put the new executable in the same folder as the scala compiler.

You now need to copy all the dlls (apart from mscorlib and the other System.* files) out of that folder into your output path folder. Then

path\to\output\HelloWorld.exe

and, suddenly:

Still slightly painful to actually perform the build, but definitely getting there. Still needs a stress-test.

Links for July

Microsoft Research's new F* language

Node.js 0.5.2 unstable branch now includes Windows

Scala.net reloaded -- still need to try it out with code that broke the old version.

A quick Scala gotcha

Wouldn't it be nice, I thought, to be able to put UI decoration as a mixin to any sort of component, like

In Scala 2.8 recent nightlies, this compiles happily -- but when you run it, the self.paintComponent(g) call just stack overflows, flipping between TiledContainer.paintComponent and Tiled.paintComponent as it goes.

At 2.7.x, it doesn't compile -- which at least prevents you getting the run-time error

error: overriding method paintComponent in class Component of type
(g: scala.swing.package.Graphics2D)Unit; method paintComponent in trait Tiled of type
(g: java.awt.Graphics2D)Unit cannot override a concrete member without a third member 
that's overridden by both (this rule is designed to prevent ``accidental overrides'')
class TiledContainer(tiledBackgroundImage : Image, constraints :Seq[String]) extends 
Form(constraints) with Tiled with HasImage {

Packaging Scala applets into one jar in NetBeans with JarJar [Updated]

Pretty much a note to self; using NetBeans that started off as 6.7.1, but has the bleeding-edge depot, and using a Scala 2.8 nightly, FWIW.

Get the jarjar tool; put it and a copy of scala-library.jar and scala-swing.jar in a lib directory under the project. Create a build.xml post-jar target to read

    <target name="-post-jar">
      <taskdef name="jarjar" classname="com.tonicsystems.jarjar.JarJarTask"
               classpath="lib/jarjar-1.0.jar"/>
      <jarjar jarfile="${dist.jar}">
        <fileset dir="${build.classes.dir}"/>
        <zipgroupfileset dir="lib" includes="scala-*.jar" />
        <keep pattern="[your.root.package].*"/>
      </jarjar>
    </target>

Clean and build, get a lot of Scala classes in your output jar file. For which there is then pack200 support to crunch further.

There is an Ant task for that, which we can add to the post-jar target in build.xml thus

    <target name="-post-jar">
      <taskdef name="jarjar" classname="com.tonicsystems.jarjar.JarJarTask"
               classpath="lib/jarjar-1.0.jar"/>
      <taskdef name="pack200"
               classname="com.sun.tools.apache.ant.pack200.Pack200Task"
               classpath="lib/Pack200Task.jar"/>
      <jarjar jarfile="${dist.jar}">
        <fileset dir="${build.classes.dir}"/>
        <zipgroupfileset dir="lib" includes="scala-*.jar" />
        <keep pattern="[your.root.package].*"/>
      </jarjar>
      <pack200 src="${dist.jar}"
               destfile="${dist.jar}.pack.gz"
               GZIPOutput="yes"
               verbose="0"
               />
    </target>

which brings in my first test case, a 2.5Mb jar down to just under 900kb.

Scala on .net — not ready for serious use...

...at least with Scala 2.7.3, and what it gets for sbaz scala-msil in any case.

Having written (well, ported) a moderate amount of Scala code over the last few weeks, towards an experiment in cross-VM code (using my tidied up version of the C# port of the to-Erlang jinterface library as the template), I thought it was time to start with the serious porting experiment.

Have you ever read the Arabian Nights (like the full thing e.g. the Penguin Classics version)?

The format goes roughly

def Quest(p : Problem) : Resolution = {
  val helper = Helper.WiseMan(p)
  helper.Ask(p) match {
     case p2 : Some[Problem] => helper.Resolve (
               Quest(p2.get()) // almost always this branch
               )
     case None => helper.Resolve()
  }
}

and so it feels with this task.

The obvious way to model the threading and mailbox code in the library is to use the scala.actors library. But that's not in predef.dll -- the source is part Java, part Scala. Noting that Scala 2.7.5 has some fixes for actors, I start with that code base.

Step 1 -- use the Java conversion assistant, build as C#, fixing up all the missing bits, then try to build the Scala code against it. It reports an abject failure to find the generated equivalent of Runnable.

Step 2 -- move the classes in SupportClass.cs into the scala.actors namespace, and rebuild. The shows up all the Java classes used in the Scala code, but also yields up the assertion that the FJTask class is broken:

4@(06 1f 09 02)
error: error while loading FJTask, type 'scala.actors.FJTask' is broken
(1f@2 in (06 1f 09 02))

-- I've not looked to what the IL means in this context. This happened even when compiling the C# with the .net 1.0 compiler.

Step 3 -- bite the bullet and port the Java code to Scala as a standalone project, with .net system calls. Fairly simple to do. Get all the syntactical errors out and the compilation throws an exception trying to do something with Object.wait()

Cannot find method class Object::wait
scope = {
  def this(): System.Object;
  protected def finalize(): Unit;
  def hashCode(): Int;
  def toString(): System.String;
  protected def MemberwiseClone(): System.Object;
  def GetType(): System.Type;
  def equals(System.Object): Boolean;
  final def ==(System.Object): Boolean;
  final def !=(System.Object): Boolean;
  final def eq(System.Object): Boolean;
  final def ne(System.Object): Boolean;
  final def synchronized(System.Object): System.Object;
  final def $isInstanceOf[T0 >: ? <: ?](): Boolean;
  final def $asInstanceOf[T0 >: ? <: ?](): T0;
  def clone(): System.Object;
  def wait(): Unit;
  def wait(Long): Unit;
  def wait(Long,Int): Unit;
  def notify(): Unit;
  def notifyAll(): Unit
}
Exception in thread "main" java.lang.Error: System.Object.wait
        at scala.tools.nsc.backend.msil.GenMSIL$BytecodeGenerator.scala$tools$nsc$backend$msil$GenMSIL$BytecodeGenerator$$getMethod(GenMSIL.scala:2373)

-- and that's after I've replaced all the instances of such a call with Monitor.Wait(), and the same for Object.notify() going to Monitor.Pulse(). But in order to get this far, I had to remove every instance of private[package-name]; and while the @volatile attribute seemed to be accepted as syntax, the emitted MSIL did not honour this decoration.

Pretty much at a dead end here.

Out of curiosity, I tried building the Erlang-bridge library I was originally intending to build for both platforms. After a little bit of tidying away calls to (java.lang.)String.format(), the compilation throws again, this time complaining about trait Projection in the context of the DigitsArray helper class (which HAS A rather than IS A Array[Int]) for immutable BigIntegers (intended to substitute for the lack of same on .net, at least this side of .net4). And this with code that passes unit tests with 90%+ coverage on the JVM.

class DigitsArray
  symbol = final class DigitsArray
  owner  = final <module> <package> <java> package platform
with methods = List(com.ravnaandtines.platform.DigitsArray.<init>, com.ravnaandt
ines.platform.DigitsArray.ShiftLeft, com.ravnaandtines.platform.DigitsArray.Shif
tRight, com.ravnaandtines.platform.DigitsArray.FreeBits, com.ravnaandtines.platf
orm.DigitsArray.getSlack, com.ravnaandtines.platform.DigitsArray.tailCountNegati
ve, com.ravnaandtines.platform.DigitsArray.GetDataUsed, com.ravnaandtines.platfo
rm.DigitsArray.ResetDataUsed, com.ravnaandtines.platform.DigitsArray.length, com
.ravnaandtines.platform.DigitsArray.update, com.ravnaandtines.platform.DigitsArr
ay.apply, com.ravnaandtines.platform.DigitsArray.DataUsed, com.ravnaandtines.pla
tform.DigitsArray.IsZero, com.ravnaandtines.platform.DigitsArray.IsNegative, com
.ravnaandtines.platform.DigitsArray.toString, com.ravnaandtines.platform.DigitsA
rray.hashCode, com.ravnaandtines.platform.DigitsArray.equals, com.ravnaandtines.
platform.DigitsArray.AsIterator, com.ravnaandtines.platform.DigitsArray.AsSeq, c
om.ravnaandtines.platform.DigitsArray.lead, com.ravnaandtines.platform.DigitsArr
ay.consistent, com.ravnaandtines.platform.DigitsArray.<init>, com.ravnaandtines.
platform.DigitsArray.<init>, com.ravnaandtines.platform.DigitsArray.<init>, com.
ravnaandtines.platform.DigitsArray.<init>, com.ravnaandtines.platform.DigitsArra
y.com$ravnaandtines$platform$DigitsArray$$data, com.ravnaandtines.platform.Digit
sArray.dataUsed_$eq, com.ravnaandtines.platform.DigitsArray.dataUsed, com.ravnaa
ndtines.platform.DigitsArray.$tag)
Exception in thread "main" java.lang.Error: trait Projection
  symbol = abstract <interface> <trait> trait Projection
  owner  = final <module> object Array with name scala.Array.Projection
        at scala.tools.nsc.backend.msil.GenMSIL$BytecodeGenerator.getType(GenMSI
L.scala:1987)

Reality check -- redo last year's cross-platform experiments ("hello world", and a Python/Scala stack). They still work just fine.

---

If the compiler worked, then it would be quite simple to port the Actors library. There would need to be some alternate code for the Debug.scala and FJTaskScheduler2.scala calls to java.lang.System; and applying some consistency to whether java.lang is explicitly or implicitly imported (simplest if always implicit, for the awkwardly inconsistent use of imports for Runnable and InterruptedException. Working around the fact that System.Threading.Thread is sealed is fairly easy.

I tried this, and the exception I get, with just stubs for the Java classes is

object Eval$2
  symbol = final case <module> object Eval$2
  owner  = final <module> object Futures
with methods = List(scala.actors.Futures.Eval$2.<init>, scala.actors.Futures.Eva
l$2.readResolve, scala.actors.Futures.Eval$2.productElement, scala.actors.Future
s.Eval$2.productArity, scala.actors.Futures.Eval$2.productPrefix, scala.actors.F
utures.Eval$2.toString, scala.actors.Futures.Eval$2.$tag)
Exception in thread "main" java.lang.RuntimeException
        at ch.epfl.lamp.compiler.msil.emit.ILGenerator.emit(ILGenerator.scala:48
6)

which refers to unmodifed code from the Scala actors library!

As it is, after the first easy side-quest to get the Actors library, the next side-quest would be the serious work of extending the compiler-to-MSIL. Alas, compilers are one of those bits of the field that I've never dabbled in before. So that'd be the next level of side-quest...

Other people ("Hi, Ivan!") seem to have gotten as far as the "Hello world!" stage too, but I've not seen any other reports of anything serious being attempted.

Which is a pity, since I quite like the language -- it's not as spiky as Erlang or as gnomic as F#, while allowing you functional style goodness, even if it is not fully a functional language in the truest sense (everything is an object, even functions).

---

Later -- test cases uploaded to my Mediafire WorksInProgress folder; the various build.bat files in scala-actors.net.7jun09.7z provoke different crashes.

Links for 9-Feb

WPF Application Quality Guide -- advice and how-to.

The basic syntax of F# -- keywords and constructs.

Recreating the button.

Interop between Java and Scala.

Book — Programming in Scala by Odersky et al.

The Scala language first crossed my consciousness over a year ago, at a stage where I was just starting to get into functional languages. But even with the various quick tour documents on the language web site, it was clear there were a lot more subtleties to this one that were not being explained.

With the stairway book, that gap has been bridged.

The book is aimed at the experienced programmer in 'C' derived imperative languages, with at least some familiarity with the Java™ language, and ideally some notion about functional programming techniques -- it is not by any stretch of the imagination a "my first programming book". For the intended audience, it is an extremely effective step-by-step guide to the features, and the syntax, of the language (this is a great contrast with e.g. Foundations of F#, which expends very little effort towards separating the accidents of the particular example from the generic syntax). While I am by no means yet fluent in the language, I feel that when I'm using it for hobby coding that I'm not just groping in the dark, but instead have a solid guide and reference to lead me.

A forward pipe (“ |> ”) operator in Scala

One of the addictive things from F# is the |> operator defined by

but I've not yet found an equivalent in Scala, even though there are a ton of useful things given by the built-in APIs -- for example, I've not had to implement the usual abstraction to manage an (array, offset, length) combination, when there's Array.slice() there already.

So, just for fun...

with unit test

Refrigerator logic — I later realise that without actually adding any variance annotations the piped-to function can be one that takes any super-class of the value type. A little more thought suggests to me that type-inference does that automagically because it is able to reconcile all inputs into the types in that single expression, and will force the value to the supertype in order to make everything match up.

Links for 12-Jan

Asynchronous fire-and-forget in C#3 (just skip to the last snippet).

Hacking anonymous type returns.

PoSh miscellany:

• Teminal Services API in win2k8R2.
• Exploring USB devices.
• capturing the console to the clipboard.
• for file transfer.
• more advanced function goodness.

volatile vs volatile, and lock-free programming.

Co-compilation of Java and Scala source.

Nice idea while it lasted…

To summarize -- if your Scala.Net project only uses .Net 1 features, then you can tweak the IL to use the .Net 2 libraries if you wish; but you can't build something that invokes full .Net 2 functionality as yet.

That means trying to build a Scala project against Silverlight assemblies is a non-starter (or at least not a trivial one) with the current build process. Using the out-of-the-box assemblies:

scalac-net.bat astroclock.scala -Xassem-path 
System.Core.dll;system.dll;System.Windows.Browser.dll;System.Windows.dll
12@(06 15 12 80 ad 02 12 80 b1 12 88 c8)
error: error while loading Control, type 'System.Windows.Controls.Control' is broken
(15@2 in (06 15 12 80 ad 02 12 80 b1 12 88 c8))
9@(06 15 12 80 b9 01 12 86 80)
error: error while loading FrameworkElement, type 'System.Windows.FrameworkElement' is broken
(15@2 in (06 15 12 80 b9 01 12 86 80))
11@(20 01 15 12 25 01 12 78 11 85 b4)
error: error while loading UIElement, type 'System.Windows.UIElement' is broken
(15@3 in (20 01 15 12 25 01 12 78 11 85 b4))
11@(06 15 12 80 ad 02 12 86 38 11 5c)
error: error while loading DependencyObject, type 'System.Windows.DependencyObject' is broken
(15@2 in (06 15 12 80 ad 02 12 86 38 11 5c))
9@(06 15 12 80 b9 01 12 86 10)
error: error while loading Application, type 'System.Windows.Application' is bro
ken
(15@2 in (06 15 12 80 b9 01 12 86 10))
astroclock.scala:90: error: System.Windows.Application does not have a constructor
class MyApp extends Application  {
                    ^
…

while rebuilding the assemblies against either .Net 2.0 or silverlight mscorlib and trying again gave the even more fundamental, but well known

scalac-net.bat astroclock.scala -Xassem-path 
System.Core.dll;system.dll;System.Windows.Browser.dll;System.Windows.dll
error: error while loading String, type 'System.String' is broken
(PEModule.getTypeDefOrRef(): TypeSpec)
one error found

So, to do this would mean upgrading the scala-net compiler to understand .Net 2.0 constructs.

---

Later:--

In constructs like (15@2 in (06 15 12 80 ad 02 12 80 b1 12 88 c8)), the 0x15 (at position 2) is the code ELEMENT_TYPE_GENERICINST, which is then followed by <an mdTypeDef metadata token> <argument Count> <arg1> ... <argN>. The type metadata turns out to be a type-spec; but the rest of it I've not yet unpicked. And then there'd be reverse-engineering the Scala MSIL decompiler...

The first culprit is DependencyObject (#22 in the typedef table), for a private field; but there are dependency properties on UIElement which suffer the same.

Scala-light?

I ordered a copy of Programming in Scala last Sunday -- and to my surprise, it arrived on my doorstep this morning. This, of course, deflected me from my F# and Silverlight activity today.

I think I shall have to try the experiment of putting Scala.Net into Silverlight, just as part of my usual "can I wedge this into that" play.

I would start with the recipe as before, in the library form; but to retro-fit a link against the Silverlight mscorlib which has the signature

rather than the mainline .Net 2.0 assembly.

What fun toys these are!