Wednesday, July 08, 2009

IronPython for build scripting

Following on from the earlier post with the snippet about generating GUIDs -- and covering a good chunk of what's been occupying me since...

I have ended up in charge of the build system for the current project at work. This started out with one framework that used a number of custom projects inside a solution to perform unit test, FxCop and coverage analysis, with a lot of magic happening in post-build steps, including direct calls to Wix command-line utilities. Another team had developed a better separated MSBuild-based system, which split out things like the analysis and installer building from the assembly-building solution. We can argue the merits of taking the unit tests out of every checking compile; but separating out the installer build does have a significant benefit in terms of cycle time for a recompilation.

Frankensteining the two together was an interesting task; and IronPython has been a valuable component of the mix. As I noted quite a while ago, the convenience of an XCOPY install on a machine with a current .net installation (any build or dev machine), and the access to the full APIs makes for a powerful tool during a build -- it's not just the fact that you get better string manipulation than a batch file, or can easily spawn off a call to source control to get a synch-level value to stamp an assembly with.

In the current context, there are a number of components being built with a common architecture, so there are plenty of opportunities to DRY the system out.

  • There are repetitive pieces of code (declaring concrete subtypes of shared base classes, to inject component specific information) which can be run by having a couple of .py files (just containing a single map initialization with common keys and component or project specific values) to define the files affected and the component-specific substitutions to make (including in some cases stable but component specific GUIDs, that can be keyed off the component specific names)
  • The shared architecture makes the MSBuild .proj file just as valid for such String.Format based substitution
  • Wix source files are just XML documents -- they can be generated programmatically from XML fragments, inspecting the solution output and filling in the appropriate entities.
  • So are Wix project files (or any MSBuild project for that matter) -- a project to build a 64-bit installer can be derived from a 32-bit installer project by a similar set of XML manipulations.

Taking the latter as an example

## many imports omitted

def LoadXml(name, namespace):
  xml = XmlDocument()
  xml.Load(name)
  manager = XmlNamespaceManager(xml.NameTable)
  manager.AddNamespace('ns', namespace)
  return (xml, manager) 

def LoadWixProject(name):
  return LoadXml(name, 'http://schemas.microsoft.com/developer/msbuild/2003')
  

sigil = '_x64'
(xml, manager) = LoadWixProject(sys.argv[1])
guid = xml.SelectSingleNode('//ns:ProjectGuid', manager)
# Use the previous example to make a new stable project GUID
guid.InnerText = '{'+GuidFromHash(guid.InnerText)+'}'

# put the 64-bit .wixobj files somewhere separate
# have to scan for all build configurations  
for intermediate in xml.SelectNodes('//ns:IntermediateOutputPath', manager):
    objdir = intermediate.InnerText
    parent = Directory.GetParent(objdir)
    base = DirectoryInfo(objdir).Name
    intermediate.InnerText = Path.Combine(parent.FullName, base + sigil).FullName + '\\'
  
# give the output a separate name
output = xml.SelectSingleNode('//ns:OutputName', manager)
output.InnerText = output.InnerText + sigil

# Require the base project to define a X64=no variable via CompilerAdditionalOptions 
for node in xml.SelectNodes('//ns:CompilerAdditionalOptions', manager):
    node.InnerText ='-dX64=yes'

# save out new project
outfile = sys.argv[1].Replace('.wixproj', '_64.wixproj')
xml.Save(outfile)

Of course, assumes that the .wxs files have Win64='$(var.X64)' attributes sprinkled appropriately.

Film — Red Cliff

Work, summer in the garden, moderating the flood around the new Eva movie and a sudden backlog of things to review have kept posting light these last couple of weeks.

So, to start the catch-up:

Red Cliff is John Woo taking Hollywood production values to the Chinese film industry, in this 2 and a half hour epic -- actually, as there are scenes in this trailer

that didn't make it to the cut I saw, there's clearly scope for an even longer Director's Cut. And even at that length, there is little fat to cut out -- only a little bit of landscapes, some self-indulgence with the CGI, and the obligatory sex scene.

This is clearly a Chinese style action movie -- you can tell that the Generals of the army are the highest level fighters, with the hit-points and special feats to match, like hurling spears and catching them on the far side they go through the target -- and the action builds up in ever bigger skirmishes, up to the final climactic battle. It's really like a live action equivalent of Senkogu Basara, only without the Engrish, and with added bullet-time whenever they had something supposedly cool going on to merit it instead. (Me, I prefer the Engrish.)

The action movie of the summer, without a doubt.

Friday, June 19, 2009

Generating strong repeatable GUIDs

In recent weeks I've had the problem of wanting to generate GUIDs that are distinct, but are repeatable -- in particular for things like COM class UIDs, and for installer component IDs, for a project using a lot of generated code.

So this is what I did in part of my build scripts

def GuidFromHash(bigstring):
  hash = MD5.Create()
  data = hash.ComputeHash(Encoding.UTF8.GetBytes(bigstring))
  hash.Clear()
  return System.Guid(data).ToString().ToUpperInvariant()

The input is the individual name of the thing I'm generating, concatenated with a constant GUID string used as a salt. The ToUpperInvariant is there to permit WiX to be run in pedantic mode.

Yes, it's exactly as vulnerable to hash collisions as MD5 -- but then my input is not maliciously aiming at such.

Later:-- or you could just use

import uuid
import clr

def GuidFromHash(explicit_salt_guid, tag):
  return str(uuid.uuid3(explicit_salt_guid, tag)).ToUpperInvariant()

but that makes the guid-for-salt explicit for you.

Links for 19-Jun

The event-based asynchronous pattern -- introduction and implementation.

Interleaving work to keep threads responsive.

PowerBoots 0.2 -- WPF in PowerShell (v1 as well as v2).

Mixing languages -- the right tool for the right job.

Is this assembly built as debug?

The elusive tail opcode in F# -- a cautionary tale about debug builds.

Monday, June 15, 2009

Links for 15-Jun

Three questions for every bug.

The single level of abstraction principle.

andand via monads -- all in C#.

Windows API code-pack -- official .net bindings for shell APIs and more.

WPF Bindings and the current culture -- a problem and a fix.

F# --

jQuery selectors tip -- a simple speed-up

Sunday, June 07, 2009

Scala on .net — not ready for serious use...

...at least with Scala 2.7.3, and what it gets for sbaz scala-msil in any case.

Having written (well, ported) a moderate amount of Scala code over the last few weeks, towards an experiment in cross-VM code (using my tidied up version of the C# port of the to-Erlang jinterface library as the template), I thought it was time to start with the serious porting experiment.

Have you ever read the Arabian Nights (like the full thing e.g. the Penguin Classics version)?

The format goes roughly

def Quest(p : Problem) : Resolution = {
  val helper = Helper.WiseMan(p)
  helper.Ask(p) match {
     case p2 : Some[Problem] => helper.Resolve (
               Quest(p2.get()) // almost always this branch
               )
     case None => helper.Resolve()
  }
}

and so it feels with this task.

The obvious way to model the threading and mailbox code in the library is to use the scala.actors library. But that's not in predef.dll -- the source is part Java, part Scala. Noting that Scala 2.7.5 has some fixes for actors, I start with that code base.

Step 1 -- use the Java conversion assistant, build as C#, fixing up all the missing bits, then try to build the Scala code against it. It reports an abject failure to find the generated equivalent of Runnable.

Step 2 -- move the classes in SupportClass.cs into the scala.actors namespace, and rebuild. The shows up all the Java classes used in the Scala code, but also yields up the assertion that the FJTask class is broken:

4@(06 1f 09 02)
error: error while loading FJTask, type 'scala.actors.FJTask' is broken
(1f@2 in (06 1f 09 02))

-- I've not looked to what the IL means in this context. This happened even when compiling the C# with the .net 1.0 compiler.

Step 3 -- bite the bullet and port the Java code to Scala as a standalone project, with .net system calls. Fairly simple to do. Get all the syntactical errors out and the compilation throws an exception trying to do something with Object.wait()

Cannot find method class Object::wait
scope = {
  def this(): System.Object;
  protected def finalize(): Unit;
  def hashCode(): Int;
  def toString(): System.String;
  protected def MemberwiseClone(): System.Object;
  def GetType(): System.Type;
  def equals(System.Object): Boolean;
  final def ==(System.Object): Boolean;
  final def !=(System.Object): Boolean;
  final def eq(System.Object): Boolean;
  final def ne(System.Object): Boolean;
  final def synchronized(System.Object): System.Object;
  final def $isInstanceOf[T0 >: ? <: ?](): Boolean;
  final def $asInstanceOf[T0 >: ? <: ?](): T0;
  def clone(): System.Object;
  def wait(): Unit;
  def wait(Long): Unit;
  def wait(Long,Int): Unit;
  def notify(): Unit;
  def notifyAll(): Unit
}
Exception in thread "main" java.lang.Error: System.Object.wait
        at scala.tools.nsc.backend.msil.GenMSIL$BytecodeGenerator.scala$tools$nsc$backend$msil$GenMSIL$BytecodeGenerator$$getMethod(GenMSIL.scala:2373)

-- and that's after I've replaced all the instances of such a call with Monitor.Wait(), and the same for Object.notify() going to Monitor.Pulse(). But in order to get this far, I had to remove every instance of private[package-name]; and while the @volatile attribute seemed to be accepted as syntax, the emitted MSIL did not honour this decoration.

Pretty much at a dead end here.

Out of curiosity, I tried building the Erlang-bridge library I was originally intending to build for both platforms. After a little bit of tidying away calls to (java.lang.)String.format(), the compilation throws again, this time complaining about trait Projection in the context of the DigitsArray helper class (which HAS A rather than IS A Array[Int]) for immutable BigIntegers (intended to substitute for the lack of same on .net, at least this side of .net4). And this with code that passes unit tests with 90%+ coverage on the JVM.

class DigitsArray
  symbol = final class DigitsArray
  owner  = final <module> <package> <java> package platform
with methods = List(com.ravnaandtines.platform.DigitsArray.<init>, com.ravnaandt
ines.platform.DigitsArray.ShiftLeft, com.ravnaandtines.platform.DigitsArray.Shif
tRight, com.ravnaandtines.platform.DigitsArray.FreeBits, com.ravnaandtines.platf
orm.DigitsArray.getSlack, com.ravnaandtines.platform.DigitsArray.tailCountNegati
ve, com.ravnaandtines.platform.DigitsArray.GetDataUsed, com.ravnaandtines.platfo
rm.DigitsArray.ResetDataUsed, com.ravnaandtines.platform.DigitsArray.length, com
.ravnaandtines.platform.DigitsArray.update, com.ravnaandtines.platform.DigitsArr
ay.apply, com.ravnaandtines.platform.DigitsArray.DataUsed, com.ravnaandtines.pla
tform.DigitsArray.IsZero, com.ravnaandtines.platform.DigitsArray.IsNegative, com
.ravnaandtines.platform.DigitsArray.toString, com.ravnaandtines.platform.DigitsA
rray.hashCode, com.ravnaandtines.platform.DigitsArray.equals, com.ravnaandtines.
platform.DigitsArray.AsIterator, com.ravnaandtines.platform.DigitsArray.AsSeq, c
om.ravnaandtines.platform.DigitsArray.lead, com.ravnaandtines.platform.DigitsArr
ay.consistent, com.ravnaandtines.platform.DigitsArray.<init>, com.ravnaandtines.
platform.DigitsArray.<init>, com.ravnaandtines.platform.DigitsArray.<init>, com.
ravnaandtines.platform.DigitsArray.<init>, com.ravnaandtines.platform.DigitsArra
y.com$ravnaandtines$platform$DigitsArray$$data, com.ravnaandtines.platform.Digit
sArray.dataUsed_$eq, com.ravnaandtines.platform.DigitsArray.dataUsed, com.ravnaa
ndtines.platform.DigitsArray.$tag)
Exception in thread "main" java.lang.Error: trait Projection
  symbol = abstract <interface> <trait> trait Projection
  owner  = final <module> object Array with name scala.Array.Projection
        at scala.tools.nsc.backend.msil.GenMSIL$BytecodeGenerator.getType(GenMSI
L.scala:1987)

Reality check -- redo last year's cross-platform experiments ("hello world", and a Python/Scala stack). They still work just fine.


If the compiler worked, then it would be quite simple to port the Actors library. There would need to be some alternate code for the Debug.scala and FJTaskScheduler2.scala calls to java.lang.System; and applying some consistency to whether java.lang is explicitly or implicitly imported (simplest if always implicit, for the awkwardly inconsistent use of imports for Runnable and InterruptedException. Working around the fact that System.Threading.Thread is sealed is fairly easy.

I tried this, and the exception I get, with just stubs for the Java classes is

object Eval$2
  symbol = final case <module> object Eval$2
  owner  = final <module> object Futures
with methods = List(scala.actors.Futures.Eval$2.<init>, scala.actors.Futures.Eva
l$2.readResolve, scala.actors.Futures.Eval$2.productElement, scala.actors.Future
s.Eval$2.productArity, scala.actors.Futures.Eval$2.productPrefix, scala.actors.F
utures.Eval$2.toString, scala.actors.Futures.Eval$2.$tag)
Exception in thread "main" java.lang.RuntimeException
        at ch.epfl.lamp.compiler.msil.emit.ILGenerator.emit(ILGenerator.scala:48
6)

which refers to unmodifed code from the Scala actors library!

As it is, after the first easy side-quest to get the Actors library, the next side-quest would be the serious work of extending the compiler-to-MSIL. Alas, compilers are one of those bits of the field that I've never dabbled in before. So that'd be the next level of side-quest...

Other people ("Hi, Ivan!") seem to have gotten as far as the "Hello world!" stage too, but I've not seen any other reports of anything serious being attempted.

Which is a pity, since I quite like the language -- it's not as spiky as Erlang or as gnomic as F#, while allowing you functional style goodness, even if it is not fully a functional language in the truest sense (everything is an object, even functions).


Later -- test cases uploaded to my Mediafire WorksInProgress folder; the various build.bat files in scala-actors.net.7jun09.7z provoke different crashes.

Saturday, May 30, 2009

Spring Holiday '09

Half term, a break in Karen's exercise classes, so Iwas free from being a taxi, so occasion to be out on the bicycle again; doing the other Norfolk itinerary from the Cycle Breaks people. The bank holiday weekend opened glorious summer, but by the Monday evening, at the base hotel, rain.


View Spring Holiday 2009 in a larger map

That meant an afternoon start, dripping trees, several places to ford, tedious portages on the one off-road bit, but at least there was one open pub, in Oxborough, at about the halfway point. Unlike in '06, I didn't even consider the Grimes Graves detour.

Wednesday was cold, very windy, and rainy, so I made a straight sprint to Gt. Bircham for the next hotel (the King's Head, the very foody place where I had lunched last time I was in the area), arriving at the same time as the luggage -- but was able to check in! Settling down to rest, alas, I found I had forgotten to return the previous room key. Aaargh!! So in a bout of ma(so)chismo, checked the map for a more direct route, and headed back, from brightening weather into renewed rain and strong headwind, having to walk a lot of the A1065 stretch, as uphill into the wind. At least it was easy coasting on the return leg.

So, 50 miles in, finally able to rest -- though a hot shower isn't quite the same as a soak in a hot bath.

Thursday, some sunshine, and a lot of sultry overcast, I did a bit of the Sandringham loop suggested for the previous day, and part of the Burnham Market/Nelson's country tour for the day. The 42 miles felt longer, after the previous day's exertions -- but finding the Gin Trap Inn at the half-way mark made it all worth while.

Friday, in brightening and hotter weather, if still often overcast, looping wide of the suggested route to keep away from the bits I'd done to death on Wednesday.

Sunday, May 24, 2009

IronRuby0.5 -- not yet running with NetBeans Ruby IDE

With IronPython done, the next step was obvious... Again, the only IronRuby/NetBeans material out there to date only concerns itself with the politics of Sun vis a vis MSFT (and red herrings like ASP.Net/MVC vs Rails). So, more rubber hitting the road to do here --

Unlike with RubyMine, NetBeans isn't satisfied with the indirection through a ruby.cmd like the one suggested in the linked article (it tells you it's not a valid Ruby interpreter); but it is clear from the file browser for ruby platforms that you need a file with name matching *ruby*.

So, copy ir.exe,ir.exe.config to ruby.exe,ruby.exe.config, and that takes as Ruby platform Ruby 1.8.6-p0 (my less than recent CRuby being Ruby 1.8.6-p111). So far so good.

Enter and run a trivial script:

puts "Hello"

get

unknown: The given path's format is not supported. (System::NotSupportedException)

even though running from the command line works just fine. Google doesn't show up any obvious correlates; and the message is one of those totally useless ones like "Could not find the path specified" which would be of some use if it actually said what the broken input actually was, rather than gaily presuming you'll know what it was.

So, anyone have any ideas?

Later:-- Using Debug, rather than Run will start the script (and by using ScottGu's IronRuby/WPF sample as the testbed script, I can be sure that it's IronRuby being launched. The process gets killed eventually, as the debugger can't attach to it, but it doesn't have the path exception.

The message given in the Run case is the same as you get from passing a mangled path-script to the ruby (née irb) executable; but there's no obvious reason for Debug to be doing anything different in this area. So I'm still baffled.

Using IronPython with NetBeans Python IDE

With the arrival of the IronPython 2.6 beta 1, with ctypes support, the IronPython 1.x only support in IronPython Studio really takes the IDE from a bit dated to seriously obsolescent.

Now, I had tried to get IronPython 2.0 to talk with NetBeans, but hadn't immediately achieved success, so, with less motivation, had put that to one side. Perhaps for similar reasons, a recipe for this wasn't out on Google already -- but now I really needed to crack the problem, I wasn't going to let that earlier abject failure put me off.

Also, since then, I'd seen someone else doing similar stuff to get IronRuby to play with an IDE, in this case, RubyMine, by tinkering with the file actually getting called by the IDE.

Well, checking my Python 2.6.1 command line with -?, and the same for IronPython 2.6 beta 1, they overlapped in almost every essential. So, I tried the experiment of copying ipy.exe,ipyw.exe to python.exe,pythonw.exe in the same C:\Program Files\IronPython 2.6 folder, and then adding the copied python.exe as a new NetBeans Python platform.

And it worked!

So, I created a new python project for platform Python 2.6.0 (as opposed to CPython 2.6.1's Python 2.6), and entered

from System import *
import sys
now = DateTime.Now

if __name__ == "__main__":
    print now
    print sys.version

and ran it, which yielded

24/05/2009 14:28:30
2.6.0 (IronPython 2.6 Beta 1 (2.6.0.10) on .NET 2.0.50727.3074)

which is of course what we wanted.

Friday, May 22, 2009

Links for 22-May

Retrieving Text from Win32 SDK's GetLastError() in C#. And obvious transpositions to other .net languages.

Also following the VS2010 beta, IronRuby 0.5 release and IronPython 2.6 beta 1 -- with CTP versions for .net 4 fx as well.

Code contracts.

Win7 -- boot to VHD : dual boot to your VMs.

Thursday, May 21, 2009

Links for 21-May

FsCheck 0.6 -- now for C# as well.

F# actors revisited.

Why Axum is relevant to distributed systems.

F# May CTP (version 1.9.6.16) for VS2008.

Concurrency features in VS2010 beta1.

False sharing and concurrent execution.

Monday, May 18, 2009

Drama

Sunshine in February

The rascal

Kamina, the male Tonkinese of our collection, has caused us a little bit of an upset these last few days. He was around the garden Thursday evening while I was weeding and watering the greenhouse, but I lost track of him as it started to go dark. Still, no worries -- he knows how to use the cat-flap...

Yoko was wailing in the morning, and was there all alone. Her brother hadn't come in (and Jemima is never going to let herself be shown how to use the flap).

No sign of pathetic little corpses on a quick scoot round the neighbourhood...

Still nothing this morning, which started to give hope that he hadn't been run over (the bodies usually get found sooner).

At lunchtime, take a little note to put up at the village shop -- to be told that someone has just come in having found a cat.

A quick phone call later -- and it is him; at the other end of the village, about half a mile away, found tangled up in a hedge (which he had had to be cut out of) -- I guess something must have chased him and he'd taken refuge that was too good.

So, a little bit scrawnier from the experience, he's on a rigorous regimen of eating (canned rather than kibble, as a treat) and sleeping (except when being fussed by the guys who came to service the hoists). And once more there's be a cat who will eat our scraps -- the others just ignored some ham rinds I put down for them as a treat.

Sunday, May 17, 2009

Retrospective — Maria-sama ga miteru Season 1

Maria-sama ga miteru is a series that was already being hailed as a classic when I first watched it, yet one that took years longer than the slighter (and later) Kasimasi, to get an official English language release; I guess perhaps because it doesn't have wacky hi-jinks, just low key slice-of-life drama.

Set in a somewhat exotic Catholic school for girls, the story centres about first-year high school student Fukuzawa Yumi, and her incorporation into the succession for the school council, the Yamayurikai, or mountain-lily council (headed by girls with the titles of White, Red and Yellow Rose as one might not expect). This is done by "adoption", under the system where each girl adopts a member of the lower years as an honorary younger sister (in a sort of den-mother arrangement). While the somewhat uncertain relationship between middle-class Yumi and Sachiko, her hopelessly aristocratic onee-sama (top left) forms the core of the season, the stories of the other Rose "families" are visited as Yumi gets to know them; including, in flashback, appearances of the previous class of Roses.

I had forgotten quite how widely the back-story spread -- including, in flashback to the previous year, appearances of the previous class of Roses (lower left). I'd also forgotten that even in the first season we got fan-service in the form of Yumi's zettai ryouiki (top right), as well as quite how boisterous Satou Sei, aka Rosa Gigantea, was.

It definitely bore the rewatching; the more so to contrast with the series just past, set one year later, with Yumi now in the position that Sachiko is here.

Anime — Astro Fighter Sunred

“This is the story of the conflict between good and evil that takes place in the city of Kawasaki.”

Q4 '08 was a strange season. I had expected to only watch Tytania; but that just lost my interest partway through. The good series were the ones that didn't actually appear so from the pre-season material. Mouryou no Hako was one; this was the other.

Sunred, a former sentai hero (but don't let on that you know about that!), lives the NEET lifestyle, sponging off his girlfriend Kayako. His opponents in the evil organization Florsheim (named after the shoe company, I believe), are a mix of monsters (usually in McJobs of their own), plushy animals, and a few actual employees, all of whom get the occasional necessary drubbing from Sunred in the local playground to keep them in line.

Meanwhile, they carry on with their domestic lives, with a strange sort of friendship between Kayako, and General Vamp, the local Florsheim leader, as they cook, clean and help each other move house. Indeed Vamp seems to be as much preoccupied with the culinary arts as he is with plotting Red's downfall and subsequent world domination.

An amusing and ironic deconstruction of the whole superhero genre in 26 half-length episodes (and the promise of another season to come).

Anime — The Daughter of Twenty Faces

I finally watched the adequate (in that they could be post-processed into English, and were terribly mistimed over the ED) subs for the last episode; which really added nothing to what had gone before.

The last episode seemed really like a filler, rather than a happily-ever-after; on an optimistic reading, it could set up a second season. Mr Forty Eyebrows makes no appearance, but we do get a "what they are doing now?" for everyone else; and a brief appearance of another character from the original material, Yoshio Kobayashi, who will eventually become Akechi's adopted son, and leader of the Detective Boys.

Oh, yes. And Engrish. Just to show that they are in England.