Distributed Memory

HTML reporting for StyleCop

2012-01-23T21:45:00.000Z

Based on the outputs from the earlier script; and using an appearance inspired by the output from the XSL report approach from codecampserver; but with the source in-lined into the report, and violation messages inserted after the affected lines. Each file (and the nested violation messages) are hidden by default, and can be expanded by clicking the file header or the source marked as violation (where the hand cursor shows):

N.B. Defined as it is within a PowerShell here-string, the jQuery script needs its $ symbols to be escaped to keep the PowerShell parser happy.

<# 
.SYNOPSIS 
    Stylecop HTML report generator. 
    
.DESCRIPTION 
    Creates *.html from *.StyleCop.xml in the current directory 
    These are reports with pretty reporting
        
.NOTES 
    File Name  : Transform-StyleCopReport.ps1 
    Requires   : PowerShell Version 2.0
#> 
function Insert-Newline([System.Xml.XmlElement] $element) {
    $break = $element.OwnerDocument.CreateSignificantWhitespace("`r`n");
    $element.AppendChild($break) | Out-Null
}

function Make-HTML([string] $file) {
    Write-Host "Processing report $file"
    $htmlfile = $file.Replace(".StyleCop.xml", ".html")
    [xml]$raw = Get-Content $file
    [xml]$report = @"
<!DOCTYPE html>
<html>
<head>
<title>StyleCop Report</title>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js"></script>
<style>
body {position: absolute; top: 0px; width: 100%;margin:0px;min-height:100%;padding:0}
body {font-family:Verdana, Helvetica, sans-serif;font-size:70%;background:#fff;color:#000;}
h1 { padding: 10px; width: 100%; background-color: #566077; color: #fff; }
h2 { margin:10px; padding: 10px; background-color: #ffc; border: #d7ce28 1px solid;  }
h3 { margin:25px; padding: 10px; background: #b9c9fe; color: #039; border: 1px solid #aabcfe; }
h3 span {float: right;}
div { margin:25px; border: #d7ce28 1px solid; }
div div { border: none; }
pre { font-family:Consolas, Inconsolata, "Lucida Console", "Courier New", monospace;font-size:100%; padding: 0; margin:0}
pre.odd { background: #eee; }
pre.even { background: #ddf; }
span.violation { background: #ff8; }
span.index { border-right : 1px solid black }
</style>
<script type="text/javascript">
`$(function(){
  // inner first, then outer
  `$('span.violation')
    .click(function(event) {
      if (this == event.target) {
        `$(this).parent().next().toggle();
      }
    return false;
  })
  .css('cursor', 'pointer')
  .click();  
  
  `$('h3')
    .click(function(event) {
      if (this == event.target) {
        `$(this).next().toggle();
      }
    return false;
  })
  .css('cursor', 'pointer')
  .click();
});
</script>

</head>

<body>
<h1>StyleCop Report</h1>

</body>

</html>
"@
    $body = (Select-Xml -Xml $report -XPath "//body").Node
    $doc = $body.OwnerDocument
    $doc.PreserveWhitespace = $true
    Insert-Newline $body
    
    $x = $doc.CreateElement("h2", "")
    $violations = (Select-Xml -Xml $raw -XPath "//Violation")
    
    $x.InnerText = "Total Violations : $($violations.Length)"
    $body.appendChild($x) | Out-Null
    Insert-Newline $body
    
    $vfiles = @($violations | % { $_.Node.Source } | Sort-Object -Unique)
    
    $vfiles | % { 
        $path = Resolve-Path $_
        $local = (Select-Xml -Xml $raw -XPath "//Violation[@Source='$_']")

        $h = $doc.CreateElement("h3", "")
        $body.AppendChild($h) | Out-Null
        Insert-Newline $body
        
        $t = $doc.CreateTextNode("SourceFile: $path")
        $span = $doc.CreateElement("span", "")
        
        $span.InnerText = "($($local.Length) violations)"
        $h.AppendChild($t)     | Out-Null
        $h.AppendChild($span)  | Out-Null
        
        $sdiv = $doc.CreateElement("div", "")
        $body.AppendChild($sdiv) | Out-Null
                    
        $source = Get-Content $path
        
        $line = 0
        $source | % {
            $line += 1
            $pre = $doc.CreateElement("pre", "")
            $num = $line.ToString()
            while ($num.Length -lt 4) { $num = " " + $num }
            $span1 = $doc.CreateElement("span", "")
            $span1.InnerText = "$num "
            $span1.SetAttribute("class", "index")
            $pre.AppendChild($span1) | Out-Null
            
            $span2 = $doc.CreateElement("span", "")
            $span2.InnerText = $_
            $pre.AppendChild($span2) | Out-Null
            
            $sdiv.AppendChild($pre) | Out-Null
            $pre.SetAttribute("class", "odd")
            if (0 -eq ($line % 2)) { $pre.SetAttribute("class", "even") }
            
            $localhere = @($local | % { $_.Node } | ? { $_.LineNumber -eq $line })
            if ($localhere.Length -gt 0) { 
                $span2.SetAttribute("class", "violation")
                $pdiv = $doc.CreateElement("div", "")
                Insert-Newline $sdiv
                $sdiv.AppendChild($pdiv) | Out-Null
                $ul = $doc.CreateElement("ul", "")
                $pdiv.AppendChild($ul) | Out-Null
                $localhere | % {
                    $li = $doc.CreateElement("li", "")
                    $li.InnerText = $_.InnerText
                    $ul.AppendChild($li) | Out-Null
                    Insert-Newline $ul
                }
            }
            
            Insert-Newline $sdiv
        }
    } # $vfiles
    
    $report.Save($htmlfile)
} # function

$files = @(dir *.StyleCop.xml)
$files | % { Make-HTML $_.FullName }

You will need to adjust the script to find the appropriate StyleCop output files from your build process, and create suitably named/located reports based on those files.

Garden rubbish

2012-01-22T20:14:00.001Z

Apart from a brief spell around last weekend, and into the start of the past week, where temperatures driving to work were in the -3C to -5C range, winter hasn't really arrived -- we've just had an extended autumn.

The cold finally did for the the antirrhinum flowers, and the last signs of life in the sunflowers, which are still serving as bird-feeders; but we still have overwintering escholzias; self-sown poached-egg plants from last summer's flowering already in bloom, primroses since before the end of last year, and now snowdrops and the first of the early crocuses.

I have at least been able to tend to a number of the tidy-up chores : pruning roses, fuchsias and apple trees; clearing out the straw from various dead annuals, and the last year's growth from the crocosmias and lemon balm.

And of course there are weeds already -- including a fine old crop of winter wheat from the porridge-like mass of spilt grain out of the bird feeder, as well as the usual stubborn perennials, misplaced self-seedings and the like. Which all means that the green bin is continually being put out full, even in the putative low season.

If it were drier, there'd even be grass clippings, as the mild wet weather is encouraging the lawn already!

Adding the missing pieces to the Standalone StyleCop script

2012-01-22T19:59:00.000Z

Rather than post the whole ~200 lines, just the salient bits so that the original can be tidied up (but without obscuring everything with obsessive error handling). Start by giving it parameters like

<# 
.SYNOPSIS 
    Stand-alone stylecop runner. 
    
.DESCRIPTION 
    Runs StyleCop over a project or a solution
        
.NOTES 
    File Name  : Run-StyleCop.ps1 
    Requires   : PowerShell Version 2.0

.PARAMETER FilesToScan

Project or Solution file

.PARAMETER StyleCopFolder

To override the heuristic looking in $env:ProgramFiles for the latest version
e.g. for StyleCop in source control

.PARAMETER OnlyBuildIntegrated

Only scan project files which include <Import Project="something with 'stylecop' in it" />
#>
param ( 
    [Parameter(Mandatory = $true)] [string] $FilesToScan,
    [string] $StyleCopFolder,
    [switch] $OnlyBuildIntegrated)

Find StyleCop in the default install location by

if (-not $StyleCopFolder) {
    $styleCopFolder =  (dir "$env:programFiles\stylecop*" | Sort-object LastWriteTimeUtc | Select-Object -Last 1).FullName
}

Get the list of projects by either just the project, or using a regex on the project file from here: http://bytes.com/topic/c-sharp/answers/483959-enumerate-projects-solution

if ($FilesToScan -like "*.csproj") {
    $projectList = ,$FilesToScan
} else {
    if ($FilesToScan -like "*.sln") {
        $solutionDir = Split-Path $FilesToScan
        $matchProjectNameRegex =
            '^Project\("(?<PROJECTTYPEGUID>.*)"\)\s*=\s* "(?<PROJECTNAME>.*)"\s*,\s*"(?<PROJECTRELATIVEPATH>.*)"\s*,\s*"(?<PROJECTGUID>.*)"$'
        # or we could filter on project guid being {FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}
        $projectList = @(Get-Content $FilesToScan | % { 
                        $matches = $null; $_ -match $matchProjectNameRegex | Out-Null; $matches.PROJECTRELATIVEPATH } | ? {
                        $_ -like "*.csproj" } | % { Join-Path $solutionDir $_ } )
    } else {
        Write-Error "$FilesToScan isn't a solution file or a C# project"
    }
}

# Check if required for a StyleCop target
if ($OnlyBuildIntegrated) {
    $projectList = @($projectList | ? {
        [xml]$content = Get-Content $_
        $imports = @($content.Project.Import)
        $flagged = (@($imports | % { $_.Project } | ? { $_ -like "*stylecop*" })).Length
        if ($flagged -eq 0) { Write-Host "Skipping project $_" }
        ($flagged -gt 0)
})}

Find a Settings.StyleCop file by looking at the project folder and up, defaulting to one in the StyleCop folder; and then scan the C# files of interest by

[xml]$content = Get-Content $projectFile
        # All C# files
        $cs = @($content.Project.ItemGroup | % { $_.Compile } | ? {$_})
        
        # Filter excluded files; get absolute paths
        $projectDir = Split-Path $_
        $cs = @($cs | ? { -not (($cs[0].ExcludeFromStyleCop -like "true") -or ($cs[0].ExcludeFromSourceAnalysis -like "true")) 
            } | % { Join-Path $projectDir $_.Include })
        
        $cs | % { $console.Core.Environment.AddSourceCode($codeproject, $_, $null) | Out-Null }

and customise the output file as

$name = (Split-Path $projectFile -Leaf).Replace(".csproj", ".StyleCop.xml")
    $outputXml = Join-Path (Get-Location) $name

Standalone StyleCop : a rough draft

2012-01-21T23:14:00.000Z

Unlike FxCop, the StyleCop tool out of the box only comes with Visual Studio and MSBuild integration; there is no stand-alone command-line tool. Still, as it's now open-source, we can look at the workings of the MSBuild task and see how to drive it directly.

Here is a proof-of-concept PowerShell script which can be used as the basis for a proper command-line tool or script; and as a preliminary "how to" to write unit tests for your rules. Well, they aren't going to be proper unit tests, since they'd have to touch the file system, but they can crawl over test source files in a dummy project in the StyleCop rule solution.

# Load StyleCop
$path = "$($env:ProgramFiles)\StyleCop 4.5\StyleCop.dll"
Add-Type -Path $path

# arguments that must be non-null, non-empty
$settings = "$($env:UserProfile)\Documents\Visual Studio 2008\Projects\TestBed\Settings.StyleCop"
$here = Get-Location
$outputFile = Join-Path $here "output.xml"

# UNTESTED : the empty array should be filled with a list of directories containing 
# StyleCop add-ins that aren't in or beneath the StyleCop install folder
$console = new-object StyleCop.StyleCopConsole -arg  @($settings, $false, $outputFile, [string[]]@(), $true)

# Usual test case #defines
$defines = [string[]]@("DEBUG", "CODE_ANALYSIS")
$configuration = new-object StyleCop.Configuration -arg (,$defines)
$project = "$($env:UserProfile)\Documents\Visual Studio 2008\Projects\TestBed\TestBed.csproj"
$codeproject = new-object StyleCop.CodeProject -Arg @($project.GetHashCode(), $project, $configuration)
[StyleCop.CodeProject[]] $projects = ,$codeproject

## TODO -- get source files from project, do AddSourceCode for each file not excluded from StyleCop
$source = "$($env:UserProfile)\Documents\Visual Studio 2008\Projects\TestBed\Class1.cs"
$console.Core.Environment.AddSourceCode($codeproject, $source, $null) | Out-Null

# These are messages suitable for MSBuild e.g. "Output: (Low) Pass 1: Class1.cs"
$outputHandler = {
    $e = $event.SourceEventArgs
    $importance = $e.Importance
    $message = $e.Output

    $colour = "Green"
    if ($importance -eq [Microsoft.Build.Framework.MessageImportance]::Low) { $colour = "Gray" }
    if ($importance -eq [Microsoft.Build.Framework.MessageImportance]::High) { $colour = "Red" }
    Write-Host "Output: ($importance) $message" -Back White -Fore $colour
}

# These are the interesting bits, which also go into the XML file e.g.
# Violation SA1200 in ...\Class1.cs (1)
# All using directives must be placed inside of the namespace.
$violationHandler = {
    $e = $event.SourceEventArgs
    if ($e.SourceCode -and $e.SourceCode.Path) {
        $file = $e.SourceCode.Path;
    } else {
 if ($e.Element -and $e.Element.Document -and $e.Element.Document.SourceCode -and $e.Element.Document.SourceCode.Path)
        {
            $file = $e.Element.Document.SourceCode.Path;
        }
    }

    $colour = "Black"
    if ($e.Warning) { $colour = "Blue" }
    Write-Host "Violation $($e.Violation.Rule.CheckId) in $file ($($e.LineNumber))`r`n`t$($e.message)" -Back White -Fore $colour
}

try {
    Register-ObjectEvent -InputObject $console -EventName OutputGenerated -SourceIdentifier "Console.OutputGenerated" -Action $outputHandler | Out-Null
    Register-ObjectEvent -InputObject $console -EventName ViolationEncountered -SourceIdentifier "Console.ViolationEncountered" -Action $violationHandler | Out-Null
    $Console.Start($projects, $true) | Out-Null
} catch [System.Exception] {
    Write-Host $_.Exception.ToString()
} finally {
    Unregister-Event "Console.OutputGenerated"
    Unregister-Event "Console.ViolationEncountered"
}

## Could dump the XML format output here, like this; or run an XSLT report generation over it
##$values = Get-Content $outputFile
##Write-Host $values$outputFile

And that's all there is to it, really.

3202.5

2011-12-31T23:51:00.001Z

That's the cumulative cycling mileage at the end of the year, passing the 3200 target I wanted to hit (~2170 over the year, plus the 90-odd on a hired bike to make over 2250 total). That's omitting those miles where I'd forgotten to put the widget back in place to count the distance; but that is unlikely to be more than about 5 miles total, and could be made up with distances pushed anyway.

I think it goes to show how dry the weather was for much of the year.

Anime -- 2011 in review (part 1)

2011-12-31T21:48:00.000Z

In total, I have watched all through just 3 series from the last year, am in the middle of two that have completed, one that hasn't and am about to start on one that has. Plus a scattering of older shows finished or in train. This post, old shows considered.

Mobile Suit Gundam

I remember when the future looked like Gundam - O'Neill colonies throughout the Earth-Moon system, but no internet. Now add brutal robot combat, the usual Tomino level of named character futile deaths, mixed with toy commercials. Like the following year's Space Runaway Ideon, this has an element of a treadmill, where each episode the enemy -- here the L2-based Principality of Zeon -- uprates its mobile suits each episode and our guys with their carrier and mobile suits continue to level up to meet them.

After ten episodes of this, the action moves ground-side, with a long meander from south-east Asia, through to the Black Sea (with a number of side trips -- "we need to reprovision with salt"; our hero has an angst attack and has wandered off with the Gundam; ...) thence to Ireland (where we meet a traditional red-headed Irish girl, with the traditional Irish name of Miharu), then across the Atlantic to a Federation base in South America, having circumnavigated the globe rather than going direct.

Then, finally, we return to space for a series of skirmishes, culminating in an escalating final battle where pretty much everything gets destroyed; but the power of Zeon is broken and all seems well (until the next series).

Even after 30 years, and with an indifferent American dub, the series holds up as more than just a piece of history. The anti-war message may be unsubtle, but there is a feeling of how even the significant players like the Gundam team are just a part of a larger concern; and the daring and charismatic recurring antagonist, Char Aznable is in turn just a minor, though equally significant, character on the Zeon side.

Catman

A quirky little flash series about a down-and-out in an alternate Canada peopled by anthropomorphic cats. It's slice of life anime done right. That means without any cute girls, school, or even cake, but with a whole lot of drinking, swearing, bowling, getting beaten up, and freezing to death on the street. And catchy ska BGM from the Planet Smashers.

Hyakko

A series I passed by back in the busy year of 2008, as another cute girls at school series like Hidamari Sketch or Sketchbook Full Colours; but which turned out to have rather more energy and enthusiasm about what it's doing. We start by having the ditzy bonde Ayumi as the first viewpoint character lost in her vast new school, then accumulate the oujo-sama Tatsuki (also lost), and the core group is then completed by the sudden eruption from an upper storey window of the tomboyish Torako, followed by her constant companion, Suzume (who are solving the navigation problem by heading in a straight line).

The series takes a little time finding its feet, and goes thematically all over the place, from out and out comedy, to amped-up melodrama; but as we learn about the quartet and the rest of the class of equally idiosyncratic types, everyone is refreshingly nice (except for the boys in the senior years, who are jerks, and Torako's family, who exist mainly for injecting drama).

Not an undying classic of the ages, but still a nice feel-good series to have watched over the holidays; especially as it was set in early summer, sunny and bright, a perfect antidote to the midwinter blues; the one exception, the April rainstorm that opens the final flashback episode, where the melancholy dawn of the first day of the school year breaks. That scene was the one where the incidental music was most noticeable, and it seemed to fit the bitter-sweet flavour of the whole episode.

Natsume Yuujinchou -- seasons 1 and 2

Another series from 2008 (and then 2009); somewhat like Mushishi, in that Natsume Takashi has the ability to see spirits that pass by normal people unobserved. However, unlike the competent adult Ginko, who has to deal with fungal or insectile mushi, the youkai that highschooler Natsume can see are people of a sort.

This unsettling aspect of his life, reactions to and conversations with the unseen, has left him shuffled from relative to relative. This time he also stumbles across some of his late grandmother's possessions -- including the eponymous book in which she inscribed the true names of many youkai; and into a shrine where a powerful spirit is bound who Natsume accidentally frees.

Having spent many years bound in the form of a lucky cat, the spirit now appears as a human-visible cat most of the time; and sets himself up as Natsume's bodyguard : no-one else is getting that book but him.

So with the rascally Nyanko-sensei to guide and protect him, he starts to find and unbind some of the youkai in the Book of Friends. The series thus starts as a simple monster of the week; but as the extended cast is gathered, it becomes as much about resolving problems amongst the youkai, and Natsume worrying about how his life is balanced between the two worlds.

While being much the typical anime protagonist most of the time, the show avoids getting Natsume into any of the usual romance clichés, without actively excluding girls from his social circle (such as it is). The fact that many of them are spirits, and one explicitly declares that she was much more interested in his grandmother, helps avoid that sort of thing.

So a quiet and delightful little series; and in the new year, I shall move onto the recently aired 3rd season, and the about to air 4th.

Film -- The Adventures Of Tintin: Secret Of The Unicorn

2011-12-28T19:55:00.000Z

Last cinema visit of the year, catching the holiday re-run of the film.

Not having read any of the source material in decades, I didn't worry about anything that might offend purists, but enjoyed the cartoony inter-war era high adventure in the style of what I did recall.

We will know that the CGI medium, especially in its 3D expression, is mature when they stop showing off gratuitous mirrors and lenses.

Writing StyleCop rules in F#

2011-12-27T01:24:00.000Z

Alas not "Writing StyleCop rules for F#" (or should that be StyleFop?), which would be nice, but a true job of work...

I don't see much need for more StyleCop rules for C# (except one to harness the FxCop spellcheck facilities to scan comments, which would have to be done via much dodgy reflection to winkle out internal types, and would anyway give issues for being tied to 32-bit native executables), so haven't tried this before. But, like mountains, it's there...

So, you have StyleCop 4.5 or 4.6 installed, and start by creating an empty class library targeted at the .net 3.5 environment, with added references to the StyleCop and StyleCop.CSharp assemblies. The bare rule class looks like

open StyleCop
open StyleCop.CSharp

namespace My.Namespace

[<SourceAnalyzer(typeof<CsParser>)>]
type MyRules() =
    inherit SourceAnalyzer()

and to go with it, add an XML file as an Embedded Resource with name My.Namespace.MyRules.xml -- first gotcha : the long name needs to be given in full because the default namespace doesn't get applied in the build like it would for C#. That can then be filled in as per the instructions in the StyleCop SDK documentation. Then just override the AnalyzeDocument method and go wild:

override self.AnalyzeDocument (document : CodeDocument) =
        let csharp = document :?> CsDocument
        self.CheckForTrailingSpaces csharp
        // add more rules here...

where I'm using the concept from the StyleCop+ rule SP2000 as an example of a new rule not yet present in the core tool.

Unlike FxCop, there is not a "one class = one rule" model here; the classes exist to group related rules, and you control how each scans the current source file of interest, and, indeed, how distinct they are in the code -- a closely related set of rules could be checked off a single scan, and in some ways StyleCop rules are more akin to the distinct named resolutions that can be defined within a single FxCop rule.

Another consequent difference from FxCop is how the object graph is visited. In FxCop, there are VisitXxx methods to override that will be called for every object of type Xxx, in an essentially stateless manner (your rule class is responsible for maintaining state between calls); here the equivalent methods are passed to a WalkXxx graph walker method as a callback, provide a mutable state object argument, and can signal through their return value whether the traversal should continue.

This approach of using a mutable object, needing down-casting, to carry inputs into each call, and contain the resultant outputs, feels a bit odd in a functional environment; fortunately it is possible to bypass this in many cases by representing the tree-walk as a seq and operating on that. For the trailing whitespace rule, where the offending lines are most easily found by scanning the raw source line by line, all we need the traversal for is to find an ICodeElement context object matching that line number. A simple walk of the code element tree looks like

let rec descent (x:CsElement) = seq {
                                                yield x
                                                if x.ChildElements <> null then
                                                    for c in x.ChildElements do
                                                        yield! descent c
                                            }

which we apply to the document RootElement and from the result find the last (so most deeply nested, and hence most constrained) item whose Location has a span that includes the line number of interest (skip while the end is before the line of interest, take while the start includes the line, reverse, get head), the line number being a value that the operation can close over.

For deployment onto machines that don't have F# installed, static linkage of the runtime via the --standalone build flag can be used rather than having to explicitly drop the F# runtime assembly into the StyleCop folder (second gotcha).

A mild Christmas, a welcome change

2011-12-26T23:58:00.000Z

Friday, with the temperature about 5C, and strong sou'westerly blowing, I cycled to work, thinking all the while that I was mad for doing so, with a weather forecast for rain, mainly heavy, in the afternoon. Unlike last year, there wasn't the need to wear anything over my spandex shorts for warmth, so that would be enough to mitigate the worst of cycling in the rain. As it turned out, when I set off home a while after 2, it was starting to sprinkle with rain; eventually getting heavy enough to want a waterproof, but not really uncomfortable at all.

Saturday, a muntjac wandered into the garden, and ate some of the remaining windfall apples, the ones too small or damaged to have harvested. The cats were not amused by this thing invading their territory, and kept a wary eye on it from a distance, and well away as it wandered off again. At least it was too big to fit through the cat-flap, unlike pigeons or the various rodents whose entrails at times have decorated the kitchen floor; more like the pheasant that was left on the doorstep, with just a clear cat-inflicted bare patch on its neck a couple of weeks ago.

Yesterday began with the exchange of presents -- where I got Vinge's latest, novel, The Children of the Sky, the long awaited sequel to 1993 Hugo winner, A Fire Upon the Deep; and Tim Worstall's book of essays on economy and the environment Chasing Rainbows, which metaphorically in text is like suikawari without the blindfold; and then lunch.

Having not thought to book until 5 weeks ago, we had a simple enough choice of places where there were still slots to be had -- lunch at the Carpenters Arms.

This was a leisurely affair, 12:30 for 1, canapés, starters, then buffet style mains, where I was unable to take up the offer of a third helping, before the cheese board, a little Xmas pudding, and coffee with petites-fours; and so home in gentle fashion by not too long after 5pm.

So today could be pure R&R, with just a little bit of pottering in the garden for some fresh air -- clearing the tomato plants out of the greenhouse, hoovering the lawn (to get rid of fallen leaves), trimming where it had straggled over the borders, and clearing the few places where plants had stopped for the winter and obligingly dried up (the marigolds are still in flower, as are the antirrhinums; not just the viburnum which I'd expect to be flowering now); leaving a little space in the green bin for more gardening later in the week.

Some of the sunflower heads that haven't already been stripped of seeds, I hung up for the birds; and the last handful of rosebuds are in a vase by the fireplace. And so the garden remains in a late-autumn limbo, until I really can't postpone pruning the roses and apple trees any more.

An approximate type-based solution to the strings problem

2011-12-14T22:22:00.001Z

As ever, neat ideas coded in Haskell can be almost but not quite ported to F#, foiled at the last moment by the fact that the .net type system cannot represent constructs that are expressed as Haskell type classes : there's always a run-time type coercion and some leakage of abstractions that have to be coded around.

One such is this idea of type-safe strings for representing structured data like XML -- which would require a little more mutually recursive and self-encapsulating type definitions than I think F# can manage. It gets close, though, and could probably be embellished a little further:

#light

open System

module TypeStrings =

    let rec foldr f z l =
        match l with
        | [] -> z
        | x::xs -> f x (foldr f z xs)

    [<AbstractClass>]
    type SafeStringBase(content : String) =
        member this.AsString with get() = content
        override this.ToString() = content
        
    // Now here #SafeStringBase really means SafeString of the corresponding 
    // concrete language type but there isn't a way to express that
    type ILanguage =
        abstract member LiteralFragment         : String -> #SafeStringBase // String is a literal language fragment
        abstract member LiteralText             : String -> #SafeStringBase // String is literal text
        abstract member NativeRepresentation    : #SafeStringBase -> String   // Gets the native-language representation
        abstract member Language                : unit -> String             // Gets the name of the language  
        abstract member Empty                   : unit -> #SafeStringBase     // creates an empty SafeString in the Language
        abstract member Add                     : #SafeStringBase -> #SafeStringBase -> #SafeStringBase
        
    type SafeString<'TLanguage when 'TLanguage :> ILanguage and 'TLanguage : (new : unit -> 'TLanguage) > (content : String) =
        inherit SafeStringBase(content)
        static let language : 'TLanguage = new 'TLanguage()
        
        // this is ugly
        static member private ToType (x:#SafeStringBase) =
            x :> SafeStringBase :?> SafeString<'TLanguage>
        
        static member Empty : SafeString<'TLanguage> =
            language.Empty() |> SafeString<'TLanguage>.ToType
        
        // takes a string that you certify as representing a fragment in the Language 
        // and returns a corresponding SafeString
        static member Fragment fragment =
            language.LiteralFragment fragment |> SafeString<'TLanguage>.ToType
        
        // takes a string that you certify as representing text and returns a 
        // corresponding SafeString in the Language
        static member Text text =
            language.LiteralText text |> SafeString<'TLanguage>.ToType
            
        static member Join (strings : seq<SafeString<'TLanguage>>) =
          let l = Seq.toList strings
          (foldr language.Add (language.Empty()) l) |> SafeString<'TLanguage>.ToType
        
        static member (+) ((self: SafeString<'TLanguage>), (other: SafeString<'TLanguage>)) : SafeString<'TLanguage> =
            (language.Add self other)  |> SafeString<'TLanguage>.ToType
            
          
    type Xml () =
      // this too is ugly
      static member private ToType (x:#SafeStringBase) =
            x :> SafeStringBase :?> SafeString<Xml>
      static member private ToHashType (x:#SafeStringBase) =
            x :> SafeStringBase :?> #SafeStringBase
            
      interface ILanguage with
        member self.LiteralFragment s = Xml.ToHashType (new SafeString<Xml>(s)) 
        member self.LiteralText s = Xml.ToHashType (new SafeString<Xml>(System.Security.SecurityElement.Escape(s)))
        member self.NativeRepresentation s =
            s.AsString.Replace("&apos;", "'").
              Replace("&quot;", "\"").Replace("&gt;", ">").
              Replace("&lt;", "<").Replace("&amp;", "&")
        member self.Language () = "XML"
        member self.Empty () = Xml.ToHashType (new SafeString<Xml>(String.Empty))
        
        // Don't think we can force this at compile time
        // The best we can do is encapsulate this operation
        member self.Add x y = 
            let left = Xml.ToType x
            let right = Xml.ToType y
            Xml.ToHashType (new SafeString<Xml>(x.AsString + y.AsString)) 
      
    [<EntryPoint>]
    let main a =
        let frag = SafeString<Xml>.Fragment "<em>wow!</em>"
        printfn "%s" frag.AsString
        let text = SafeString<Xml>.Text "ham & eggs"
        printfn "%s" text.AsString
        
        // We're safe from doing
        //
        // frag + "ham & eggs"
        //
        // Error FS0001: Type constraint mismatch. The type      string     is not compatible 
        // with type     SafeString<Xml>     
        // The type 'string' is not compatible with the type 'SafeString<Xml>'
        
        let sum = frag + text
        printfn "%s" sum.AsString
        0

Deleting files, keeping a few -- in Python

2011-12-14T19:58:00.001Z

And the same again, only as a script, just because:

import glob
import os
  
def fcompare(f1, f2):
  t1 = os.path.getmtime(f1)
  t2 = os.path.getmtime(f2)
  if t1 < t2:
    return -1
  if t1 > t2:
    return 1
  return 0

# TODO: parametrise the pattern and the number to retain
for f in sorted(glob.iglob('*.log'), cmp = fcompare, reverse = True)[3:]:
  os.remove(f)

If there wasn't that comparison function to write, it'd be terser...

Deleting files, keeping a few

2011-12-13T23:03:00.002Z

Inspired by this post, how it works in PowerShell:

Get-ChildItem *.log | Sort-Object lastWriteTimeUtc -descending | Select-Object -skip 3 | Remove-Item

will delete all but the most recent 3 files ending in ".log"; and it would be the same sort of thing in any scripting language.

Lovely weather for the time of year

2011-12-04T22:51:00.001Z

The garden today

A year ago

Yesterday was a mild and sunny day, so I took a spin out in the countryside, stopping at the Carpenters Arms for a light lunch, of pumpkin and parsnip soup and a ham and cheese toasted sandwich. It was great to enjoy the chance to get out on the bike again -- as it seemed did a number of the lycra crowd, though fortunately they don't seem to hunt in packs at this time of year.

What a contrast with a year ago, when needing to go into town, I wrapped up with long-johns, heavy gloves, multiple layers, goggles, just leaving my nose exposed -- and regretting that for the first couple of miles before getting properly warmed up.

PowerShell and GTK-Server

2011-11-28T22:15:00.001Z

Another little five-finger exercise, porting the example VBScript stdin driver to PowerShell. The only annoying thing is that the Start-Process cmdlet doesn't give you direct access to the streams, so we have to drop into .net to fire up the GTK server process.

<# 
.SYNOPSIS 
    demo to use the GTK-server using STDIN. 
    
.DESCRIPTION 
    This script is based on the VBScript example by
    Peter van Eerten published at http://www.gtk-server.org/demo-stdin.vbs.txt
        
.NOTES 
    File Name  : Try-GTKServer.ps1 
    Requires   : PowerShell Version 1.0

.PARAMETER Help

Show this help text.
#> 
param ( 
    [switch] $Help)

if ($help)
{
    Get-Help $MyInvocation.MyCommand.Definition
    return
}

# hard-coded rather than passed in as a parameter
# I'm feeling lazy tonight...
$gtkpath = "path\to\gtk-server.exe"

## As we want to get at the process stdin, stdout we can't use the Start-Process cmdlet
## $gtkserver = Start-Process -FilePath $gtkpath -ArgumentList "-stdin" -PassThru
## which only allows file I/O
$gtkinfo = new-object System.Diagnostics.ProcessStartInfo @($gtkpath, "-stdin")
$gtkinfo.UseShellExecute = $false
$gtkinfo.RedirectStandardInput = $true
$gtkinfo.RedirectStandardOutput = $true
$gtkinfo.WindowStyle = [System.Diagnostics.ProcessWindowStyle]::Hidden

$gtkserver = [System.Diagnostics.Process]::start($gtkinfo)

Function Invoke-GTK 
{
Param([string] $command)
$gtkserver.StandardInput.WriteLine($command)
$line = $gtkserver.StandardOutput.ReadLine()
if ($line -cne "ok") { $line }
}


#Define GUI
Invoke-GTK("gtk_init NULL NULL")
$win = Invoke-GTK("gtk_window_new 0")
Invoke-GTK("gtk_window_set_title $win `"PowerShell Script demo program using STDIN`"")
Invoke-GTK("gtk_widget_set_usize $win 450 400")
$table = Invoke-GTK("gtk_table_new 50 50 1")
Invoke-GTK("gtk_container_add $win $table" )
$button = Invoke-GTK("gtk_button_new_with_label Exit")
Invoke-GTK("gtk_table_attach_defaults $table $button 41 49 45 49")
$entry = Invoke-GTK("gtk_entry_new")
Invoke-GTK("gtk_table_attach_defaults $table $entry 1 40 45 49")
$text = Invoke-GTK("gtk_text_new NULL NULL")
Invoke-GTK("gtk_table_attach_defaults $table $text 1 49 8 44")

$radio1 = Invoke-GTK("gtk_radio_button_new_with_label_from_widget NULL Yes")
Invoke-GTK("gtk_table_attach_defaults $table $radio1 1 10 1 4")
$radio2 = Invoke-GTK("gtk_radio_button_new_with_label_from_widget $radio1 No")
Invoke-GTK("gtk_table_attach_defaults $table $radio2 1 10 4 7")
Invoke-GTK("gtk_widget_show_all $win" )
Invoke-GTK("gtk_widget_grab_focus $entry" )

# message/event loop
do {
    $event = Invoke-GTK("gtk_server_callback wait")
    if ($event -ceq $entry) {
    $tmp = Invoke-GTK("gtk_entry_get_text $entry" )
    if($tmp.Length -gt 1) {
            Invoke-GTK("gtk_text_insert $text NULL NULL NULL `"$tmp`n`" -1")
       }
    # Empty entry field
       Invoke-GTK("gtk_editable_delete_text $entry 0 -1")
    }
} while ($event -cne $button)

Invoke-GTK("gtk_server_exit")

Another PowerShell egg-timer (using events)

2011-11-18T22:15:00.001Z

Recording a learning exercise for how PowerShell handles events, and how to communicate into event handlers

Note that every registered event must be unregistered (otherwise running the script leaves droppings in your PowerShell session in the form of those claims on events). The events that are handled -- the timer 250ms wake-ups -- are consumed by being handled; but the one that is merely waited on remains latched after triggering the wait to release and must be explicitly cleared.

Additional AV frills would be the same as any of the polling loop timer examples out there.

<# 
.SYNOPSIS 
    This script provides simple egg-timer like functionality. 
    
.DESCRIPTION 
    This script counts down a time interval then speaks some text
        
.NOTES 
    File Name  : Run-Timer.ps1 
    Requires   : PowerShell Version 2.0
    
.PARAMETER Minutes

Delay in minutes

.PARAMETER Seconds

Additional delay in seconds

.PARAMETER Text

What to say at the end of the time

.PARAMETER Help

Show this help text.
#>
param ( 
    [ValidateRange(0,59)] [int] $Minutes,
    [ValidateRange(0,59)] [int] $Seconds,
    [string] $Text="It is time",
    [switch] $Help)

if ($help)
{
    Get-Help $MyInvocation.MyCommand.Definition
    return
}

$delay = new-object "system.timespan" @(0, $Minutes, $Seconds)
$fireAt = [System.DateTime]::UtcNow + $delay
Write-Host "Requested alarm time = `t$fireAt"

# Do the heavy lifting up front
$Voice = new-object -com SAPI.SpVoice
$Voice.Rate = -5
$timer = new-object "system.timers.timer" @(250.0)
$timer.autoreset = $true

$handler =  {
    # extract arguments (no closures here)
    $fireAt = $Event.MessageData
    $timer = $Sender
  
    # round time to wait to integer seconds and display
    $delta = $fireAt - [System.DateTime]::UtcNow
    $delta = new-object "system.timespan" @(0, $delta.Minutes, [System.Math]::Round($delta.Seconds))
    [System.Console]::Write("$delta`r")
  
    # After the time allotted, kill the timer
    if ([System.DateTime]::UtcNow -gt $fireAt) { 
        $timer.enabled = $false
        $timer.autoreset = $false
        $timer.Dispose() 
    }
}

Register-ObjectEvent -InputObject $timer -EventName Elapsed -SourceIdentifier "Timer.Elapsed" -Action $handler -MessageData $fireAt | Out-Null
Register-ObjectEvent -InputObject $timer -EventName Disposed -SourceIdentifier "Timer.Disposed"

$timer.Start() 

# Wait until done
if (-not (Wait-Event "Timer.Disposed" -Timeout 3600 )) {
    Write-Host "Timed out"
    $timer.Stop() 
    $timer.Dispose()
} else {
    Write-Host "Completed"
}
Write-Host "Actual alarm time = `t$([System.DateTime]::UtcNow)"

# Tidy event registrations and event queue
# The Elapsed events are spent in the handler
# but the one we wait on must be explicitly cleared
Remove-Event "Timer.Disposed"
Unregister-Event "Timer.Elapsed"
Unregister-Event "Timer.Disposed"

#audible alarm
$Voice.Speak($Text) | Out-Null
Write-Host "`r`nDone"

C# : The null object pattern and the poor man's Maybe Monad

2011-11-06T23:41:00.001Z

Lots of people have re-invented the Maybe monad in C#, as a simple search will show, usually as a way of bypassing null checking if statements and to be able to write code like

return Music.GetCompany("4ad.com")
            .IfNotNull(company => company.GetBand("Pixies"))
            .IfNotNull(band => band.GetMember("David"))
            .IfNotNull(member => member.Role);

But we have the tools given to us already if we observe that null has no type -- as we can see when

var x = new string[1];  // default null value
            var n = x.OfType<string>().Count();
            Console.WriteLine("Length = {0}", n);

reports a length of zero!

So our Some<T> is just a length-1 IEnumerable<T>; and None<T> an empty one -- the null object pattern, in fact.

For the price of having to specify types at each stage -- as we would anyway have had to in the past for declaring intermediate the values to test against null -- we can use existing methods to finesse the null check. Returning to the borrowed example, it would be

return new[] { Music.GetCompany("4ad.com") }
            .OfType<RecordCompany>().Select(company => company.GetBand("Pixies"))
            .OfType<Band>().Select(band => band.GetMember("David"))
            .OfType<Member>().Select(member => member.Role);

The ~~monadic return~~ monad execution operation is just FirstOrDefault(); which is fine for reference types. Value types are different in any case as we wouldn't be null checking those -- as the later steps in a chain though, stopping at the filtering to the value type and making an Any test may be preferred.

Looking at value types, switching to this example, the code looks like:

public static IEnumerable<int> Div(this int numerator, int denominator)
        {
            //return denominator == 0
            //    ? Enumerable.Empty<int>()
            //    : new[] { numerator/denominator };
            if (denominator != 0) yield return numerator/denominator;
        }

        public static IEnumerable<int> DoSomeDivision(int denominator)
        {
            return from a in 12.Div(denominator)
                   from b in a.Div(2)
                   select b;
        }
...
            var result = 
                from a in new[] { "Hello World!" }
                from b in DoSomeDivision(2)
                from c in new[] { new DateTime(2010, 1, 14) }
                select (a + " " + b.ToString() + " " + c.ToShortDateString());
            Console.WriteLine(result.Any() ? result.First() : "Nothing");

which leaks a little bit in that we have to do the ~~monadic return~~ monad execution by hand at the end, but otherwise behaves exactly as the explicit monad implementation.

3070.1

2011-10-31T23:17:00.001Z

That's the cycle odo at the end of the month, well over my stretch goal for the month of hitting of 3000. The question now is how far over 3100 by the end of the year.

And then to see what I can do by the end of next June : in order of difficulty, 2000 miles in the year, to 4000 recorded, to 1000 miles in the half year.

F# -- where has our Roslyn been?

2011-10-29T23:33:00.000+01:00

The big thing about the new Roslyn CTP is that it's a C# compiler built in C#, so accessible simply from C# code via a standardized set of APIs; including ones that permit parse tree rewriting (and hence DIY language extensions) as well as giving a more official mechanism for such things as source-level inspection (e.g. StyleCop). Indeed, the lack of any such managed language service for VB is a reason that there's no StyleCop equivalent for the other major managed language.

F# has had the managed compiler from the start -- but presumably (as I've not noticed anything being done with it by way of meta-programming or suggesting coding conventions in the manner of StyleCop) not the suitable APIs. There is a project or several here for someone to take up.

F# under the covers XIV -- Match expressions : Does my complexity look big in this?

2011-10-29T22:23:00.000+01:00

The code generation in F# in debug builds is fond of adding plenty of extra branch points that will confuse any naive computation of code complexity based on counting branch instructions at the IL level. Let us take the innocent method

let function alist [...other args...] =
    match alist with
    | h :: tail -> // some computation
    | [] -> []

The instructions that map to the match alist with context include

IL_0004: call instance class [FSharp.Core]Microsoft.FSharp.Collections.FSharpList`1<!0> class [FSharp.Core]Microsoft.FSharp.Collections.FSharpList`1<class [FSharp.Core]Microsoft.FSharp.Core.FSharpFunc`2<...`1<float64>>>::get_TailOrNull()
 IL_0009: brtrue.s IL_000d
 IL_000b: br.s IL_0044
 IL_000d: ldloc.0

where the first branch could be replaced with a brfalse.s IL_0044 and the code at offsets 0xb, 0xc replaced by nops with no change in the semantics -- but as given it adds an extra cycle to the flow of control graph, and hence false complexity.

The release build of the code retains the first two instructions, but then rather than jumping to get and return an empty list, it in-lines that fraction of the code. If we look at it in C#, the release build looks like

FSharpList<...> function(FSharpList<...> alist, ...)
{
         if (plist.TailOrNull == null)
         {
            return FSharpList<...>.Empty;
         }
        // some computation

where the debug version has a goto in that if clause that jumps to a statement that returns the empty list.

As an aside, it's also worth noting that nowhere in the .pdb information is there a map to any of the match expressions : there is never a source context which matches the | h :: tail -> or | [] ->; it's all collapsed into the match ... with fragment.

For more complex match expressions such as

match context with
          | :? Type1 as t1 -> // type 1
          | :? Type2 as t2 -> // type 2
          | :? Type3 as t3 -> // type 3 
          | other -> // something else

the debug IL organizes into trying each of the branch criteria in turn, then jumping ahead to the matched expression; this ends up with a series of redundant branches like

IL_001f: brfalse.s IL_0023
 IL_0021: br.s IL_003f
 IL_0023: ldloc.0

culminating in a final triple burst

IL_0039: brfalse.s IL_003d
 IL_003b: br.s IL_0077
 IL_003d: br.s IL_0085
 IL_003f: ldloc.1

whereas the release build treats it more as an if..elseif..elseif..else construct.

As I noted yesterday, this means that there can be a factor of at least two difference in the naively computed cyclomatic complexity of a method depending whether the release or debug build is chosen.

Computing cyclomatic complexity with PowerShell and FxCop

2011-10-28T16:55:00.000+01:00

A measure of the cyclomatic complexity of a .net method can be gauged by counting the number of IL branch instructions that do nor branch to the next instruction, and which have a distinct target from any other branch -- this is essentially the algorithm used in NDepend 1.x to make the computation. The introspection mechanism of FxCop provides enough decompilation of the IL that getting the instruction type and offset, and the target offset of a branch. An actual FxCop rule, though feasible to write, however, would be less useful than one could hope.

For one thing, FxCop itself doesn't provide any convenient way of passing parameters to custom rules (to e.g. set a trigger threshold); and for another, any analysis is likely to be run over a debug build (DEBUG and CODE_ANALYSIS variables defined there to not contaminate the released code with [SuppressMessage] annotations), which is likely to have a different underlying complexity to a release build (this is especially true in F# debug builds where there are a lot of sequences that look like

  IL_00ec: brtrue.s IL_00f0
  IL_00ee: br.s IL_00f2
  IL_00f0: br.s IL_010a
  IL_00f2: ...

which can be replaced by

  IL_00ec: brtrue IL_010a
  [enough nop opcodes]
  IL_00f2: ...

which, as it turns out, is pretty much what the release build does.

A more configurable and flexible way of performing the operation is to drive the FxCop facilities from a script -- these days I'm doing a lot of my .net scripting in PowerShell, but it could equally well be done with IronPython or similar .net scripting language. The result looks like this:

<# 
.SYNOPSIS 
    This script estimates cyclomatic complexities over a folder full of assemblies. 
    
.DESCRIPTION 
    It loads the assemblies, and then introspeccts over each method, estimating the 
    complexity as number of branch instructions which do not target the next instruction
    and which have a unique target instruction.
        
.NOTES 
    File Name  : Get-Complexity.ps1 
    Requires   : PowerShell Version 2.0 (3.0 or alternative launcher for .net 4/FxCop 10.0)
    
.PARAMETER FxCopPath

The path to the directory where FxCop has been installed

.PARAMETER AssemblyPath

The path to the folder containging the assemblies to inspect

.PARAMETER ReportLevel

If given, print out methods matching or exceeding this complexity,
otherwise return the whole analysis to the pipeline
#> 
param ( 
    [string] $FxCopPath,
    [Parameter(Mandatory = $true)] [string] $AssemblyPath,
    [int] $ReportLevel,
    [switch] $Help)

$fxcopVersion = @{ "v2"='Microsoft FxCop 1.36'; "v4"='Microsoft FxCop 10.0' }

if (-not $FxCopPath) {
  $programFiles = [System.Environment]::GetFolderPath([System.Environment+SpecialFolder]::ProgramFiles)
  $dotNetVersion = [System.Runtime.InteropServices.RuntimeEnvironment]::GetSystemVersion().Split('.') | Select-Object -First 1
  $FxCopPath = Join-Path -Path $programFiles -ChildPath ($fxcopVersion[$dotNetVersion])
}

if ($help -or (-not (Test-Path $FxCopPath -PathType Container)) -or 
              (-not (Test-Path $AssemblyPath -PathType Container)) -or
              (-not (Test-Path $(Join-Path -Path $FxCopPath -ChildPath 'FxCopSdk.dll') -PathType Leaf)))
{
    Get-Help $MyInvocation.MyCommand.Definition
    return
}

Add-Type -Path $(Join-Path -Path $FxCopPath -ChildPath 'FxCopSdk.dll')

## IL branch opcodes
$branchOpCodes =  @([Microsoft.FxCop.Sdk.OpCode]::Beq, 
    [Microsoft.FxCop.Sdk.OpCode]::Beq_S, 
    [Microsoft.FxCop.Sdk.OpCode]::Bge,
    [Microsoft.FxCop.Sdk.OpCode]::Bge_S, 
    [Microsoft.FxCop.Sdk.OpCode]::Bge_Un, 
    [Microsoft.FxCop.Sdk.OpCode]::Bge_Un_S, 
    [Microsoft.FxCop.Sdk.OpCode]::Bgt,
    [Microsoft.FxCop.Sdk.OpCode]::Bgt_S, 
    [Microsoft.FxCop.Sdk.OpCode]::Bgt_Un, 
    [Microsoft.FxCop.Sdk.OpCode]::Bgt_Un_S, 
    [Microsoft.FxCop.Sdk.OpCode]::Ble, 
    [Microsoft.FxCop.Sdk.OpCode]::Ble_S, 
    [Microsoft.FxCop.Sdk.OpCode]::Ble_Un, 
    [Microsoft.FxCop.Sdk.OpCode]::Ble_Un_S, 
    [Microsoft.FxCop.Sdk.OpCode]::Blt, 
    [Microsoft.FxCop.Sdk.OpCode]::Blt_S, 
    [Microsoft.FxCop.Sdk.OpCode]::Blt_Un, 
    [Microsoft.FxCop.Sdk.OpCode]::Blt_Un_S, 
    [Microsoft.FxCop.Sdk.OpCode]::Bne_Un, 
    [Microsoft.FxCop.Sdk.OpCode]::Bne_Un_S, 
    [Microsoft.FxCop.Sdk.OpCode]::Br,
    [Microsoft.FxCop.Sdk.OpCode]::Br_S,
    [Microsoft.FxCop.Sdk.OpCode]::Brtrue,
    [Microsoft.FxCop.Sdk.OpCode]::Brtrue_S,
    [Microsoft.FxCop.Sdk.OpCode]::Brfalse, 
    [Microsoft.FxCop.Sdk.OpCode]::Brfalse_S)

# Compute method complexity (based on the algorithm of NDepend 1.3.2 by Patrick Smacchia)
Function Compute-Complexity ([Microsoft.FxCop.Sdk.InstructionCollection] $body)
{
  $offsets = @()
  $body | ? { $branchOpCodes -contains $_.OpCode } | ? {
                $_.Value -ne ($_.Offset + 2) } | ? { ## don't count branch to next instruction
                  -not ($offsets -contains $_.Offset) }  | % {
                    $offsets += $_.Offset } | Out-Null
  $offsets.Length
}

# load assemblies (no symbols needed) and explore methods
$assemblies = dir $AssemblyPath | ? { $_.Name.EndsWith(".exe") -or $_.Name.EndsWith(".dll") }  | % { 
        $assembly = [Microsoft.FxCop.Sdk.AssemblyNode]::GetAssembly($_.FullName)
        $compilerGenerated = $assembly.GetType([Microsoft.FxCop.Sdk.Identifier]::For("System.Runtime.CompilerServices"), [Microsoft.FxCop.Sdk.Identifier]::For("CompilerGeneratedAttribute"), $true)
        New-Object PSObject -Property @{   
        Name = $_.Name;
        Types = $assembly.Types | % {
            New-Object PSObject -Property @{ 
                Name = $_.FullName;
                Methods = $_.Members | ? { $_.NodeType -eq [Microsoft.FxCop.Sdk.NodeType]::Method } | ? {
                    -not (($_.Attributes | % { $_.Type }) -contains $compilerGenerated) } | % { ## skip compiler generated code
                    New-Object PSObject -Property @{ 
                        Name = $_.GetUnmangledNameWithTypeParameters();
                        Complexity = Compute-Complexity($_.Instructions);  }}}}}}

if ($ReportLevel) { 
$assemblies | % { $aname = $_.Name
$_.Types | % { $cname = "`t$($_.Name)"
$_.Methods | ? { $_.Complexity -ge $ReportLevel } | % {
if ($aname) { Write-Host $aname; $aname = $null }
if ($cname) { Write-Host $cname; $cname = $null }
Write-Host "`t`t$_" }}}} else { $assemblies }

and running it over a set of F# code shows that the reported complexity of the release build of a method halves (or more) what is reported for a debug build. I haven't made a comparison over an significant amount of C# as yet, to see how much complexity the compiler removes between the two configurations.

End of Season Cycling

2011-10-26T19:42:00.000+01:00

View Autumn 2011 in a larger map

The autumn having settled to being bright, mild and breezy, a last chance to have a cycle break before winter -- indeed I was apparently the last customer of the season at Suffolk Cyclebreaks; going through the south of Suffolk and the north of Essex, which I haven't done as much as places further north in the region.

An early start on Sunday took me most of the way past Ipswich by shortly after midday; though having been pointed at the restaurant at Wherstead Hall, I'd aimed for that, rather than again to the Butt and Oyster at Pin Mill, taking the bridlepath past Jimmy's Farm to cut the corner, and only have to do a little way along the main road until diving off into Wherstead village. The concrete track past the church brings you to the "hall" -- a large modern building, housing a local produce market, garden centre, and restaurant serving locally sourced food -- for me, a cheeseburger with local beef and cheese, plus a bottle of the local lager.

A14 bridge over the Orwell

From there, I carried on along the coast, deciding as the hour was advancing (past 1:30, but in the low sun, feeling later) not to detour to the Butt and Oyster for more beer; but carried on, until suddenly happening upon the Baker's Arms at Harkstead, which advertised food still being served, let alone beer, so had a pint of Wherry, before carrying on. The official route as I took it last weaves a long way to fit through crossings of the Stour and the A12. This time I decided that being a quiet Sunday afternoon, the main road through Manningtree would be a simpler route, arriving at Dedham by 4pm -- and the Sun Inn was open, and serving amongst others, Arizona Bitter (Phoenix Brewery) and Wolf Bitter (Wolf Brewery). Thus fortified, I reached Milsoms (which in the last three years has realised that it makes no more sense to charge separately for wifi than it does for hot water).

Dinner was bresaola, king prawn and green mango curry, and lemon cheesecake, with lots of mineral water; and then with the windows closed in the cool night, not even the steady drone of traffic on the A12 disturbed me.

Morning, fortified with a full English, etc, I decided to avoid the rather unpleasant country between Dedham and Stoke-by-Nayland, and head rather directly to Hadleigh (where I got tricked by the cyclepath again), pushed up the long grind after crossing the A1071, then with most of the pubs mentioned in the guide being closed Mondays, struck straight cross country to Sudbury (cadging a coffee from my surprised parents), before taking a loop west on the way to the night's rest at Lavenham.

Partly guided by the OS map's PH markers, I struck through Glemsford, to find the first two pubs closed at lunchtime, and as it was close upon 2pm, I was despairing of finding a place to pause, when suddenly, the Cherry Tree Inn "Open all day for quaff and scoff". There was one other customer there when I went in, but more regulars drifted in as I committed a couple of pints of shandy, and I ended up joining in the attack on the Times' crossword.

Then when I could justify hanging around indoors on a glorious sunny afternoon no longer, made the last leg to Lavenham, parking up at the Angel. I took a bit of a wander around, including a pint at the Greyhound (where a greyhound wandered out from behind the bar and looked for attention), before dining at the hotel -- potted duck with Agen prunes, shepherds' pie with peas and spinach, then the cheeseboard; which I was ready for, having not actually felt the need for lunch.

The rain, which long range forecasts had suggested might wash the whole holiday out, went over in the night, leaving it somewhat raw but otherwise fair cycling weather in the morning. Approaching Gedding, there were some diversion signs, and I saw that the route I was planning to take was marked as road closed. But then I saw that the other end of the closure was a few yards away, and between, just a fenced off hole with more than ample room to cycle past; so I went on as I intended, until just after midday, when the Brewer's Arms called.

A burger of local red poll beef, and a pint later, and the sheet of cloud that had been hanging overhead all morning gone, I decided to amble north of the A14, then loop through the lanes back to Alder Carr farm. This time I ran into a rather more serious road closure on the approved route, but fortunately I could back off and take a separate cycle path through the new development.

And then, shortly after crossing the A1120, a sudden rattle rattle pop! as a bulge near the valve on the back wheel finally wore through and the inner-tube burst spectacularly. At least it was warm and sunny, and the road quiet, as I replaced the inner tube, moved the tyre around, and pumped up to do the last couple of miles. I suspect the tyre may be a little narrow for the wheel, as the new tube is also bulging in the same way. Still, now it'll soon be past the end of BST, and cycling to work, I can put it in for a service and get everything replaced that needs it.

With all this under the wheels, my odo is now well over the 3000 mile mark, getting me well past my stretch goal for the month.

2908.4

2011-10-18T23:35:00.002+01:00

Back at the end of June, I was hoping to have managed another 1000 miles on my bike by the end of the year.

Tonight getting home, on a clear if breezy day, the odo was showing 2908.4, or 1001.6 miles since then. The question now becomes whether I can roll it over the 3000 mile mark before the end of the month, let alone the end of the year!

Harnessing the PowerShell command line parser in .net

2011-10-18T23:24:00.000+01:00

One of the annoying niggles about the .net framework is the lack of an intrinsic command line parser -- yes, there's Mono.Options or the F# power-pack, but they're not always just there to hand. PowerShell is there on Win7 and up, and is likely to be there on machines with older OS versions in a suitably technical environment, So if you just want to whip up a simple command line tool, and you're not writing it in PowerShell, why not at least borrow its features?

First, we want to get the argument list as provided -- for this we have to get the semi-raw line via Win32 (semi-raw as the shell will already have processed escapes -- in PowerShell that means that backticks, semi-colons and double-dashes are significant):

module CommandLineParser =

    module private NativeMethods = 
        // Retrieves the command line as processed by the shell
        [<DllImport("kernel32.dll", CharSet = CharSet.Auto)>]
        extern System.IntPtr GetCommandLine();
    
    // Remove the executable name (first token) from the command-line
    let ExtractCommandLineArgumentString() =
        let ptr = NativeMethods.GetCommandLine();
        let commandLine = Marshal.PtrToStringAuto(ptr);
        
        let result = PSParser.Tokenize(commandLine)
        commandLine.Substring( (fst result).[0].EndColumn )

This uses the PowerShell tokenizer to strip out the executable name. We also want to be able to describe what the command line parameters are like, and other usage text

type Switch = {
         Name : string;
         Usage : string }
        
    type Value = {
         Name : string;
         Required : bool;
         Type : Type;
         Usage : string }
         
    type Argument = 
     | Switch of Switch
     | Value of Value
     
    type Application = {
         Name : string;
         Synopsis : string;
         Description : seq<string>;
         Arguments : seq<Argument> }

Then we can build a PowerShell function that takes the parameters we've defined, and just places them in a hash : this is the meat of the exercise

// Build a simple PowerShell function with the command line we want
    let GenerateFunction (app : Application) =
        let preamble = String.Join("\r\n", [| "<#"; 
            ".SYNOPSIS ";
            app.Synopsis;
            "   "; 
            ".DESCRIPTION \r\n"; |]) + String.Join("\r\n\r\n", Seq.toArray app.Description) + "\r\n\r\n"
        let usage = String.Join("\r\n", 
                            app.Arguments 
                            |> Seq.map(fun a -> match a with
                                                | Switch {Name=name; Usage=usage} -> (name, usage)
                                                | Value {Name=name; Usage=usage} -> (name, usage))
                            |> Seq.map (fun (name, usage) -> String.Format(System.Globalization.CultureInfo.InvariantCulture,
                                                                           ".PARAMETER {0}\r\n\r\n{1}\r\n",
                                                                           name, usage))
                            |> Seq.toArray) + "\r\n"                                                     
        let decl = String.Join("\r\n", [| "#>" ;
            "function Parse-CommandLine";
            "{"; 
            "  param ( " |]) + "\r\n"
            
        let prefix (isMandatory : bool) =
            if isMandatory then
                "[Parameter(Mandatory=$true)]"
            else
                String.Empty
         
        let parameters = String.Join(",\r\n", 
                            app.Arguments 
                            |> Seq.map(fun a -> match a with
                                                | Switch {Name=name} -> (name, "switch", false)
                                                | Value {Name=name; Type=``type``; Required=required} -> (name, ``type``.FullName, required))
                            |> Seq.map (fun (name, ``type``, required) -> String.Format(System.Globalization.CultureInfo.InvariantCulture,
                                                                                   "    {2} [{1}] ${0}",
                                                                                   name, ``type``,
                                                                                   prefix required))
                            |> Seq.toArray) + "\r\n"                                               
        let body = String.Join("\r\n", [| ")";
            "   @{" |]) + "\r\n"
        let assignment = String.Join(";\r\n", 
                            app.Arguments 
                            |> Seq.map(fun a -> match a with
                                                | Switch {Name=name} -> name
                                                | Value {Name=name} -> name)
                            |> Seq.map (fun name -> String.Format(System.Globalization.CultureInfo.InvariantCulture,
                                                                                   "    {0}=${0}", name))
                            |> Seq.toArray) + "\r\n"                                               
        let coda =  String.Join("\r\n", [| "'$args' = $args" ;
            "    }";
            "}" |])
            
        preamble + usage + decl + parameters + body + assignment + coda

For producing a usage description, we can ask PowerShell for one, flatten it to a sequence of strings, and then strip out the unwanted PowerShell-isms

let Runspace = new RunspaceInvoke()
    
    let RunScript (script: string) =
        Runspace.Invoke(script)
        
    let Usage (app : Application) =
        let text = (GenerateFunction app) + 
                    "\r\n$a = Get-Help Parse-CommandLine -full\r\n" +
                    "$a.Name = \"" + app.Name + "\"\r\n" + // replace the fake name
                    "$a.details.name = \"" + app.Name + "\"\r\n" + // replace the fake name
                    "$a.Syntax.SyntaxItem.Name = \"" + app.Name + "\"\r\n" + // replace the fake name
                    "$a | Out-String -Stream"
       
        let result = RunScript(text)
        (result |> Seq.map (fun p -> p.BaseObject)).OfType<string>() 
        |> Seq.takeWhile (fun l -> l.Trim() <> "<CommonParameters>")
        |> Seq.map (fun l -> l.Replace("[<CommonParameters>]", String.Empty))
        |> Seq.filter (fun l -> not <| l.Trim().StartsWith("Accept pipeline input?", StringComparison.Ordinal))
        |> Seq.filter (fun l -> not <| l.Trim().StartsWith("Accept wildcard characters?", StringComparison.Ordinal))
        |> Seq.filter (fun l -> not <| l.Trim().StartsWith("Default value", StringComparison.Ordinal))
        |> Seq.iter (fun x -> Console.WriteLine("{0}", x))

Finally, we can parse a supplied command-line, writing a failure reason and usage info if things fail

let Parse (app : Application) (commandline : string) =
        let text = (GenerateFunction app) + 
                      "\r\nParse-CommandLine " + commandline + "\r\n"
        try 
            let r = RunScript(text)
            r.[0].BaseObject :?> System.Collections.Hashtable
        with
        | :? ParameterBindingException as fault ->
            Console.WriteLine("{0}", fault.Message)
            Usage app
            reraise()

A simple example driver would be

[<EntryPoint>]
    let main a =
        let arg = Value {Name = "ValueArg"; Required = false; Type = typeof<string>; Usage = "This is the value usage"}
        let arg2 = Value {Name = "NextArg"; Required = false; Type = typeof<int>; Usage = "This is the 2nd value usage"}
        let switch = Switch {Name = "SwitchArg"; Usage = "This is the switch usage" }
        let here = Assembly.GetExecutingAssembly().Location
        let h = (new FileInfo(here)).Name
        
        
        let app = { Name = h; Synopsis = "This is the synopsis" ;
                    Description = [| "this command does stuff"; "wonderful stuff" |] ;
                    Arguments = [| arg; arg2; switch |] }
        
        let hash = CommandLineParser.Parse app <| CommandLineParser.ExtractCommandLineArgumentString()
        hash.Keys |> Seq.cast 
        |> Seq.iter (fun k -> printfn "%A -> %A" k (hash.Item(k)))
        
        0

There are some quirks -- Mandatory positional parameters and arbitrary nameless options don't mix : the current example program if given a command line "-Val hello -N 23 -S a b c" will yield

"$args" -> [|"a"; "b"; "c"|]
"ValueArg" -> "hello"
"SwitchArg" -> True
"NextArg" -> 23

Make NextArg mandatory, and instead it goes

A positional parameter cannot be found that accepts argument 'a'.

NAME
    ConsoleApplication1.exe

SYNOPSIS
    This is the synopsis

SYNTAX
    ConsoleApplication1.exe [[-ValueArg] <string>] [-NextArg] <int32> [-SwitchArg]...

So, perhaps not industrial strength; but suited to gentle use when there's nothing else to hand.

Links for 9-Oct

2011-10-09T20:56:00.001+01:00

The world has moved on, have you? Xml APIs you should avoid using. (Microsoft XML Team's WebLog) -- tl;dr = XmlText* classes should be retired in favour of objects minted by Xml*.Create() factories.

5 Awesome Free jQuery Plugins -- toast, sliders, tickers...

What you see vs what you get : matching source and IL with Mono.Cecil and PowerShell

2011-10-09T10:29:00.002+01:00

Following on from yesterday, looking at how Mono.Cecil pairs up debug information in the .pdb file with just the sequence points the IL -- which follows the approach taken by coverage tools such as dot-net-coverage. The code is much the same as before:

<# 
.SYNOPSIS 
    This script inspects a method by introspection. 
    
.DESCRIPTION 
    This script uses Mono.Cecil introspection to tie instructions with source.
    It loads the referenced assemblies into the current AppDomain, so should be run in a
    separate process to your main script
        
.NOTES 
    File Name  : Inspect-MethodWithMonoCecil.ps1 
    Requires   : PowerShell Version 2.0 (3.0 or alternative launcher for .net 4/FxCop 10.0)
    
.EXAMPLE
 .\Inspect-MethodWithMonoCecil.ps1 -CecilPath ...\Mono.Cecil -AssemblyPath ...\Tinesware.Rules.dll 
         -ClassName Tinesware.Rules.SpellCheckRule 
         -MethodName VisitField | ? { $_.Context -ne "<None>" } | Out-GridView  
           
.PARAMETER CecilPath

The path to the directory where Mono.Cecil.*.dll have been installed

.PARAMETER AssemblyPath

The path to the assembly to inspect

.PARAMETER ClassName

The name of the class containg the Method.  Nested types are separated with + e.g.
  Project.Namespace.Type+NestedType

.PARAMETER MethodName

The name of the method to inspect
#> 
param ( 
    [Parameter(Mandatory = $true)] [string] $CecilPath,
    [Parameter(Mandatory = $true)] [string] $AssemblyPath,
    [Parameter(Mandatory = $true)] [string] $ClassName,
    [Parameter(Mandatory = $true)] [string] $MethodName,
    [switch] $Help)
    

if ($help -or (-not (Test-Path $CecilPath -PathType Container)) -or 
              (-not (Test-Path $AssemblyPath -PathType Leaf)) -or
              (-not (Test-Path $(Join-Path -Path $CecilPath -ChildPath 'Mono.Cecil.dll') -PathType Leaf)))
{
    Get-Help $MyInvocation.MyCommand.Definition
    return
}

Add-Type -Path $(Join-Path -Path $CecilPath -ChildPath 'Mono.Cecil.dll')
Add-Type -Path $(Join-Path -Path $CecilPath -ChildPath 'Mono.Cecil.Pdb.dll')

# load assembly with debug symbols
# We could reflect into $assembly to get the built-in symbol path, which is messy
# In most case it's safe to assume the files are co-located
$assembly = [Mono.Cecil.AssemblyDefinition]::ReadAssembly($AssemblyPath)
$symbolPath = [System.IO.Path]::ChangeExtension($assembly.MainModule.FullyQualifiedName, ".pdb")
$provider = New-Object Mono.Cecil.Pdb.PdbReaderProvider
$reader = $provider.GetSymbolReader($assembly.MainModule, $symbolPath)
$assembly.MainModule.ReadSymbols($reader)

# load the type
$splitnames = @($ClassName.Split('+'))

$types = @($assembly.Modules | % { $_.Types })
$type = $types | ? { $_.FullName -eq $splitnames[0] } | Select-Object -First 1

# handle nested types TODO
if (($splitnames.Length -gt 1) -and $type) {
  $splitnames[1..$($splitnames.Length-1)] | % {
    $name = $_
    if ($type) { $type = $type.NestedTypes | ? { $_.Name -eq $name } | Select-Object -First 1 }
    }
}

if (-not $type) {
  Write-Error "Class named '$ClassName' not found" -Category ObjectNotFound -TargetObject $ClassName
  return
}  

# load the method, being ready to resolve overloads
$methods = @($type.Methods  | ? { $_.Name -eq $MethodName })
if (-not $methods) {
  Write-Error "Method named '$MethodName' not found" -Category ObjectNotFound -TargetObject "$ClassName.$MethodName"
  return
}

# User choice adapted from http://blogs.technet.com/b/jamesone/archive/2009/06/24/how-to-get-user-input-more-nicely-in-powershell.aspx
Function Select-Item 
{
Param(   [String]$Caption="Please make a selection", 
         [String]$Message="Choices are presented below", 
         [String[]]$choiceList, 
         [int]$default=0) 
   $choicedesc = New-Object System.Collections.ObjectModel.Collection[System.Management.Automation.Host.ChoiceDescription] 
   $choiceList | foreach  { $choicedesc.Add((New-Object "System.Management.Automation.Host.ChoiceDescription" -ArgumentList $_))} 
   $Host.ui.PromptForChoice($caption, $message, $choicedesc, $default) 
}  

$index = 0
if ($methods.Length -gt 1) {
    $choices = @($methods | % { 
        $parameters = @($_.Parameters | % { "$($_.ParameterType) $($_.Name)" } )
        $parameters = $parameters -join ", "
        [string] "&$($index)$([char]8)$($_.Name)($Parameters)" 
        $index += 1
    })
    $index = Select-Item "Choose which overload to use" "The various overloads are:" $choices 0
}

# Overloads resolved, get the source
$method = $methods[$index]
$source = $method.Body.Instructions | % { $_.SequencePoint } | ? { $_ } | % { $_.Document } | ? { $_ } | % { $_.Url } | ? { $_ } | Select-Object -First 1
$sourceListing = Get-Content $source 

# Source context to code snippet
Function Extract-Snippet ([Mono.Cecil.Cil.SequencePoint] $context)
{
    if ( (-not $context.Document) -or (-not $context.Document.Url) -or (-not $context.EndLine) -or ($context.StartLine -ge $sourceListing.Length)) {
      "<None>"
    } else {
      $snippet = @($sourceListing[$($context.StartLine-1)..$($context.EndLine-1)])
      $snippet[0] = (" " * ($context.StartColumn-1)) + $snippet[0].SubString($context.StartColumn-1)
      $snippet[-1] = $snippet[-1].SubString(0, $context.EndColumn-1)
    }
}

## Report Instructions
$method.Body.Instructions | % {
    $sequencePoint = @($_.SequencePoint.StartLine, $_.SequencePoint.StartColumn, $_.SequencePoint.EndLine, $_.SequencePoint.EndColumn)
    if (-not $_.SequencePoint.EndLine) { $sequencePoint = "null" }
    if ($_.SequencePoint.StartLine -ge $sourceListing.Length) { $sequencePoint = "0xfeefee" }
    New-Object PSObject -Property @{   
        OpCode = $_.OpCode;
        Context = [System.String]::Join("`r`n", (Extract-Snippet $_.SequencePoint));
        Details = $_;
        SequencePoint = $sequencePoint
    }
}

with the addition of the line/column data in SequencePoint ("null" if the values are not given, "0xfeefee" for the well-known compiler generated fake line number).

Launching PowerShell 2.0 into .net4 via F# Interactive or IronPython 2.7

2011-10-08T21:24:00.003+01:00

With .net4 being 18 months old, and the Win8/PowerShell 3.0 being both in CTP and only for Win7+, it's a drag having to do things with .net 4 code when you really could do them faster and better in PowerShell. So, why not bootstrap ourselves from out of the box PowerShell 2.0 into the .net 4 world with a little help from another scripting language which is .net 4 aware? This saves messing about with environment activation variables, centralized registry settings or application config files, and takes advantage of the fact that PowerShell, like other scripting languages, can be hosted by another .net process.

The recipe is based around Bart de Smet's Option 2 – Hosting Windows PowerShell yourself.

Via F# interactive:

<# 
.SYNOPSIS 
    This script launches a basic PowerShell console running in .net v4
    
.DESCRIPTION 
    This script launches a basic PowerShell console running in .net v4, using F# interactive 
    for .net 4 as an intermediate platform
        
.NOTES 
    File Name  : Launch-PowerShellInNet4ViaFSharpInteractive.ps1
    Requires   : PowerShell Version 2.0
#> 

$programFiles = [System.Environment]::GetFolderPath([System.Environment+SpecialFolder]::ProgramFiles)

$fsharp4 = Join-Path -Path $programFiles -childPath 'Microsoft F#\v4.0\Fsi.exe'
$automation = Join-Path -Path $programFiles -childPath 'Reference Assemblies\Microsoft\WindowsPowerShell\v1.0\System.Management.Automation.dll'
$powershell = Join-Path -Path $programFiles -childPath 'Reference Assemblies\Microsoft\WindowsPowerShell\v1.0\Microsoft.PowerShell.ConsoleHost.dll'

$filename = [System.IO.Path]::GetRandomFileName()
$tempfile = Join-Path -path $env:temp -childPath "$filename"
$launcher = @"
open System
open System.Management.Automation.Runspaces
open Microsoft.PowerShell

let config = RunspaceConfiguration.Create()
ConsoleShell.Start(config,
                          "Windows PowerShell - Hosted on CLR v4\nCopyright (C) 2010 Microsoft Corporation. All rights reserved.",
                          String.Empty,
                          [| "-NoExit" ; "-Command" ; "[System.Console]::Title = \"PowerShell on .net \" + [System.Runtime.InteropServices.RuntimeEnvironment]::GetSystemVersion()" |]);;
"@
$source = $tempfile + ".fsx"

Set-Content -Path $source -Value $launcher

Write-Host "launching..."
$process = Start-Process -FilePath $fsharp4 -ArgumentList "`"-r:$automation`" `"-r:$powershell`" `"$source`"" -Wait
Remove-Item -Force $source

Via IronPython 2.7 (augmented to take command line arguments via the -Options parameter):

<# 
.SYNOPSIS 
    This script launches a basic PowerShell console running in .net v4
    
.DESCRIPTION 
    This script launches a basic PowerShell console running in .net v4, using IronPython 2.7 
    for .net 4 as an intermediate platform
        
.NOTES 
    File Name  : Launch-PowerShellInNet4ViaIronPython27.ps1
    Requires   : PowerShell Version 2.0
    
.PARAMETER Options
    Array of command line arguments passed to the inner PowerShell
#> 

param ( [string[]]$Options = @("-NoExit" , "-Command" , '[System.Console]::Title = "PowerShell on .net " + [System.Runtime.InteropServices.RuntimeEnvironment]::GetSystemVersion()') )


$programFiles = [System.Environment]::GetFolderPath([System.Environment+SpecialFolder]::ProgramFiles)

$ipy = Join-Path -Path $programFiles -childPath 'IronPython 2.7\ipy.exe'
$poshpath = Join-Path -Path $programFiles -childPath 'Reference Assemblies\Microsoft\WindowsPowerShell\v1.0'

$filename = [System.IO.Path]::GetRandomFileName()
$tempfile = Join-Path -path $env:temp -childPath "$filename"
$list = $Options -join "', '"
$launcher = @"
import clr
import sys

sys.path.append(r'$poshpath')
clr.AddReferenceToFile('System.Management.Automation.dll')
clr.AddReferenceToFile('Microsoft.PowerShell.ConsoleHost.dll')

from System import *
from System.Management.Automation.Runspaces import *
from Microsoft.PowerShell import *

config = RunspaceConfiguration.Create()
ConsoleShell.Start(config,
                   "Windows PowerShell - Hosted on CLR v4\nCopyright (C) 2010 Microsoft Corporation. All rights reserved.",
                   String.Empty,
                   Array[str]([ '$list' ]))
"@

$source = $tempfile + ".py"

Set-Content -Path $source -Value $launcher

Write-Host "launching..."
$process = Start-Process -FilePath $ipy -ArgumentList "`"$source`"" -Wait
Remove-Item -Force $source

Since we can't pipe into Start-Process, the inner script is written to a temporary file; and then the initial PowerShell process waits so that it can safely delete it. If you would rather, it's reasonable to save the script to a constant .fsx or .py, replace $source with the path to that file, then the -Wait and Remove-Item -Force $source can be removed.

Replacing the fixed command line arguments for the F# interactive launched PowerShell with ones input to the script as per IronPython follows the same pattern. In either case, care has to be taken with getting your quotation marks right, and the IPy example is rather rough-and-ready. Besides, doing appropriate quotation parsing and escaping would obscure the real point of the exercise.

Note: Launching PowerShell directly from the .net4 command line launches PowerShell in the plain old .net 2 configuration; it is, alas, not sticky.

Note: These scripts are intended to give you a .net4 based interactive session; an alternative approach aimed at running individual commands can be found on Jason Stangroome's GitHub.

What you see vs what you get : matching source and IL with FxCop and PowerShell

2011-10-08T19:04:00.001+01:00

The mapping between the code you write and the object code that gets generated is not always a simple one; F# code in particular is heavily restructured in the compilation phase. So when trying to perform code analysis to detect source level artefacts in the compiled code, some guide is useful to navigate between the two representations.

So, here is a PowerShell script to do just that, looking at the IL instructions or logical statements for a method and the corresponding source code as given via the .pdb debugging information -- PowerShell rather than F# interactive because the former has a much nicer user interaction model, especially when asking the user to resolve which overloaded method name was really intended.

<# 
.SYNOPSIS 
    This script inspects a method by introspection. 
    
.DESCRIPTION 
    This script uses FxCop introspection to tie statements or instructions with source.
    It loads the referenced assemblies into the current AppDomain, so should be run in a
    separate process to your main script
        
.NOTES 
    File Name  : Inspect-Method.ps1 
    Requires   : PowerShell Version 2.0 (3.0 or alternative launcher for .net 4/FxCop 10.0)
    
.PARAMETER FxCopPath

The path to the directory where FxCop has been installed

.PARAMETER AssemblyPath

The path to the assembly to inspect

.PARAMETER ClassName

The name of the class containg the Method.  Nested types are separated with + e.g.
  Project.Namespace.Type+NestedType

.PARAMETER MethodName

The name of the method to inspect

.PARAMETER Instructions

By default, the output matches FxCop statements to source; with this switch, matches
IL instructions to the source.
#> 
param ( 
    [string] $FxCopPath,
    [Parameter(Mandatory = $true)] [string] $AssemblyPath,
    [Parameter(Mandatory = $true)] [string] $ClassName,
    [Parameter(Mandatory = $true)] [string] $MethodName,
    [switch] $Instructions,
    [switch] $Help)

$fxcopVersion = @{ "v2"='Microsoft FxCop 1.36'; "v4"='Microsoft FxCop 10.0' }

if (-not $FxCopPath) {
  $programFiles = [System.Environment]::GetFolderPath([System.Environment+SpecialFolder]::ProgramFiles)
  $dotNetVersion = [System.Runtime.InteropServices.RuntimeEnvironment]::GetSystemVersion().Split('.') | Select-Object -First 1
  $FxCopPath = Join-Path -Path $programFiles -ChildPath ($fxcopVersion[$dotNetVersion])
}

if ($help -or (-not (Test-Path $FxCopPath -PathType Container)) -or 
              (-not (Test-Path $AssemblyPath -PathType Leaf)) -or
              (-not (Test-Path $(Join-Path -Path $FxCopPath -ChildPath 'FxCopSdk.dll') -PathType Leaf)))
{
    Get-Help $MyInvocation.MyCommand.Definition
    return
}

# load assembly with debug symbols
Add-Type -Path $(Join-Path -Path $FxCopPath -ChildPath 'FxCopSdk.dll')
$assembly = [Microsoft.FxCop.Sdk.AssemblyNode]::GetAssembly($AssemblyPath, $null, $false, $true, $true, $false)

# load the type
$splitnames = @($ClassName.Split('+'))
$type = $assembly.Types | ? { $_.FullName -eq $splitnames[0] } | Select-Object -First 1

# handle nested types
if (($splitnames.Length -gt 1) -and $type) {
  $splitnames[1..$($splitnames.Length-1)] | % {
    $name = $_
    if ($type) { $type = $type.NestedTypes | ? { $_.Name.Name -eq $name } | Select-Object -First 1 }
    }
}

if (-not $type) {
  Write-Error "Class named '$ClassName' not found" -Category ObjectNotFound -TargetObject $ClassName
  return
}  

# load the method, being ready to resolve overloads
$methods = @($type.Members  | ? { ($_.Name.Name -eq $MethodName) -and ($_.NodeType -eq [Microsoft.FxCop.Sdk.NodeType]::Method) })
if (-not $methods) {
  Write-Error "Method named '$MethodName' not found" -Category ObjectNotFound -TargetObject "$ClassName.$MethodName"
  return
}

# User choice adapted from http://blogs.technet.com/b/jamesone/archive/2009/06/24/how-to-get-user-input-more-nicely-in-powershell.aspx
Function Select-Item 
{
Param(   [String]$Caption="Please make a selection", 
         [String]$Message="Choices are presented below", 
         [String[]]$choiceList, 
         [int]$default=0) 
   $choicedesc = New-Object System.Collections.ObjectModel.Collection[System.Management.Automation.Host.ChoiceDescription] 
   $choiceList | foreach  { $choicedesc.Add((New-Object "System.Management.Automation.Host.ChoiceDescription" -ArgumentList $_))} 
   $Host.ui.PromptForChoice($caption, $message, $choicedesc, $default) 
}  

$index = 0
if ($methods.Length -gt 1) {
    $choices = @($methods | % { 
        [string] "&$($index)$([char]8)$($_.Name.Name)($($_.Parameters))" 
        $index += 1
    })
    $index = Select-Item "Choose which overload to use" "The various overloads are:" $choices 0
}

# Overloads resolved, get the source
$method = $methods[$index]
$source = $method.Instructions | % { $_.SourceContext.FileName } | ? { $_ } | Select-Object -First 1
$sourceListing = Get-Content $source 

# Source context to code snippet
Function Extract-Snippet ([Microsoft.FxCop.Sdk.SourceContext] $context, [string] $prefix="")
{
    if ( (-not $context.FileName) -or (-not $context.EndLine) -or ($context.StartLine -ge $sourceListing.Length)) {
      "<None>"
    } else {
      $snippet = @($sourceListing[$($context.StartLine-1)..$($context.EndLine-1)])
      $snippet[0] = (" " * ($context.StartColumn-1)) + $snippet[0].SubString($context.StartColumn-1)
      $snippet[-1] = $snippet[-1].SubString(0, $context.EndColumn-1)
      $snippet | % { $prefix + $_ }
    }
}

## Report Instructions if requested, then exit
if ($Instructions) {
    $method.Instructions | % {
        New-Object PSObject -Property @{   
            OpCode = $_.OpCode;
            Context = [System.String]::Join("`r`n", (Extract-Snippet $_.SourceContext));
            Details = $_
        }
    }
    return
}

# Expand a StatementCollection recursively, indicating block structure with levels of ">")
Function Expand-Block([Microsoft.FxCop.Sdk.StatementCollection] $block, [string] $prefix="") {
  $block | % {
    New-Object PSObject -Property @{   
        NodeType = $_.NodeType;
        Context = [System.String]::Join("`r`n", (Extract-Snippet $_.SourceContext $prefix));
        Details = $_
        }
    ## recurse into blocks
    if ($_.NodeType -eq [Microsoft.FxCop.Sdk.NodeType]::Block) { Expand-Block $_.Statements ($prefix+">") }
    }
}

## Report Statements
Expand-Block $method.Body.Statements

As the script writes rich objects in the successful output to pipeline, it can be used as a starting point for further analysis; or the output can be piped into e.g. Out-GridView for inspection.

Summer at last

2011-10-02T11:39:00.001+01:00

So, we get the hottest days of the year at the start of October : it's like summer was put on back to front, going from warm, to April showers, to autumn, to high summer.

After clocking up to ~2705 miles in Q3 -- 800 miles in the quarter -- I took a ~40 mile spin on the bike yesterday, with lunch at the Carpenter's Arms in Gt. Wilbraham, then carrying on out past Six Mile Bottom, West Wratting, Balsham and back. With the lack of wind, and low humidity, it was a beguilingly easy ride (apart from the long grinds up out of Six Mile Bottom towards Brinkley; and from West Wratting to Balsham). So easy, that it wasn't obvious quite how much I was sweating, and thus how much I needed to rehydrate, until I finished a 75cl bottle of Lucozade Sport in pretty much one gulp.

In the evening, it was still so warm that we sat out in the garden until well after sunset, something that we never had weather for over the nominal summer.

Recent reading

2011-09-22T22:05:00.003+01:00

Kethani by Eric Brown

Aliens send artefacts to Earth to enslave the human race, as told through the perspective of the inhabitants of a Yorkshire village with an unusually high rate of cancer and suicide. Nobody seems to notice what the aliens are doing, not even the author.

Cosy British SF as it is done at the start of the new century.

Yellow Blue Tibia by Adam Roberts

Cold War nostalgia in a tale of an SF writer given an unusual assignment by Stalin, and then being told to forget everything, which works for the next 30-some years. Spends a long time setting up the mood, then infodumps a not-quite-resolution.

Rocket Girls and Rocket Girls: The Last Planet by Housuke Norjiri

The two light novels (each slim enough to finish during one bathtime) that were the original source of 2007's delightful anime series, now available in English. The anime was a very faithful adaptation of the source material -- just omitting the by then very dated visit to Mir at the end of the first book, and segueing that mission into the one at the start of the second.

Ouroboros Wave by Jyouji Hayashi

A series of almost puzzle-driven hard-SF shorts scattered across several decades in a fairly standard Earth vs space colonists setting -- a very modern execution of something that was at one time almost the backbone of the genre.

A summer evening at last

2011-09-02T22:25:00.001+01:00

After work, along with a couple of colleagues, I cycled to the Toft Beer Festival, where we sat at the table outside in the warm evening sunshine and enjoyed the beer, until it came sunset, and time to be heading home.

For the first time this year, the warmth lingered as the light went. It was pleasant cycling in shorts and T-shirt; and even just a few minutes ago, going out to drop tea-bags and apple cores into the compost bin, wearing nothing but a towel around my waist, the paving was warm underfoot, and the air mild long after sunset.

I had in the past come to expect the light to be going before the warmth through July, but not so in recent years. Now that is a most unusual thing to happen until an Indian Summer day like today.

Cambridge Film Festival

2011-08-28T22:49:00.006+01:00

I pretty much started this blog as as a home for "My Cambridge Film Festival film reviews", however much my blogging has mutated over the years. And this weekend, the programme for this year's CFF is out -- and even more than last year, I'm feeling underwhelmed by the torrent of gritty and edgy features, right-on documentaries, and the usual run of "there's adultery in it, so it must be artistic". It's like it was all set out with a ground rule of "No fun allowed, SRS BZNS ONLY". Oh, and they're exhuming and showing the totally abysmal Silent Running (down there near the nadir of my list of the worst movies I've watched). Twice.

I suppose I could go watch The Seventh Seal and The Day the Earth Caught Fire again; but apart from being in two minds about this locally produced film, Dimensions (which could be another Time Crimes, but could equally well be artsy tosh with a few bits of stage dressing like non-genre writers generate when thinking they're doing SF), the new material singularly failed to catch my eye.

Late summer cycling break

2011-08-25T18:48:00.004+01:00

Same sort of gig as last year, only with different weather.

View August 2011 in a larger map

This time I arrived in Great Bircham about 10:30 on a beautiful summer's day, and headed out on the long run I'd meant to do last year, without the interruption by squall lines.

With the later start, I reached Little Walsingham after the first course of packed lunch (stopping for a pint at the Black Lion), and then kept going east as far as Binham (another pint at the Chequers Inn -- which, despite the name, does not seem to offer accommodation, but would have made a good base for another time), then looping back.

Second course of lunch -- cheesy rolls and a beefsteak tomato from the greenhouse -- at 15:00, partway along the coastal cyclepath, before heading for the first time through the grounds of Holkham Hall from the north. I'd steered clear of it, having previously taken other paths from the south which didn't make good cycling.

This side, the paths were properly made, easy going up and over the rolling terrain, then rolling downhill most of the way to Burnham Market; then the grind up the Ringstead road was where my legs started to tire, but still back to base not long after 17:00, having done about 48 miles.

Tuesday was wet, so read, did internet and generally rested up, with a little stroll around the local area late afternoon.

Wednesday was dry, if cool and generally cloudy, so another clockwise loop through the area (with one misstep when the path I'd meant to take was less obvious on the ground than the one I didn't), through lanes I'd mainly not taken before, with the goal of lunch at the Gin Trap Inn at Ringstead.

After their substantial club sandwich and chocolate mousse, I didn't want to take the rather punishing direct route to Sedgeford, but instead detoured out to Heacham, before finally ambling back (with a number of pauses where a full stomach made it more comfortable to get off and push rather than grind an uphill).

So that was about 28 miles; but between the end-of-day fade on Monday, and the push while digesting, my logged average speed went down from a steady 11 mph as it had been all summer, to only 10.8 :(

Powershell -- higher order functions with arguments that take arguments

2011-08-15T22:35:00.004+01:00

Dead simple example based on this:

# Create a function that accepts a function that takes arguments
Function Test ([scriptblock]$code, $name) {

    & $code "$name+$name" | % {
        New-Object PSObject -Property @{
            Name = $name
            Prop = "**$_**"
        }
    }
}    

# Create a function to call
Function  Set-Lower ([string]$x) {
  $x.ToLowerInvariant()
}   

# Create a suitable script block to wrap the function and take a parameter
$b = [scriptblock] { param([string]$Target = "PARAM") Set-Lower $Target }            

# Invoke the Test function passing it a scriptblock and argument
Test $b Junku

which results in

Name                                                  Prop                                                 
----                                                  ----                                                 
Junku                                                 **junku+junku**

Also -- reminder to self : ShowUI -- WPF in PowerShell

Trivia

2011-08-15T21:36:00.002+01:00

At end of day 11-Aug the car was reading 3474 miles; at the hour that I picked it up last year, 3485. Subtracting the 14 that were on the clock when I picked it up, 3460 or 3471 miles -- a lot less than in previous years.

Sunday evening, the bike was showing 2280 miles; 373 miles in the first 6 weeks of the new year, which is in keeping with last year's rate

The new guided busway cyclepath is a great new way into the town centre avoiding traffic, even though I have to do a mile and a bit on bridlepath or detour twice that distance to get onto the cyclepath feeding to it. If only there were paved spurs to the neighbouring villages. And even though the last hundred yards or so at the station end aren't finished yet.

In the garden, the plums have been in full flood for a couple of weeks -- much earlier than the usual Bank Holiday peak I usually plan for; indeed everything has been early, ever since the early onset of last winter.

What do you get when you iterate null?

2011-08-05T23:19:00.006+01:00

In most cases, an exception; but in PowerShell you get $null.

Suppose I have folders with 0, 1 and more than 1 file in (called empty, full and two, say) and I want to enumerate the names of the files contained in each, with some general purpose code.

That should be no problem --

function Get-FileNames($dir) {
  $files = Get-ChildItem $dir
  foreach ($file in $files) {
    Split-Path -Leaf $file
  }
}

and let's write a little reporting function

function Get-Report($arg) {
  $arg
  $arg.GetType().FullName
  $arg.Length
  $arg | % { Write-Host "++$_++" }
  Write-Host "-----"
}

When we report on the folders in descending order of contained files we get

test (2).txt
test.txt
System.Object[]
2
++test (2).txt++
++test.txt++
-----
test.txt
System.String
8
++test.txt++
-----
Split-Path : Cannot bind argument to parameter 'Path' because it is null.
At C:\Users\Steve\Documents\scratch\Untitled1.ps1:6 char:15
+     Split-Path <<<<  -Leaf $file
    + CategoryInfo          : InvalidData: (:) [Split-Path], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.Split 
   PathCommand
 
You cannot call a method on a null-valued expression.
At C:\Users\Steve\Documents\scratch\Untitled1.ps1:12 char:15
+   $arg.GetType <<<< ().FullName
    + CategoryInfo          : InvalidOperation: (GetType:String) [], RuntimeException
    + FullyQualifiedErrorId : InvokeMethodOnNull
 
++++

Not happy.

The first error is what you get when you work on the foreach over $null, yielding $null; and calling code has to special-case the single item return -- getting the .Length property of the return is not the number of files! For that replace $arg.Length by $arg | Measure-Object | % {write-host $_.Count} .

So try this

function Get-Array($list)
{
  if (-not $list) {
    ,(@())
  } else {
    if ($list -is [array]) {
      $list
    } else {
      ,(,$list)
    }
  }
}

function Get-FileNamesX($dir)
{
  $files = Get-Array(Get-ChildItem $dir)
  foreach ($file in $files)
  {
    Split-Path -Leaf $file
  }
}

function Get-FileNames2($dir)
{
  Get-Array(Get-FileNamesX($dir))
}

where we fight PowerShell's fragmentation of sequences all the way. Now we get what we expected

test (2).txt
test.txt
System.Object[]
2
++test (2).txt++
++test.txt++
-----
test.txt
System.Object[]
1
++test.txt++
-----
System.Object[]
0
-----

Oh, yeah -- the more PowerShell way of doing all this:

function Get-FileNames($dir)
{
  Get-ChildItem $dir | % {
    Split-Path -Leaf $_
  }
}

function Get-Report {
  PARAM ($InputObject) 
  END {
  $arg = @($InputObject + $Input)
  $arg
  $arg.GetType().FullName
  $arg.Length
  $arg | Measure-Object | % {write-host "Measure = $($_.Count)"}
  $arg | % { Write-Host "++$_++" }
  Write-Host "-----"
 }
}

Get-FileNames(".\two") | Get-Report
Get-FileNames(".\full") | Get-Report
Get-FileNames(".\empty") | Get-Report

Links for July

2011-07-31T23:40:00.004+01:00

Microsoft Research's new F* language

Node.js 0.5.2 unstable branch now includes Windows

Scala.net reloaded -- still need to try it out with code that broke the old version.

July wrap-up

2011-07-31T23:33:00.003+01:00

Well, it's been a busy month, hence quiet on the blogging front, starting out with:

A summer holiday

While Karen was in hospital last year, I noticed a flyer for Vitalise respite care holidays in amongst various papers, and thought "Let's do this next summer."

So we did.

We chose the first week of July at the Netley Waterside centre as a part of the country neither of us had been to for many years, went through all the paperwork about care requirements and doctor's reports, and crossed our fingers about how well we would get on.

In the end, it was wonderful.

It started off a bit fraught, though; as on the outward journey on a too hot and bright day for comfortable travel, as we were going past Tring, I realised that amongst the things I'd forgotten to pack was my wallet. So I just unloaded Karen and all the other bits, and turned around, to drive down again the next day with all the stuff we've forgotten (doing in two days way more miles than I drive in the average month).

Going back home in the heat was not much fun, especially when combined with stumbling into special traffic arrangements for the Henley Regatta just as everyone was heading home for the evening. But at home, I was cheered to get a phone call from Karen, who was enjoying being able to choose her own bed time.

Second time around I was actually able to find a parking space (as holidays run Saturday to Saturday, the first time there were arrivals and departures jostling for space), freshen up and join the Sunday chill-out.

The centre is really on the water's edge

with the gardens sloping down to the path that runs along the sea wall, though the path is only wheelchair accessible in one direction.

Monday, we opted out of the excursion to the Sea Life Center, and after I'd scouted out the area, we went along to the local (the Prince Consort) and took advantage of the ramp up to the beer garden at back. to just sit, enjoy a pint and the summer weather, the two of us on holiday together for the first time in years.

But that was the end of the good weather; cooler, windier and wet was the order of the day for the rest of the week. Tuesday's expedition to the Hawk Conservancy Trust at Andover was dampened by the rain arriving as we did, and continuing at varying strength until we left; but despite the weather, the display was spectacular, with sometimes a dozen birds in the air.

We did manage to miss the rain for Wednesday's visit to Marwell Zoo, which we only manage to see less than half of; Thursday started wet, so the planned trip to Winchester became a shopping center trip, which we opted out of; and the Friday was an official at-leisure day where we took the opportunity to slope off to the pub (using the industrial strength ramp for the front step) for lunch.

And so, home.

For the holiday, there was the inevitable fact of separate hospital beds for sleeping; the food all home cooked, if unadventurous, with a very early 18:00 dinner time, with breakfast being as much a main meal as that in the evening.

Karen really enjoyed the comparative freedom to choose a bed time -- including at well gone midnight on Thursday, when it was party night.

And for me -- well, when I helped Karen with chasing the last morsels of a meal around a plate, I was told off, that one of the volunteers should do that while I enjoyed a break!

The volunteers are a key part of the operation -- a cheerful, enthusiastic and cosmopolitan crowd from all over the globe, working for bed and board, to just take over much of the step-and-fetch level care, from simply getting a cup of coffee, to pushing manual chairs while accompanying otherwise unaccompanied guests.

So, we'll be back, though next time, assuming I don't have to do the trip 3 times in quick succession, we may opt out of scheduled trips and do our own at our own pace (without being caught up in the half hour or more loading and unloading process).

Trivia -- at just before 10pm on Monday the 4th, I observed a 60-hour old nail-paring crescent moon setting over the port, the sun having set 40 minutes before.

30 years on

And today has been out Pearl wedding. Doesn't time fly!

Logistics being what they are, celebration was lunch, at the recently refurbished Carpenters Arms at Gt. Wilbraham. Being a Sunday, that meant roast, but by being so standard, allowed a like-for-like comparison : big chunks of roast, huge selection of veg, and the best Yorkshire pudding I've eaten in years.

Film — Detective Dee and the Mystery of the Phantom Flame

I this up picked up now it's out here on DVD, having heard of it some months ago.

The film is pure hokum wire-fu, and about as faithful to any of the source material as most movies (about on the level of the wartime Sherlock Holmes vs. Nazi spy films) but entertaining enough for an evening's viewing.

While van Gulik is in the writing credits according to imdb, I saw nothing of what he did by way of exhuming Chinese detective stories (except possibly the fact of popularising Judge Dee in the west) made it into the movie.

Film — Arietty

Seeing this after lunch today, we caught the UK theatrical dub (separate UK production team mentioned in the end credits, and some English regional accents -- most noticeably for the rat catchers -- amongst the cast).

It's been a long long time since I read the original (I have slightly more memories of Adrift and Aloft than of Afield), but the adaptation felt right, though one wonders if it wasn't the speech about the Borrowers dying out that attracted Miyazaki, something to slide in as a soapboxing scene.

Overall, the best thing out of Ghibli since The Cat Returns.

PowerShell strings, ints and comparisons

2011-07-25T22:45:00.002+01:00

I hit this doing a little bit of playing around over the weekend, porting some old BASIC progams into PowerShell.

Most of the time treating strings of digits as numbers "just works":

>$line = "5, 22.5"
>$bits = $line.split(",")
>$bits
5
 22.5
>$ratio = $bits[1]/$bits[0]
>$ratio
4.5
>write-host ($bits[0] -eq 5)
True

but at times it all goes horribly wrong

>write-host ($bits[0] -gt 30)
True
>write-host ($bits[0] -gt 60)
False
>write-host ($bits[0] -gt 50)
False
>write-host ($bits[0] -gt 49)
True
>write-host ($bits[0] -eq 50)
False

with the ASCII value of '5' being 53, we see that these comparisons are being done lexically.

What is needed in these circumstances is an explicit coercion some place down the line; like:

>write-host ([int]$bits[0] -gt 45)
False

Select-Xml and nested XPath evaluations

2011-06-30T22:21:00.004+01:00

Recall that

>$s = Select-Xml -Xml $x -XPath '//Rule'
>$s | foreach-object { $_.Node ; Select-Xml $_.Node -XPath 'descendant::BooleanProperty' |  foreach-object { $_.Node } }

worked but

>$s = Select-Xml -Xml $x -XPath '//Rule'
>$s | foreach-object { $_.Node ; Select-Xml $_ -XPath 'descendant::BooleanProperty' |  foreach-object { $_.Node } }

failed with error messages showing that the current iterated value has an XML string representation.

An alternative fix is to explicitly cast the subject of the inner XPath selection to XML, thus

>$s = Select-Xml -Xml $x -XPath '//Rule'
>>$s | foreach-object { $_.Node ; Select-Xml ([xml] $_) -XPath 'descendant::BooleanProperty' |  foreach-object { $_.Node } }

though surprisingly

>$s = Select-Xml -Xml $x -XPath '//Rule'
>>$s | foreach-object { [xml] $_ ; Select-Xml ([xml] $_) -XPath 'descendant::BooleanProperty' |  foreach-object { $_.Node } }

fails to yield any results from the inner search; and I've not so far determined why. Replacing the final $_.Node by [xml] $_ of course fails because False is not an XML document.

Target Achieved

2011-06-30T22:13:00.002+01:00

Arriving home this evening, the bike odometer read 1906.8 miles since I installed it at the end of the first week in July last year. Adding the 97 miles I logged on my spring holiday on a hired bike, that pushes me over the 2000 miles mark. for the year (with 8 days to spare).

So, onwards to 3000 miles by year's end.

More PowerShell, XML and XPath -- Select-Xml and multi-node selections

2011-06-22T18:14:00.011+01:00

Let's start with the same document as before. Now at PowerShell 2, we can also do

>$n = Select-Xml -Xml $x -XPath "//Rule[@Name='FileHeaderMustShowCopyright']//BooleanProperty[@Name='Enabled']"
>$n

Node                                    Path                                    Pattern
----                                    ----                                    -------
BooleanProperty                         InputStream                             //Rule[@Name='FileHeaderMustShowCopy...

>$n.GetType()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     False    SelectXmlInfo                            System.Object
>$n.Node

Name                                                        #text
----                                                        -----
Enabled                                                     False

>$n.Node.ParentNode.ParentNode

Name                                                        RuleSettings
----                                                        ------------
FileHeaderMustShowCopyright                                 RuleSettings

which avoids the explicit call to the .net infrastructure, but wraps up the actual content we got in the previous example as the Node field inside an object. This is not so much of a problem when picking a single node, but when you want to do the equivalent of SelectNodes nested (e.g. for each Rule, do something with each setting) some disassembly is required in order to perform the inner selection:

>$s = Select-Xml -Xml $x -XPath '//Rule'
>$s | foreach-object { $_.Node ; Select-Xml $_.Node -XPath 'descendant::BooleanProperty' |  foreach-object { $_.Node } }

Name                                                        RuleSettings
----                                                        ------------
FileMustHaveHeader                                          RuleSettings
Enabled
FileHeaderMustShowCopyright                                 RuleSettings
Enabled

compare

>$nn = $x.SelectNodes("//Rule")
>$nn | ForEach-Object { $_ ; $_.SelectNodes("descendant::BooleanProperty") }

Name                                                        RuleSettings
----                                                        ------------
FileMustHaveHeader                                          RuleSettings
Enabled
FileHeaderMustShowCopyright                                 RuleSettings
Enabled

It is, however, a bit of an oversight that the Select-Xml cmdlet doesn't consume SelectXmlInfo objects through the usual type conversion mechanisms inside the cmdlet infrastructure:

>$s | foreach-object { Select-Xml -xml $_ -XPath 'descendant::BooleanProperty' |  foreach-object { $_.Node } }
Select-Xml : Cannot bind parameter 'Xml'. Cannot convert the "<RuleSettings><BooleanProperty Name="Enabled">False</Bool
eanProperty></RuleSettings>" value of type "Microsoft.PowerShell.Commands.SelectXmlInfo" to type "System.Xml.XmlNode".
At line:1 char:38
+ $s | foreach-object { Select-Xml -xml <<<<  $_ -XPath '//BooleanProperty' |  foreach-object { $_.Node } }
    + CategoryInfo          : InvalidArgument: (:) [Select-Xml], ParameterBindingException
    + FullyQualifiedErrorId : CannotConvertArgumentNoMessage,Microsoft.PowerShell.Commands.SelectXmlCommand

Select-Xml : Cannot bind parameter 'Xml'. Cannot convert the "<RuleSettings><BooleanProperty Name="Enabled">False</Bool
eanProperty></RuleSettings>" value of type "Microsoft.PowerShell.Commands.SelectXmlInfo" to type "System.Xml.XmlNode".
At line:1 char:38
+ $s | foreach-object { Select-Xml -xml <<<<  $_ -XPath '//BooleanProperty' |  foreach-object { $_.Node } }
    + CategoryInfo          : InvalidArgument: (:) [Select-Xml], ParameterBindingException
    + FullyQualifiedErrorId : CannotConvertArgumentNoMessage,Microsoft.PowerShell.Commands.SelectXmlCommand

but once you know there's that gotcha, it can be worked around.

PowerShell, XML and XPath

2011-06-19T22:21:00.004+01:00

Some of the time, PowerShell makes picking apart an XML file nice and easy by providing properties on the XML object matching nodes; but inevitably you get to a point where there are more than one node of a given type as the child of the one you are at. So then it's time to break out the XPath.

Take a sample XML file (this one being a StyleCop settings file, being to hand, and complicated enough to be interesting):

<StyleCopSettings Version="4.3">
  <Analyzers>
    <Analyzer AnalyzerId="Microsoft.StyleCop.CSharp.DocumentationRules">
      <Rules>
        <Rule Name="FileMustHaveHeader">
          <RuleSettings>
            <BooleanProperty Name="Enabled">False</BooleanProperty>
          </RuleSettings>
        </Rule>
        <Rule Name="FileHeaderMustShowCopyright">
          <RuleSettings>
            <BooleanProperty Name="Enabled">False</BooleanProperty>
          </RuleSettings>
        </Rule>
      </Rules>
      <AnalyzerSettings />
    </Analyzer>
  </Analyzers>
</StyleCopSettings>

And we want to extract the "FileHeaderMustShowCopyright" enabled property. So we can do something like this:

>[xml] $x = Get-Content "Settings.StyleCop"
>$x.stylecopsettings.Analyzers.Analyzer.Rules.SelectSingleNode("Rule[@Name='FileHeaderMustShowCopyright']").RuleSettings.SelectSingleNode("BooleanProperty[@Name='Enabled']")

Name                                                        #text
----                                                        -----
Enabled                                                     False


>

using PowerShell to navigate where there is no ambiguity, and XPath at each for in the road; or

>$x.SelectSingleNode("//Rule[@Name='FileHeaderMustShowCopyright']").RuleSettings.SelectSingleNode("BooleanProperty[@Name='Enabled']")

>$x.SelectSingleNode("//Rule[@Name='FileHeaderMustShowCopyright']").SelectSingleNode("//BooleanProperty[@Name='Enabled']")

or simply

>$x.SelectSingleNode("//Rule[@Name='FileHeaderMustShowCopyright']//BooleanProperty[@Name='Enabled']")

depending how many of the steps above where we want to end up are of interest.

Splitting Pascal-Cased names in PowerShell

2011-06-17T22:50:00.004+01:00

Something I needed to whip up recently, now slightly more polished. Starting with Jon Galloway's Splitting Camel Case with RegEx, and transposing to PowerShell's built-in regex syntax, the core operation becomes like

("ZomgWtfBbq" -creplace '[A-Z]', ' $&').Trim().Split($null)

where we have to use the case sensitive -creplace operator to be able to insert spaces before capital letters. This outputs

Zomg
Wtf
Bbq

Wrapping it up into a function that handles a general batch of strings or things looks like this

function PascalSplit {
  $args | ForEach-Object {
    if ($_ -is [array]) { 
      $_ | ForEach-Object { PascalSplit $_ }
    } else {
      ($_.ToString() -creplace '[A-Z]', ' $&').Trim().Split($null)
    }  
  }
}

so we can do this:

>PascalSplit @("ZomgWtfBbq", "ShazBot")
Zomg
Wtf
Bbq
Shaz
Bot
>

flattening the string array as we go (there is probably a better way to do that, but this works).

Following the link to James "poprhythm" Kolpack's enhancement, we can upgrade the simple function with his look-behind extensions to handle TLAs in names:

function PascalSplit {
  $args | ForEach-Object {
    if ($_ -is [array]) { 
      $_ | ForEach-Object { PascalSplit $_ }
    } else {
      ($_.ToString() -creplace '(?<!^)([A-Z][a-z]|(?<=[a-z])[A-Z])', ' $&').Split($null)
    }  
  }
}

which we can test with

>PascalSplit LearnWCFInSixEasyMonths
Learn
WCF
In
Six
Easy
Months
>$a = PascalSplit @("ZomgWtfBbq", "ShazBot", "LearnWCFInSixEasyMonths")
>$a
Zomg
Wtf
Bbq
Shaz
Bot
Learn
WCF
In
Six
Easy
Months
>$a.Length
11
>

And he still thinks "Referism" would work

2011-06-04T08:24:00.005+01:00

One of those rare political rants...

Referism -- the quaint notion that a yearly budget referendum would do anything to check the ruling class.

The EU Referendum blog has started pushing this idea recently -- and in a massive act of cognitive dissonance does so while reporting how, when faced with a tight budget, the pols will protect their core function of providing sinecures to the nomenklatura, and trim the fluff at the margins (like providing services for the hoi polloi). And if that had been voted on, then next year, after parading the bleeding stumps, they'd come back for more ("or we'll kill this kitten/shut this library/sack these nurses..."). And that's assuming they don't take the get-out clause of Parliament being unable to bind itself and pass whatever emergency legislation to bypass.

The key problems are that politicians lie, money is fungible, and any stated budget breakdowns would be worth exactly as much as manifesto commitments are under universal suffrage. Oh, and it cuts out all the persiflage in the way of the root problem with democracy:

"A democracy cannot survive as a permanent form of government. It can last only until its citizens discover that they can vote themselves largesse from the public treasury."

Even a radical replacement like having MPs and Senators chosen by lot from the populace -- a way of taking power from those who most obviously actively seek it -- would still leave the problem of the entrenched Civil Service and the quangocrats.

CodeDOM and poor man's lambdas -- Part 2

2011-05-31T22:54:00.005+01:00

Unfortunately there is one major sticking point in getting the generation to work for F# -- local variables are declared as let mutable rather than as a ref type which means that the necessary closure cannot be made; this is rather stronger than the lack of support for CodeDefaultValueExpression, which can be fudged around, or for nested types (which just become mutually recursive), though with up-front decision as to the language to generate (rather than making the choice following the expression tree), we could replace problematic elements with snippets.

That aside, the main operation looks like this, similar to the previous examples

let GenerateWrapper (target:Type) =

  // Create a compile unit with a namespace
  let compileUnit = new CodeCompileUnit()
  let scope = new CodeNamespace("Derived")
  compileUnit.Namespaces.Add( scope ) |> ignore
  
  // Create a GeneratedCodeAttribute expression
  let generated = typeof<GeneratedCodeAttribute>
  let ga = new CodeAttributeDeclaration(TypeRef generated)
  ga.Arguments.Add(new CodeAttributeArgument( new CodePrimitiveExpression("Generator"))) |> ignore
  ga.Arguments.Add(new CodeAttributeArgument( new CodePrimitiveExpression("0.0.0.0"))) |> ignore
  
  // Create the wrapper type and attribute it
  let wrapper = new CodeTypeDeclaration(target.Name + "Wrapper")
  wrapper.TypeAttributes <- TypeAttributes.Public ||| TypeAttributes.Sealed
  wrapper.CustomAttributes.Add(ga) |> ignore
  scope.Types.Add(wrapper) |> ignore
  
  // Add a field for the wrapped object
  let this = new CodeMemberField(TypeRef target, "this")
  wrapper.Members.Add(this) |> ignore
  
  // Create a simple constructor to set the wrapped object
  let construct = new CodeConstructor()
  construct.Parameters.Add(new CodeParameterDeclarationExpression(TypeRef target, target.Name)) |> ignore
  let self = new CodeThisReferenceExpression()
  let atThis = new CodeFieldReferenceExpression(self, "this")
  let assign = new CodeAssignStatement(atThis, new CodeArgumentReferenceExpression(target.Name))
  construct.Statements.Add(assign) |> ignore
  wrapper.Members.Add(construct) |> ignore  
  
  // Create a set to hold namespaces
  let namespaces = new HashSet<string>() 
       
  // Get all the methods of interest (properties would be similar)
  let methods = target.GetMethods(BindingFlags.Instance ||| BindingFlags.Public ||| BindingFlags.DeclaredOnly)
                |> Seq.filter (fun x -> not x.IsSpecialName )
                |> Seq.sortBy (fun p -> p.Name)
      
  
  // Accumulate all referenced namespaces (aside from generic types)          
  methods
  |> Seq.iter (fun x -> namespaces.Add(x.ReturnType.Namespace) |> ignore)
                     
  methods
  |> Seq.collect (fun m -> m.GetParameters())
  |> Seq.iter (fun x -> namespaces.Add(x.ParameterType.Namespace) |> ignore)
       
  // Generate a proxy class for each call    
  methods
  |> Seq.map (GenerateClass target ga)
  |> Seq.map wrapper.Members.Add
  |> Seq.iter ignore
  
  // Generate a delegating method for each call
  methods
  |> Seq.map (GenerateProxy target)
  |> Seq.map wrapper.Members.Add
  |> Seq.iter ignore  
  
  // Add in all the namespaces
  namespaces.Add(target.Namespace) |> ignore
  namespaces.Add(generated.Namespace) |> ignore
  namespaces
  |> Seq.sort
  |> Seq.iter (fun n-> scope.Imports.Add(new CodeNamespaceImport(n)))
  compileUnit

If we eschewed F# support entirely, and used a partial class, then we could skip the constructor and field declarations and provide the input by any other mechanism of our choice in a hand-written part.

Generating the closure classes is a simple matter -- especially as the names of the parameters can be used as fields directly without any sigils, provided that they never contain the @this or proxy name by convention. This makes the closure class approach slightly simpler than a snippet driven use of direct lambda syntax, where local variable names to hold out parameters would in general have to be generated so as not to clash with any of the arguments:

let GenerateClass (target:Type) (ga:CodeAttributeDeclaration)(m:MethodInfo) =
  // Generate the type and mark as generated
  let newtype = new CodeTypeDeclaration(m.Name + "__Proxy")
  newtype.TypeAttributes <- TypeAttributes.NestedPrivate /// .NotPublic ||| TypeAttributes.Sealed
  newtype.CustomAttributes.Add(ga) |> ignore
  
  // add a field to point to the wrapped object
  let this = new CodeMemberField(TypeRef target, "this")
  this.Attributes <- MemberAttributes.Public
  newtype.Members.Add(this) |> ignore
  
  // Fields to allow us to close over all the parameters
  m.GetParameters()
  |> Seq.map GenerateField
  |> Seq.map newtype.Members.Add
  |> Seq.iter ignore
  
  // The lambda method
  let proxy = new CodeMemberMethod()
  proxy.Name <- m.Name
  proxy.Attributes <- MemberAttributes.Public ||| MemberAttributes.Final
  
  // Fudge the case of a void method
  proxy.ReturnType <- TypeRef(m.ReturnType)
  if m.ReturnType = typeof<System.Void> then proxy.ReturnType <- TypeRef(typeof<int>)
  
  // Build the call to the wrapped object
  let self = new CodeThisReferenceExpression()
  let atThis = new CodeFieldReferenceExpression(self, "this")
  let call = new CodeMethodReferenceExpression(atThis, m.Name)
    
  let parameters = m.GetParameters()
                   |> Seq.map (fun x -> let ex = new CodeFieldReferenceExpression(self, x.Name)
                                        let dir = if x.IsIn && x.IsOut then FieldDirection.Ref
                                                  else if x.IsOut then FieldDirection.Out 
                                                  else FieldDirection.In
                                        new CodeDirectionExpression(dir, ex) :> CodeExpression)
                   |> Seq.toArray

  let invoke = new CodeMethodInvokeExpression(call, parameters)
  
  // return a dummy zero instead of void, or the result
  if m.ReturnType = typeof<System.Void> then 
    proxy.Statements.Add invoke |> ignore
    let result = new CodeMethodReturnStatement(new CodePrimitiveExpression(0))
    proxy.Statements.Add  result |> ignore
  else
    proxy.Statements.Add  (new CodeMethodReturnStatement(invoke)) |> ignore
    
  // add this method to the class
  newtype.Members.Add(proxy) |> ignore
 
  newtype

where the individual field declarations have to work around the decorations for out or ref, thus:

let GenerateField (p:ParameterInfo) =
  let t = if p.ParameterType.IsByRef then p.ParameterType.GetElementType() else p.ParameterType
  let field = new CodeMemberField(TypeRef t, p.Name)
  field.Attributes <- MemberAttributes.Public
  field

The generation of the proxy methods is equally mechanical

let GenerateProxy (target:Type) (m:MethodInfo) =
  // Generate the method and its parameter list
  let proxy = new CodeMemberMethod()
  proxy.Name <- m.Name
  proxy.Attributes <- MemberAttributes.Public ||| MemberAttributes.Final
  proxy.ReturnType <- TypeRef(m.ReturnType)
  m.GetParameters()
  |> Seq.iter ( fun (t:ParameterInfo) -> 
                let px = new CodeParameterDeclarationExpression()
                px.Direction <- if t.IsIn && t.IsOut then FieldDirection.Ref
                                else if t.IsOut then FieldDirection.Out 
                                else FieldDirection.In
                px.Name <- t.Name
                let tt = if t.ParameterType.IsByRef then t.ParameterType.GetElementType() else t.ParameterType
                px.Type <- TypeRef(tt)
                proxy.Parameters.Add(px) |> ignore
               )
  
  // construct the closure object -- the value is "let mutable" in F#
  let proxyType = new CodeTypeReference(m.Name + "__Proxy")
  let construct = new CodeVariableDeclarationStatement(proxyType, "proxy", 
                         CodeObjectCreateExpression((m.Name + "__Proxy"), [||]))
  proxy.Statements.Add construct |> ignore
  
  // create a reference to the closure
  let self = new CodeThisReferenceExpression()
  let atThis = new CodeFieldReferenceExpression(self, "this")
  let proxyRef = CodeVariableReferenceExpression("proxy")
                
  // initialize its reference to the wrapped object
  let assign = new CodeAssignStatement( new CodeFieldReferenceExpression(proxyRef, "this"), atThis)
  proxy.Statements.Add assign |> ignore
  
  // initialize all the other fields (to default if out only)
  m.GetParameters()
  |> Seq.map (fun x -> let left = new CodeFieldReferenceExpression(proxyRef, x.Name)
                       let right = if x.IsOut && (not x.IsIn) then 
                                      // Doesn't work in the F# CodeDOM
                                      new CodeDefaultValueExpression(TypeRef (x.ParameterType.GetElementType())) :> CodeExpression
                                   else 
                                      new CodeArgumentReferenceExpression(x.Name) :> CodeExpression
                       new CodeAssignStatement(left, right)
             )
  |> Seq.map proxy.Statements.Add
  |> Seq.iter ignore
  
  // Expect the lambda to return int for a void method
  let typeRef = if m.ReturnType = typeof<System.Void> then TypeRef(typeof<int>)
                else TypeRef(m.ReturnType)
  
  // call the Wrapping method which takes the lambda
  let funcType = CodeTypeReference("Func")
  funcType.TypeArguments.Add typeRef |> ignore
  
  // because proxyRef is "let mutable" in F#, this attempt to close over it here is a compiler error in F#, but not in C#
  let func = CodeObjectCreateExpression(funcType, new CodeMethodReferenceExpression(proxyRef, m.Name))
  let computing = new CodeMethodInvokeExpression(
                    new CodeTypeReferenceExpression("Wrapper"),
                    "LogCall", 
                    new CodePrimitiveExpression(m.Name),
                    func)
  
  // and store the result
  let returnValue = new CodeVariableDeclarationStatement(typeRef, "return", computing)

  proxy.Statements.Add returnValue |> ignore
  
  // copy back all the out or ref parameters
  m.GetParameters()
  |> Seq.filter (fun x -> x.IsOut)
  |> Seq.map (fun x -> let left = new CodeArgumentReferenceExpression(x.Name)
                       let right = new CodeFieldReferenceExpression(proxyRef, x.Name)
                       new CodeAssignStatement(left, right)
             )
  |> Seq.map proxy.Statements.Add
  |> Seq.iter ignore
  
  // and for a non-void method, return the value from the lambda
  if m.ReturnType <> typeof<System.Void> then 
    let r = new CodeMethodReturnStatement(CodeVariableReferenceExpression("return"))
    proxy.Statements.Add r |> ignore
  
  proxy

So, given a simple type

    public class Subject
    {
        public int DoIt(int argument, out string result) {...}

        public void DoItSilently(int argument, out string result) {...}
    }

we generate

namespace Derived {
    using Base;
    using System;
    using System.CodeDom.Compiler;
    
    
    [GeneratedCodeAttribute("Generator", "0.0.0.0")]
    public sealed class SubjectWrapper {
        
        private Subject @this;
        
        private SubjectWrapper(Subject Subject) {
            this.@this = Subject;
        }
        
        public int DoIt(int argument, out string result) {
            DoIt__Proxy proxy = new DoIt__Proxy();
            proxy.@this = this.@this;
            proxy.argument = argument;
            proxy.result = default(string);
            int @return = Wrapper.LogCall("DoIt", new Func<int>(proxy.DoIt));
            result = proxy.result;
            return @return;
        }
        
        public void DoItSilently(int argument, out string result) {
            DoItSilently__Proxy proxy = new DoItSilently__Proxy();
            proxy.@this = this.@this;
            proxy.argument = argument;
            proxy.result = default(string);
            int @return = Wrapper.LogCall("DoItSilently", new Func<int>(proxy.DoItSilently));
            result = proxy.result;
        }
        
        [GeneratedCodeAttribute("Generator", "0.0.0.0")]
        private class DoIt__Proxy {
            public Subject @this;
            public int argument;
            public string result;
            public int DoIt() {
                return this.@this.DoIt(this.argument, out this.result);
            }
        }
        
        [GeneratedCodeAttribute("Generator", "0.0.0.0")]
        private class DoItSilently__Proxy {
            public Subject @this;
            public int argument;
            public string result;
            public int DoItSilently() {
                this.@this.DoItSilently(this.argument, out this.result);
                return 0;
            }
        }
    }
}

The F# code contains

                let mutable (proxy:SubjectWrapper_DoIt__Proxy) = new SubjectWrapper_DoIt__Proxy()
...
                proxy.result <- (* Unknown expression type 'CodeDefaultValueExpression' please report this to the F# team. *)

Manually replacing this with

                let mutable (proxy:SubjectWrapper_DoIt__Proxy) = new SubjectWrapper_DoIt__Proxy()
...
                proxy.result <- Unchecked.defaultof<string>

then shows up on the next line

                let mutable (_return:int) = Wrapper.LogCall("DoIt", new Func(proxy.DoIt))

as an error The mutable variable 'proxy' is used in an invalid way. Mutable variables cannot be captured by closures. Consider eliminating this use of mutation or using a heap-allocated mutable reference cell via 'ref' and '!'.; and as there are no readonly locals in the CLR -- it's all F# compiler magic that gives the illusion of same -- we're stuck with no control to tweak to make proxy immutable in the F# output.

CodeDOM and poor man's lambdas -- Part 1

2011-05-30T17:32:00.009+01:00

The CodeDOM expression methods are stuck at the level of .net 2.0, and look unlikely to grow further support for further syntax outside of feeding it in as literal strings using the System.CodeDom.CodeSnippet* classes -- which takes away the cross-language spirit of the whole library.

So, what to do when we want to generate code containing lambdas?

Well, the obvious way is to strip away the syntactic sugar involved in the lambda/closure notation and generate explicitly what the compiler is doing for you under the covers anyway.

So, to take a case which is relevant to my interests, if we have a method (of static type Wrapper, say) --

        public static T LogCall<T>(string name, Func<T> call)

that will perform some pre- and post- call logging around invoking the delegate; and we have a method

        public int DoIt(int argument, out string result)

which we want to invoke through the LogCall method (considered as a decorator to the action). Writing this by hand, we would want a decorator type containing a matching method something like this:

        public int DoIt(int argument, out string result)
        {
            string local_result = default(string);
            var computed = Wrapper.LogCall("DoIt", () => wrapped.DoIt(argument, out local_result));
            result = local_result;
            return computed;
        }

which exposes the same interface, and delegates to the real logic in the contained object wrapped.

If we examine the code that is actually generated by compiling this, we see that the whole closure is replaced by a synthetic private inner class with a public field for each object closed over -- in this case all the arguments to the DoIt method, plus a this reference for the decorator -- and a method that stands for the lambda, which, generated names aside, looks like:

        public int DoIt()
        {
                return this.@this.wrapped.DoIt(argument, out result);
        }

while the wrapping method looks like

        public int DoIt(int argument, out string result)
        {
            var proxy = new DoItProxy();
            proxy.result = default(string);
            proxy.argument = argument;
            proxy.@this = this;
            var computed = Wrapper.LogCall("DoIt", new Func<int>(proxy.DoIt));
            result = proxy.result;
            return computed;
        }

And now, while the analogous F# code would probably return an int * string tuple, rather than an out parameter for multiple returns, the equivalent code snippet will do the job, just changing the types of the Func object; and the literal transposition of the C# code, out parameters and all, should still work, cross language.

In the more general case, of lambdas with arguments, then the method in the proxy type has the same arguments as the lambda needs; and in the case where no variables are being closed over e.g.

        Func<int, int> square = x => x*x;

there is an optimization that the supporting method can be made static on the enclosing type, rather than requiring an object to hold the closure references.

In part 2, having simplified the problem, we can move on to using the CodeDOM to generate the LogCall decorating method and closure, given a MethodInfo.

C# under the covers -- Enumerators

2011-05-29T09:49:00.009+01:00

Following up on the comment stream to an earlier post, a few little bits revealed by the investigations, with a little help from ILSpy:

yield break;

is the equivalent of

return;

inside an iterator function -- by which I mean that any silent exit from the method is compiled to the equivalent of an explicit yield break;, and

foreach(T x in y) { ... }

on exit from the iteration calls the Dispose() method of the IEnumerator<T> it uses implicitly to iterate over the collection y; whereas, of course, explicit use of the enumerator MoveNext() and Current do not.

The behavioural constraints of IEnumerable are weak enough as it is, being silent on whether replayability is part of the contract or not; this inconsistent disposal (with the IDisposable interface being added only to the generic IEnumerator<T> and not the .net 1.x vanilla version) makes guessing behaviours even worse when using LINQ methods like Skip which does the dispose via using when the iteration is exhausted

// System.Linq.Enumerable
private static IEnumerable<TSource> SkipIterator<TSource>(IEnumerable<TSource> source, int count)
{
 using (IEnumerator<TSource> enumerator = source.GetEnumerator())
 {
  while (count > 0 && enumerator.MoveNext())
  {
   count--;
  }
  if (count <= 0)
  {
   while (enumerator.MoveNext())
   {
    yield return enumerator.Current;
   }
  }
 }
 yield break;
}

and Take, which does it implicitly by the foreach route before the iterator has been completely drained

// System.Linq.Enumerable
private static IEnumerablelt;TSource> TakeIteratorlt;TSource>(IEnumerablelt;TSource> source, int count)
{
 if (count > 0)
 {
  foreach (TSource current in source)
  {
   yield return current;
   if (--count == 0)
   {
    break;
   }
  }
 }
 yield break;
}

There are probably similar wonders waiting to be unearthed in the F# collections APIs as well.

Belated Reviews — Anime

2011-05-27T18:44:00.010+01:00

Otome Youkai Zakuro

Japan, about 100 years ago; and the pace of modernization is causing disturbances amongst the spirit folk. So a Ministry of Spirit Affairs is set up, and a number young army officers assigned to liaise with representatives of the magical folk. Being a shoujo, this means that there is a pretty fox girl (or two) for each of the handsome young men, including the eponymous Zakuro for Agemaki, the male lead, and all the necessary flowers and sparkles to show where True Love is blossoming.

The series is best at the start -- introducing the characters, setting up the different romances, and nodding to the premise with a spirit-of-the-week to resolve, from the kami whose shrine has been demolished to make way for a station hotel, to the mysterious presence haunting Agemaki's family home. Alas, after the half-way point, it grows a LOLplot, involving Zakuro's mysterious past, and rather trips over itself -- the Serious Business being rather too much for the previously established light-hearted romance set-up.

Super Robot Wars OG — The Inspector

Coming in from the cold to what is actually the second season of this title, I just rolled with it, figuring out as I went along which of the many factions were supposed to be the good guys and which the villains -- because I didn't find this guide -- and just kicking back to watch the escalating battles, goofy robots, and Itano circus.

This is what anime ought to be more like -- not the endless highschool rom-com, or cute girls being cute for the otaku to perv over.

Letter Bee REVERSE

The first season ended on a cliff-hanger, where Lag's erstwhile mentor and role-model Gauche Suede appeared as one of the enemies of the whole Amberground system; and the second season carries on with the struggle between the Letter Bees, and the conspirators in REVERSE who are plotting to extinguish the artificial sun which lights this world (hinted to be Earth in some distant future).

Surprisingly, for the thwarting of such an existential threat, there is little by way of revelation (we never see the sunlit lands of Akatsuki, only the twilight regions; the various hints at deeper things going on, the very nature of the artificial sun are raised as hints, but never revisited), revolution (we return pretty much to the status quo ante after the whole secret war), or repercussion (after the threat is averted, all the elements that would disturb the status quo simply migrate to a backwater town, while Lag returns to his normal rounds as a delivery boy).

Definitely a case of better to travel hopefully that to arrive.

Level E

In what seems to be Japan's answer to Men in Black, we have a setting where Earth is a way-station for many alien races, all staying undercover, in an uneasy truce despite existing enmities. And then the Prince of the race in nominal charge of the planet comes to visit.

This would not be so bad, were it not for the fact that Prince Baka is a trickster with little restraint, while his minders are trying to keep any interstellar incident from being set off.

Thus kicks off the case of the long-term enemy race who are serious baseball fans, the random highschoolers who are turned into a sentai team (and their alien assassin teacher), the hive-queen who comes to Earth looking for a mate (and finds something other than she'd expected), the last of the twin-tailed mermaids, and finally Baka's betrothed and brother who are determined to see him wed. Baka sails through all this in total serenity, while his minders sink further into helpless rage.

Adapted from a 15 year old manga, it stuck out from all the other titles on offer for the first quarter.

Currently watching

Mobile Suit Gundam -- and that's it.

The only one of the current season's offerings I might have tried (Hyouge Mono) isn't on Crunchyroll, and even were I to go back into the grey world of fansubs, it's a title that has all the hallmarks of a series that will never get fully subbed for years.

Spring Cycling 2011

2011-05-24T21:07:00.001+01:00

A short holiday in a spell of warm, bright but very windy weather. I started by doing a quick spin out to lunch at the Crown and Punchbowl on Saturday, returning along the towpath; then set out to re-do one of the short tours that I did a couple of years ago

View Spring 2011 in a larger map

There was a sprinkling of rain on the Sunday lunchtime, and again overnight on Monday, but that was really of no consequence, unlike the previous time. The wind, however, was a different story, making the going a slow grind, and the journeys not so full of detours as before.

The minor road into Stowmarket was closed, as I discovered on the way in; so I tried to follow the intended route -- only to miss a turn and only realise on reaching the A140, so looped back to the intended route after crossing.

So, by the time I got to Cotton, it was about opening time -- and thus started a litany of non-functioning pubs, ending up at the Waterfront Inn in Diss where I could get food as well as drink in mid-afternoon. Just as well I had a substantial meal, as it turned out that the kitchens at the Scole Inn close on Sunday evenings.

The next morning, looking at the time and weather, I made the journey to Framlingham, stopping at Stradbroke to pick up some juice and biscuits, and them pausing on convenient benches to rest and read a while, arriving in time for a late lunch, and plenty of time to amble around the castle.

Castle from Gatehouse

The weather continuing windy, the massive detour I did the previous time was out of the question, so instead I stopped at Sutton Hoo, doing all the woodland walks, as well as the exhibits; and rehydrating with coffee and soup.

The GAR seat at Sutton Hoo

One final stop at the Dog at Grundisburgh, then the final leg into the wind; and the official return route using the cycle crossing for the A140, rather than my more usual approach from other directions.

CodeDOM -- generating an interface to a type

2011-05-20T08:07:00.002+01:00

This is a fairly simple process -- reflect over the public properties and methods, and generate a matching interface member, rather like this:

let GenerateInterface (target:Type) =
  let compileUnit = new CodeCompileUnit()
  
  let scope = new CodeNamespace("Generated")
  
  let namespaces = new HashSet<string>() 
  
  compileUnit.Namespaces.Add( scope ) |> ignore
  let wrapper = new CodeTypeDeclaration("I" + target.Name + "Wrapper")
  wrapper.IsInterface <- true
  scope.Types.Add(wrapper) |> ignore
  IncludeProperties wrapper target namespaces
  IncludeMethods wrapper target namespaces
       
  namespaces
  |> Seq.sort
  |> Seq.iter (fun n-> scope.Imports.Add(new CodeNamespaceImport(n)))
  compileUnit

which puts the required namespaces into order (TODO: sort System.* ahead of the rest).

The two functions look much the same, so could be factored further:

let IncludeProperties (klass:CodeTypeDeclaration) (target:Type) (namespaces:HashSet<string>) =
  target.GetProperties(BindingFlags.Instance ||| BindingFlags.Public)
  |> Seq.sortBy (fun p -> p.Name)
  |> Seq.iter (fun p ->
                   let m = new CodeMemberProperty()
                   m.Name <- p.Name
                   namespaces.Add(p.PropertyType.Namespace) |> ignore
                   m.Type <- TypeRef(p.PropertyType)
                   m.HasGet <- p.CanRead
                   m.HasSet <- p.CanWrite
                   p.GetIndexParameters() 
                   |> Seq.iter ( fun t -> 
                                 let px = new CodeParameterDeclarationExpression()
                                 px.Direction <- FieldDirection.In
                                 px.Name <- t.Name
                                 namespaces.Add(t.ParameterType.Namespace) |> ignore
                                 px.Type <- TypeRef(t.ParameterType)
                                 m.Parameters.Add(px) |> ignore
                                )
                   if m.Parameters.Count > 0 then m.Name <- "Item"
                   klass.Members.Add( m ) |> ignore
              )

which special cases the indexer property; and

let IncludeMethods (klass:CodeTypeDeclaration) (target:Type) (namespaces:HashSet<string>) =
  target.GetMethods(BindingFlags.Instance ||| BindingFlags.Public)
  |> Seq.filter (fun x -> not x.IsSpecialName )
  |> Seq.sortBy (fun p -> p.Name)
  |> Seq.iter (fun p ->
                   let m = new CodeMemberMethod()
                   m.Name <- p.Name
                   m.ReturnType <- TypeRef(p.ReturnType)
                   namespaces.Add(p.ReturnType.Namespace) |> ignore
                   p.GetParameters()
                   |> Seq.iter ( fun (t:ParameterInfo) -> 
                                 let px = new CodeParameterDeclarationExpression()
                                 px.Direction <- if t.IsIn && t.IsOut then FieldDirection.Ref
                                                 else if t.IsOut then FieldDirection.Out 
                                                 else FieldDirection.In
                                 px.Name <- t.Name
                                 namespaces.Add(t.ParameterType.Namespace) |> ignore
                                 px.Type <- TypeRef(t.ParameterType)
                                 m.Parameters.Add(px) |> ignore
                                )
                   klass.Members.Add( m ) |> ignore
              )

(TODO: generics and attributes, which I don't need for my immediate use case)

To make the code look less cluttered, to get type specifications which aren't fully namespace qualified everywhere, we want to specify types with keyword synonyms by type object, and the rest by name (with the namespaces included by using or open). So we have helper functions

let SysType (t:Type) =
  if t = typeof<System.Object> ||
     t = typeof<System.Char> ||
     t = typeof<System.Boolean> ||
     t = typeof<System.String> ||
     t = typeof<System.Int32> ||
  // also uint, float, etc...  ; probably also Nullables
     t = typeof<System.Void> then true
  else false
    
let TypeRef (t:Type) =
  if t.IsArray then
    if SysType <| t.GetElementType() then new CodeTypeReference(t)
    else new CodeTypeReference(t.Name)
  else if SysType t then new CodeTypeReference(t)
  else new CodeTypeReference(t.Name)

where again, TODO handling generic types, especially collections, to get them looking pretty.

So if we use string as the type, and dump to C# we get

namespace Generated {
    using System;
    using System.Globalization;
    using System.Text;
    
    
    public interface IStringWrapper {
        
        char this[int index] {
            get;
        }
        
        int Length {
            get;
        }
        
        object Clone();

        ...
        
        string TrimStart(char[] trimChars);
    }
}

and in F#

namespace Generated
    // Generated by F# CodeDom stuff omitted...
    open System
    open System.Globalization
    open System.Text
    
    type
        IStringWrapper = interface
            abstract Item : int -> char with get
            
            abstract Length : int with get
            
            abstract TrimStart : char[] -> string
            
            ...
            
            abstract Clone : unit -> obj
        end

where interestingly, the sorted order of the inserted members is reversed.

Hello CodeDOM -- building C# and F# code from F#

2011-05-16T21:44:00.004+01:00

A five-finger exercise in CodeDOM using the "Hello World" example from F# -- more interesting stuff to follow later:

module ProofOfConcept

open System.CodeDom
open System.CodeDom.Compiler
open System.IO
open Microsoft.CSharp
open Microsoft.FSharp.Compiler.CodeDom

let GenerateCode(compileunit: CodeCompileUnit) (provider:CodeDomProvider) =
    // Build the output file name.
    let sourceFile = Path.ChangeExtension("HelloWorld.", provider.FileExtension)

    // Create a TextWriter to a StreamWriter to the output file.
    use sw = new StreamWriter(sourceFile, false)
    let tw = new IndentedTextWriter(sw, "  ")

    // Generate source code using the code provider.
    provider.GenerateCodeFromCompileUnit(compileunit, tw,
            new CodeGeneratorOptions())
    tw.Close()
    sourceFile

let GenerateCSharpCode(compileunit: CodeCompileUnit) =
    // Generate the code with the C# code provider.
    let provider = new CSharpCodeProvider();
    GenerateCode compileunit provider
    
let GenerateFSharpCode(compileunit: CodeCompileUnit) =
    // Generate the code with the F# code provider.
    let provider = new FSharpCodeProvider();
    GenerateCode compileunit provider

[<EntryPoint>]
let main a =
  let compileUnit = new CodeCompileUnit()
  let samples = new CodeNamespace("Samples")
  samples.Imports.Add(new CodeNamespaceImport("System"))
  compileUnit.Namespaces.Add( samples ) |> ignore
  let class1 = new CodeTypeDeclaration("Class1")
  samples.Types.Add(class1) |> ignore
  let start = new CodeEntryPointMethod();
  let cs1 = new CodeMethodInvokeExpression(
                    new CodeTypeReferenceExpression("System.Console"),
                    "WriteLine", new CodePrimitiveExpression("Hello CodeDOM World!"))
  start.Statements.Add(cs1) |> ignore
  class1.Members.Add( start ) |> ignore
  GenerateCSharpCode compileUnit |> ignore
  GenerateFSharpCode compileUnit |> ignore
  0

which yields up

//------------------------------------------------------------------------------
// <auto-generated>
//     This code was generated by a tool.
//     Runtime Version:2.0.50727.4211
//
//     Changes to this file may cause incorrect behavior and will be lost if
//     the code is regenerated.
// </auto-generated>
//------------------------------------------------------------------------------

namespace Samples {
    using System;
    
    
    public class Class1 {
        
        public static void Main() {
            System.Console.WriteLine("Hello CodeDOM World!");
        }
    }
}

which isn't StyleCop compliant, but is otherwise well formatted; and

//------------------------------------------------------------------------------
// <autogenerated>
//     This code was generated by a tool.
//     Runtime Version: 2.0.50727.4211
//
//     Changes to this file may cause incorrect behavior and will be lost if 
//     the code is regenerated.
// </autogenerated>
//------------------------------------------------------------------------------

namespace global

namespace Samples
    // Generated by F# CodeDom
    #nowarn "49" // uppercase argument names
    #nowarn "67" // this type test or downcast will always hold
    #nowarn "66" // this upcast is unnecessary - the types are identical
    #nowarn "58" // possible incorrect indentation..
    #nowarn "57" // do not use create_DelegateEvent
    #nowarn "51" // address-of operator can occur in the code
    #nowarn "1183" // unused 'this' reference
    open System
    
    exception ReturnException2ea1f686ef3e4730bedf29e248ff4aab of obj
    exception ReturnNoneException2ea1f686ef3e4730bedf29e248ff4aab
    [<AutoOpen>]
    module FuncConvertFinalOverload2ea1f686ef3e4730bedf29e248ff4aab =
      // This extension member adds to the FuncConvert type and is the last resort member in the method overloading rules. 
      type global.Microsoft.FSharp.Core.FuncConvert with
          /// A utility function to convert function values from tupled to curried form
          static member FuncFromTupled (f:'T -> 'Res) = f
    
    type
        
        Class1 = class
            new() as this =
                {
                }
            static member UnnamedMethod_0  () =
                System.Console.WriteLine("Hello CodeDOM World!") |> ignore
        end
    [<CompilationRepresentation(CompilationRepresentationFlags.ModuleSuffix)>]
    module __EntryPoint =
        [<EntryPoint>]
        let Main (args:string[]) =
            Class1.UnnamedMethod_0()
            0

which is a little idiosyncratic; the GUID-named types seem to be there in case of need from some construct that is not actually used in this simple example.

Now it just comes down to assembling to code object graph for something practical.

Not sure if bug...

2011-05-15T11:40:00.005+01:00

...but I set it to the fsbugs e-mail anyway.

Let's start with this F# code:

module Test.Component

// After the Haskell type
type public Either<'TFault, 'TSuccess> =
  | Left of 'TFault 
  | Right of 'TSuccess
  with 
   [<CompiledName("ForEvery")>]
   static member iter (fault : 'TFault -> unit) (fix : 'TSuccess -> unit) (value : Either<'TFault, 'TSuccess>) =
     match value with
     | Left a -> fault a
     | Right b -> fix b

   [<CompiledName("ForEach")>]
   static member iter0 = 
     Either<'TFault, 'TSuccess>.iter ignore
  
   [<CompiledName("ToOption")>]
   static member toOption (value:Either<'TFault, 'TSuccess>) = 
     match value with
     | Right y -> Some y
     | _ -> None 
end

The last function maps to C# as

namespace Test
{
 [CompilationMapping(SourceConstructFlags.Module)]
 public static class Component
 {
  [DebuggerDisplay("{__DebugDisplay(),nq}"), CompilationMapping(SourceConstructFlags.SumType)]
  [Serializable]
  [StructLayout(LayoutKind.Auto, CharSet = CharSet.Auto)]
  public abstract class Either<TFault, TSuccess> : IEquatable<Component.Either<TFault, TSuccess>>, IStructuralEquatable, IComparable<Component.Either<TFault, TSuccess>>, IComparable, IStructuralComparable
  {
...
   [CompilationSourceName("toOption")]
   public static FSharpOption<TSuccess> ToOption(Component.Either<TFault, TSuccess> value)
   {
    if (value is Component.Either<TFault, TSuccess>.Right)
    {
     Component.Either<TFault, TSuccess>.Right right = (Component.Either<TFault, TSuccess>.Right)value;
     TSuccess y = right.item;
     return FSharpOption<TSuccess>.Some(y);
    }
    return null;
   }
...
    }
  }
}

Now change the initial F# line to be namespace rather than module...

File1.fs(9,18): error FS0755: The 'CompiledName' attribute cannot be used with this language element
File1.fs(15,18): error FS0755: The 'CompiledName' attribute cannot be used with this language element
File1.fs(19,18): error FS0755: The 'CompiledName' attribute cannot be used with this language element

So for comparison purposes, let's replace CompiledName with Obsolete, and see what we've built...

namespace Test.Component
{
 [DebuggerDisplay("{__DebugDisplay(),nq}"), CompilationMapping(SourceConstructFlags.SumType)]
 [Serializable]
 [StructLayout(LayoutKind.Auto, CharSet = CharSet.Auto)]
 public abstract class Either<TFault, TSuccess> : IEquatable<Either<TFault, TSuccess>>, IStructuralEquatable, IComparable<Either<TFault, TSuccess>>, IComparable, IStructuralComparable
 {
...
  [Obsolete("ToOption")]
  public static FSharpOption<TSuccess> toOption(Either<TFault, TSuccess> value)
  {
   if (value is Either<TFault, TSuccess>.Right)
   {
    Either<TFault, TSuccess>.Right right = (Either<TFault, TSuccess>.Right)value;
    TSuccess y = right.item;
    return FSharpOption<TSuccess>.Some(y);
   }
   return null;
  }
...
  }
}

where I can't for the life of me see the difference.

This happens when building using the latest compiler release in either VS2008 shell or VS2010 shell; or from the command line using MSBuild for either .net3.5 or 4.

Fixing missing volume and network tray icons

2011-05-06T06:52:00.002+01:00

On some Vista systems, there's an intermittent corruption of the icons that give network status and volume control in the system notification "tray".

Running this PowerShell script will clear the wedged icons and restart explorer

remove-itemproperty -path "hkcu:\Software\Classes\Local Settings\Software\Microsoft\Windows\CurrentVersion\TrayNotify" -
name IconsStream
remove-itemproperty -path "hkcu:\Software\Classes\Local Settings\Software\Microsoft\Windows\CurrentVersion\TrayNotify" -
name PastIconsStream
$explorer = get-process -Name explorer
$path = $explorer.Path
stop-process -Name explorer
start-process $path

Graduation day

2011-04-30T23:58:00.003+01:00

So, 32 years after taking Part III of the Maths Tripos, I've finally received a degree for it, the MMath, the most junior of the three Masters degrees I hold from Cambridge.

With this batch of retrospective degrees, my old college -- Emmanuel -- wanted to make big celebration of it, so as well as going to be admitted to the degree in person, it was an occasion to take up, for the very first time, my MA dining rights. And with such a retrospective grant there were people ranging from retirees to those who had still to take the MA -- but comparatively few who had actually done much mathematical for long after Part III! Stranger still to see some people for the first time in half a lifetime or more -- "Did I used to supervise you?"

The weather was bright and rather breezy, though the Chapel Court, where the photo was taken (if only I'd noticed I'd still not untucked the bands from under the hood), was quite still, so while waiting for the reception before lunch we waited in the shade in New Court -- admiring the roses and irises already blooming in the sheltered and sunny parts. And it was at that point that I realised that the stitching had failed down the side of both of my new pair of shoes...

There was discreet disabled access into the Old Library for the drinks, and then, it Hall, Karen was seated at the end of one table, avoiding any problems with the benches.

After lunch we separated as I went to get placed into the correct order in the crocodile, have the ceremony rehearsed, and then marched to the Senate House. Once inside it was possible to discreetly dab at the perspiration from a brisk walk in the sun in all the gear -- graduation days are usually either hot or wet -- while waiting our turn.

After having been processed, without tripping over my gown, or losing bits of shoe, time to go out onto the lawn, get some professional photos, chat with other people I knew also getting the MMath, before heading off, handing back the hood and gown (Karen was wearing our joint MA gown, as a Member of the University attending a Congregation), and then, at last, taking off the shoes that were just about holding upper and soles together, to go barefoot back to the car.

After a short pause to recover, I then cycled back into town for Hall dominated by newly minted MMath graduates; a very pleasant meal followed by cheese, fruit and chocolates with coffee port and Madeira in the parlour, all with a great deal of reminiscence.

And then, about half past ten, back on the bike to cycle back home in the pleasantly cool evening, and with surprisingly little traffic on any of the roads.

Applying IEnumerable to chunked IEnumerable in F# -- Cipher Feedback mode

2011-04-15T22:05:00.007+01:00

Having developed the supporting infrastructure to pull data from any IEnumerable into a sequence fixed length buckets, we can now apply it to some practical use. One such is to perform encryption of an input byte sequence in a block-cipher feeback (CFB) mode.

CFB mode is so simple, it almost feels like cheating. Given an initialization vector (IV) -- an arbitrary block of bytes -- as the 0-th ciphertext block, the iterative definition is that the n-th ciphertext block is the encrypted value of the n-1-th block XOR'd with the n-th plaintext block; and by the symmetry of XOR, the n-th plaintext block is recovered from the n-th ciphertext block by the same XOR.

So, let's implement it, and do a simple test that we can easily work out the cipherstream for -- and doesn't actually involve real encryption. So let's take 128-bit blocks, all zeros for the IV, and the identity transformation for our "encryption", and have the integers 0 to 41 as input.

The first ciphertext block will be all zeroes (IV encrypted) XOR'd with 0-15, i.e. 0-15. That XORs with 16-31 to give a second block of all 16s; the final block is 32-41 XORd with 16s, i.e. 48-57.

So, armed with expectations we can write the implementation with appropriate self-test code, thus --

namespace Tinesware.Enumerables

open System
open System.Collections.Generic
open System.IO
open System.Linq
open Should.Fluent

// An internal type to wrap an IEnumerator<'a>
// and disallow arbitrary disposal.
type private Controlled<'a>(iterator: IEnumerator<'a>) =
    interface IEnumerator<'a> with
      member self.Current = iterator.Current
    interface System.Collections.IEnumerator with
      member self.Current = iterator.Current :> Object
      member self.Reset() = iterator.Reset()
      member self.MoveNext() = iterator.MoveNext()
    interface IDisposable with
      member self.Dispose() = () // The whole purpose of the type -- prevent premature disposal

// Changes the semantics of any sequence to be streamlike
// There is just the one iterator which marches forwards
// inexorably, whatever the underlying sequence
type StreamingSequence<'a>(source: IEnumerable<'a>) =
    let enumerator = source.GetEnumerator()
    let iterator = new Controlled<'a>(enumerator) :> IEnumerator<'a>
    member self.Release() = enumerator.Dispose()
    interface IEnumerable<'a> with
        member self.GetEnumerator() = iterator
    interface System.Collections.IEnumerable with
        member self.GetEnumerator() = iterator :> System.Collections.IEnumerator
        
module Chunk =
   
    /// <summary>
    /// Split an enumerable into an enumeration of arrays
    /// </summary>
    /// <typeparam name="T">Element type of the array</typeparam>
    /// <param name="source">The IEnumerable to operate on</param>
    /// <param name="chunk">The window size</param>
    /// <returns>The enumeration</returns>
    let Window (chunk : int) (source : 'a seq) =
        let rec innerWindowing (x :StreamingSequence<'a>) =
           seq {
               let result = x |> Seq.truncate chunk |> Seq.toArray
               if result.Length > 0 then
                 yield result
                 yield! innerWindowing x
               else
                 x.Release()
               }
        
        // StreamingSequence makes the truncate/skip behaviour
        // predictable, whatever the semantics of the 
        // underlying sequence provider
        innerWindowing <| new StreamingSequence<'a>(source)
        
    // Exclusive-or two byte sequences to the 
    // length of the shorter; returns an array
    let (^) (a: byte seq) (b: byte seq) =
      Seq.zip a b
      |> Seq.map (fun x -> (fst x ^^^ snd x))
      |> Seq.toArray
        
    // Produce a function that performs cfb-mode enciphering
    // given the block cipher forward encryption and an IV
    // Alas the type system doesn't permit us to say that all
    // these byte arrays must be the same size -- that would
    // take C++, I fear
    let CfbEncrypt (encrypt : byte[] -> byte[]) (iv : byte[])  =
      let cn_1 = ref iv
      let operation (plaintext : byte[]) = 
        let ecn_1 = encrypt (!cn_1)
        let cn = ecn_1 ^ plaintext
        cn_1 := cn
        cn
      operation
      
    /// <summary>
    /// A Self-test program for CFB mode
    /// </summary>
    [<EntryPoint>]
    let main a =
        let input = seq { 0 .. 41 }
                    |> Seq.map byte
                    |> Seq.toArray
                    
        let iv = seq { 1 .. 16 }
                 |> Seq.map (fun x -> 0uy)
                 |> Seq.toArray

        let cfb = CfbEncrypt id iv
        let output = input 
                        |> Window 16
                        |> Seq.map cfb
                        |> Seq.collect id // unWindow
                        |> Seq.toArray
                        
        let expected = [| 0uy;  1uy;  2uy;  3uy;  4uy;  5uy;  6uy;  7uy; 
                          8uy;  9uy; 10uy; 11uy; 12uy; 13uy; 14uy; 15uy; // zeros xor 0-15
                         16uy; 16uy; 16uy; 16uy; 16uy; 16uy; 16uy; 16uy; 
                         16uy; 16uy; 16uy; 16uy; 16uy; 16uy; 16uy; 16uy; // 0-15 xor 16-31
                         48uy; 49uy; 50uy; 51uy; 52uy; 53uy; 54uy; 55uy;
                         56uy; 57uy; |]                                  //16s xor 32-41
                         
        output.Length.Should().Equal(input.Length) |> ignore
        let matching = Seq.forall2 (fun x y -> x = y) expected output 
        matching.Should().Be.True |> ignore
        
        // new instance to reset the IV
        let cfb = CfbEncrypt id iv

        let decrypt = output
                        |> Window 16
                        |> Seq.map cfb
                        |> Seq.collect id // unWindow
                        |> Seq.toArray
        
        decrypt.Length.Should().Equal(input.Length) |> ignore
        let matching = Seq.forall2 (fun x y -> x = y) input decrypt 
        matching.Should().Be.True |> ignore

        0

where I've renamed the previous private Ratchet class to the public StreamingSequence type -- a more self-explanatory name for possible re-users; and exposed the Dispose method of the constant (reference, mutable internal state) IEnumerator for end-of-enumeration use.

Here, Seq.collect is being used to flatten the sequence after the block-level operation has been done.

CFB mode is so simple that -- having already done the hard work of breaking any sequence into blocks -- the only thing that stopped the test running through first time was forgetting to re-initialize the CFB operation for the decryption pass.

Actually, the decryption operation shouldn't be working -- but somehow it does, even when I push in random input, and replace the identity encryption operation with a one-way hash! Something weird is going on here.

F# under the covers XIII -- Sequence expressions

2011-04-13T07:56:00.004+01:00

After more than a year since the previous instalment...

A C# method that returns an enumerator via yield return generates a synthetic class that looks like this:

private sealed class <ToEnumerable>d__0 : IEnumerable<T>, IEnumerable, IEnumerator<T>, IEnumerator, IDisposable
{
    // data fields omitted
    private T System.Collections.Generic.IEnumerator<System.T>.Current { get; }
    private object System.Collections.IEnumerator.Current { get; }
    public <ToEnumerable>d__0(int <>1__state) : base() { ... }
    private IEnumerator<T> System.Collections.Generic.IEnumerable<System.T>.GetEnumerator() { ... }
    private IEnumerator System.Collections.IEnumerable.GetEnumerator(){ ... }
    private bool MoveNext() { ... }

    private void System.Collections.IEnumerator.Reset()
    {
        throw new NotSupportedException();
    }
    private void System.IDisposable.Dispose()
    {
    }
}

where I omit unimportant detail based on the particular enumeration, but draw attention to how the other IEnumerator methods are implemented.

The F# result from a sequence expression is very different, with an explicit close and dispose:

internal sealed class ToEnumerable@18 : GeneratedSequenceBase<T>
{
    // data fields omitted
    public ToEnumerable@18(Stream stream, int result, T v, IEnumerator<T> @enum, int pc, T current) { ... }
    public override int GenerateNext(IEnumerable<T> next) { ... }
    public override void Close() { ... }
    public override bool get_CheckClose() { ... }

    public override T get_LastGenerated()
    {
        return this.current;
    }
    public override IEnumerator<T> GetFreshEnumerator()
    {
        return new Chunk.ToEnumerable@18( ... );
    }
}

and in GeneratedSequenceBase we have

private virtual void System-IDisposable-Dispose()
{
 if (this.redirect)
 {
  GeneratedSequenceBase<T> arg_11_0 = this.redirectTo;
  tail();
  arg_11_0.Close();
  return;
 }
 else
 {
  GeneratedSequenceBase<T> arg_1A_0 = this;
  tail();
  arg_1A_0.Close();
  return;
 }
}

The Close method renders the enumeration non-functional, though the closed enumerator does not throw, it simply yields no more. That the resulting sequence has an active Dispose behaviour, unlike the C# case, explains why in the Window example I needed to add the Controlled wrapper to block that, so that the Stream based enumeration written as a sequence expression would draw more than one tranche before stopping.

By contrast, you don't need the Controlled wrapper for windowing the BCL derived array-as-enumerator; presumably that has a no-op Dispose.

Generalized IEnumerable to chunked IEnumerable in F#

2011-04-12T23:07:00.002+01:00

F# sequence expressions, under the covers, are complicated beasts; for the purposes of today's exercise, the factor of interest is that they Dispose() internal enumerators aggressively, unlike in the C# case, so the simple Ratchet wrapper isn't enough by itself. So, with some refactoring, we get

namespace Tinesware.Enumerables

open System
open System.Collections.Generic
open System.IO
open System.Linq
open Should.Fluent

module Chunk =

    /// <summary>
    /// Turns a stream into a byte enumeration
    /// </summary>
    /// <param name="stream">The stream to wrap</param>
    /// <returns>The stream as an enumeration</returns>
    let rec ToEnumerable (stream : Stream) =
        seq { 
            let result = stream.ReadByte()
            if result >= 0 then
              yield (byte result)
              yield! ToEnumerable stream
            stream.Close()
        }
        
    type private Controlled<'a>(iterator: IEnumerator<'a>) =
        interface IEnumerator<'a> with
          member self.Current = iterator.Current
        interface System.Collections.IEnumerator with
          member self.Current = iterator.Current :> Object
          member self.Reset() = iterator.Reset()
          member self.MoveNext() = iterator.MoveNext()
        interface IDisposable with
          member self.Dispose() = () // The whole purpose of the type -- prevent premature disposal
        
    type private Ratchet<'a>(iterator: IEnumerator<'a>) =
      interface IEnumerable<'a> with
        member self.GetEnumerator() = iterator
      interface System.Collections.IEnumerable with
        member self.GetEnumerator() = iterator :> System.Collections.IEnumerator
        
    /// <summary>
    /// Split an enumerable into an enumeration of arrays
    /// </summary>
    /// <typeparam name="T">Element type of the array</typeparam>
    /// <param name="source">The IEnumerable to operate on</param>
    /// <param name="chunk">The window size</param>
    /// <returns>The enumeration</returns>
    let Window (chunk : int) (source : 'a seq) =
        let rec innerWindowing x =
           seq {
               let result = x |> Seq.truncate chunk |> Seq.toArray
               if result.Length > 0 then
                 yield result
                 yield! innerWindowing x
               }
               
        innerWindowing <| new Ratchet<'a>(new Controlled<'a>(source.GetEnumerator()))

      
    /// <summary>
    /// A Self-test program
    /// </summary>
    [<EntryPoint>]
    let main a =
        let input = seq { 0 .. 41 }
                    |> Seq.map byte
                    |> Seq.toArray
                    
        let inStream = new MemoryStream(input)
        let channel = inStream |> ToEnumerable

        let chunk = channel |> Seq.truncate 16 |> Seq.toArray
        chunk.Length.Should().Equal(16) |> ignore
        chunk.[0].Should().Equal((byte)0) |> ignore

        let chunk = channel |> Seq.truncate 16 |> Seq.toArray
        chunk.Length.Should().Equal(16) |> ignore
        chunk.[0].Should().Equal((byte)16) |> ignore

        let chunk = channel |> Seq.truncate 16 |> Seq.toArray
        chunk.Length.Should().Equal(10) |> ignore
        chunk.[0].Should().Equal((byte)32) |> ignore
        
        let chunks = input |> Window 16 |> Seq.toArray
        chunks.Length.Should().Equal(3) |> ignore
        chunks.[0].Length.Should().Equal(16) |> ignore
        chunks.[1].Length.Should().Equal(16) |> ignore
        chunks.[2].Length.Should().Equal(10) |> ignore
        
        let inStream = new MemoryStream(input)
        let chunks = inStream |> ToEnumerable |> Window 16 |> Seq.toArray
        chunks.Length.Should().Equal(3) |> ignore
        chunks.[0].Length.Should().Equal(16) |> ignore
        chunks.[1].Length.Should().Equal(16) |> ignore
        chunks.[2].Length.Should().Equal(10) |> ignore

        0

Note that this version closes the stream after the iteration completes -- we could leave the management of the stream entirely outside the iterator if we wished. Since we are re-using our Enumerator instance, we only have the one left for garbage collection, so suppressing the disposal doesn't litter too badly.

Overall, if you're in the simple use-case of pulling chunks out of a file in order, the earlier version of the system will suffice.

Edit 1-May-11 : Comparing with 2#4u's Seq.breakBy I get on my laptop

Real: 00:00:03.948, CPU: 00:00:03.946 for Seq.breakByV1
Real: 00:00:00.345, CPU: 00:00:00.374 for Seq.breakByV2
Real: 00:00:00.910, CPU: 00:00:00.904 for Chunk.Window

making the functional Chunk.Window implementation about 2.4 times slower than the faster -- but with more internal mutability -- of the the two.

Generalized IEnumerable to chunked IEnumerable

2011-04-12T20:07:00.006+01:00

As noted in the comments to the previous post, if you're not using the Window on an Enumerable that maintains its own internal state (so that the different IEnumerator yielded up each time will always be at the start of the iteration), it will loop forever. So if you're not using the two in conjunction for the purpose of "read a file in chunks for e.g. passing over a network" or similar, you want this variant:

namespace Tinesware.Enumerables
{
    using System;
    using System.Collections.Generic;
    using System.IO;
    using System.Linq;
    using Should.Fluent;

    /// <summary>
    /// More extension methods for enumerables
    /// </summary>
    public static class Chunk
    {
        /// <summary>
        /// Turns a stream into an IEnumerable<byte>
        /// </summary>
        /// <param name="stream">The stream to wrap</param>
        /// <returns>The stream as an enumeration</returns>
        public static IEnumerable<byte> ToEnumerable(this Stream stream)
        {
            while (true)
            {
                var result = stream.ReadByte();
                if (result >= 0)
                {
                    yield return (byte)result;
                }
                else
                {
                    break;
                }
            }
        }
        
        /// <summary>
        /// Split an IEnumerable<T> into an IEnumerable<T[]>
        /// </summary>
        /// <typeparam name="T">Element type of the array</typeparam>
        /// <param name="source">The IEnumerable to operate on</param>
        /// <param name="chunk">The window size</param>
        /// <returns>The enumeration</returns>
        public static IEnumerable<T[]> Window<T>(this IEnumerable<T> source, int chunk)
        {
         var readonce = new Ratchet<T>(source);
         
            while (true)
            {
                var result = readonce.Take(chunk).ToArray();

                if (result.Any())
                {
                    yield return result;
                }
                else
                {
                    break;
                }
            }
        }

        private class Ratchet<T> : IEnumerable<T>
        {
            private readonly IEnumerator<T> iterator;
         
            public Ratchet(IEnumerable<T> source)
            {
                this.iterator = source.GetEnumerator();
            }
         
            public IEnumerator<T> GetEnumerator()
            {
                return iterator;
            }
         
            System.Collections.IEnumerator System.Collections.IEnumerable.GetEnumerator()
            {
                return iterator;
            }
        }

        /// <summary>
        /// A Self-test program
        /// </summary>
        private static void Main()
        {
            var input = Enumerable.Range(0, 42).Select(x => (byte)x).ToArray();
            var inStream = new MemoryStream(input);
            var channel = inStream.ToEnumerable();

            var chunk = channel.Take(16).ToArray();
            chunk.Length.Should().Equal(16);
            chunk[0].Should().Equal((byte)0);

            chunk = channel.Take(16).ToArray();
            chunk.Length.Should().Equal(16);
            chunk[0].Should().Equal((byte)16);

            chunk = channel.Take(16).ToArray();
            chunk.Length.Should().Equal(10);
            chunk[0].Should().Equal((byte)32);

            inStream.Position = 0;

            var chunks = inStream.ToEnumerable().Window(16).ToArray();
            chunks.Length.Should().Equal(3);
            chunks[0].Length.Should().Equal(16);
            chunks[1].Length.Should().Equal(16);
            chunks[2].Length.Should().Equal(10);
            
            chunks = input.Window(16).ToArray();
            chunks.Length.Should().Equal(3);
            chunks[0].Length.Should().Equal(16);
            chunks[1].Length.Should().Equal(16);
            chunks[2].Length.Should().Equal(10);
        }
    }
}

where the Ratchet type ensures that the same IEnumerator instance is yielded up each time -- at this point you have to be rather Jesuitical about what is immutable anyway, the sequence that returns you different mutable objects or the one that returns you the same object you may already have mutated...

F# is less convenient for this one, though :(

App.config or app.exe.config? Open Question.

2011-04-09T09:41:00.006+01:00

Another thing that caught me on W2k3, 64-bit, recently, is that on some, but not all, systems if I have an application appname.exe which is C#, .net 3.5 sp1, compiled in AnyCPU mode, but installed to Program Files (x86), if you get the configuration file path by

var path = ConfigurationManager.OpenExeConfiguration(ConfigurationUserLevel.None).FilePath

it may report appname.config, as opposed to the more usual appname.exe.config as the file name.

And when that happens, it is not joking -- if your installer puts a file at appname.exe.config on such a system, your program will not read it. If you manually move the file to appname.config and restart the application, all will be well.

The internet hasn't been very forthcoming in either cause or remedy -- I did find several copies of a similar issue for 32-bit apps on Vista x64, but no resolution.

There's a similar bug report against .net 3.5 -- but by observation the issue only happens in a minority of cases. The only difference I've spotted between two machines where one shows the bug and the other doesn't is that the bug shows in the one with .net 3.0 and 3.5 sp1 not fully patched up to date -- but the patches don't touch any of the libraries that are likely to be involved (System.Configuration and dependencies).

More than you wanted to know about sc.exe and service isolation in Windows 2k3 SP2

2011-04-09T09:30:00.004+01:00

At the level of Win2k3 SP2 unpatched, sc.exe did not support the sidtype parameter for setting a per-service SID. If you issue

sc.exe sidtype MyService Unrestricted

it spits out help text, then pauses for user input

Would you like to see help for the QUERY and QUERYEX commands? [ y | n ]:

At patch MS09-012/KB959454 (or maybe KB956572), if not before, sc.exe was upgraded to interpret this parameter. However...

If on 64-bit Win 2k3 with that patch, you run the 32-bit -- Program Files (x86) -- version, that will accept the sidtype parameter and do nothing, at least on the various systems I've tested on.

Stream to IEnumerable; IEnumerable to chunked IEnumerable

2011-04-08T23:30:00.002+01:00

I may have missed these in F#, and they don't seem at all obvious in C# standard libraries. A couple of handy functions/extension methods with simple tests using the Should.Fluent assertion library.

In F#:

namespace Tinesware.Enumerables

open System
open System.Collections.Generic
open System.IO
open System.Linq
open Should.Fluent

module Chunk =

    /// <summary>
    /// Turns a stream into a byte sequence
    /// </summary>
    /// <param name="stream">The stream to wrap</param>
    /// <returns>The stream as an enumeration</returns>
    let ToEnumerable (stream : Stream) =
        seq { 
            let result = ref <| stream.ReadByte()
            while !result >= 0 do
              yield (byte !result)
              result := stream.ReadByte()
        }
        
    /// <summary>
    /// Split a 'a seq into an 'a[] seq
    /// </summary>
    /// <typeparam name="'a">Element type of the array</typeparam>
    /// <param name="source">The IEnumerable to operate on</param>
    /// <param name="chunk">The window size</param>
    /// <returns>The enumeration</returns>
    let Window (chunk : int) (source : 'a seq) =
        seq { 
            let result = ref (source |> Seq.truncate chunk |> Seq.toArray)
            while (!result).Length > 0 do
              yield !result
              result := source |> Seq.truncate chunk |> Seq.toArray
        }
      
    /// <summary>
    /// A Self-test program
    /// </summary>
    [<EntryPoint>]
    let main a =
        let input = seq { 0 .. 41 }
                    |> Seq.map byte
                    |> Seq.toArray
                    
        let inStream = new MemoryStream(input)
        let channel = inStream |> ToEnumerable

        let chunk = channel |> Seq.truncate 16 |> Seq.toArray
        chunk.Length.Should().Equal(16) |> ignore
        chunk.[0].Should().Equal((byte)0) |> ignore

        let chunk = channel |> Seq.truncate 16 |> Seq.toArray
        chunk.Length.Should().Equal(16) |> ignore
        chunk.[0].Should().Equal((byte)16) |> ignore

        let chunk = channel |> Seq.truncate 16 |> Seq.toArray
        chunk.Length.Should().Equal(10) |> ignore
        chunk.[0].Should().Equal((byte)32) |> ignore

        inStream.Position <- int64 0

        let chunks = inStream |> ToEnumerable |> Window 16 |> Seq.toArray
        chunks.Length.Should().Equal(3) |> ignore
        chunks.[0].Length.Should().Equal(16) |> ignore
        chunks.[1].Length.Should().Equal(16) |> ignore
        chunks.[2].Length.Should().Equal(10) |> ignore

        0

In C#:

namespace Tinesware.Enumerables
{
    using System;
    using System.Collections.Generic;
    using System.IO;
    using System.Linq;
    using Should.Fluent;

    /// <summary>
    /// More extension methods for enumerables
    /// </summary>
    public static class Chunk
    {
        /// <summary>
        /// Split an IEnumerable<T> into an IEnumerable<T[]>
        /// </summary>
        /// <typeparam name="T">Element type of the array</typeparam>
        /// <param name="source">The IEnumerable to operate on</param>
        /// <param name="chunk">The window size</param>
        /// <returns>The enumeration</returns>
        public static IEnumerable<T[]> Window<T>(this IEnumerable<T> source, int chunk)
        {
            while (true)
            {
                var result = source.Take(chunk).ToArray();

                if (result.Any())
                {
                    yield return result;
                }
                else
                {
                    break;
                }
            }
        }

        /// <summary>
        /// Turns a stream into an IEnumerable<byte>
        /// </summary>
        /// <param name="stream">The stream to wrap</param>
        /// <returns>The stream as an enumeration</returns>
        public static IEnumerable<byte> ToEnumerable(this Stream stream)
        {
            while (true)
            {
                var result = stream.ReadByte();
                if (result >= 0)
                {
                    yield return (byte)result;
                }
                else
                {
                    break;
                }
            }
        }

        /// <summary>
        /// A Self-test program
        /// </summary>
        private static void Main()
        {
            var input = Enumerable.Range(0, 42).Select(x => (byte)x).ToArray();
            var inStream = new MemoryStream(input);
            var channel = inStream.ToEnumerable();

            var chunk = channel.Take(16).ToArray();
            chunk.Length.Should().Equal(16);
            chunk[0].Should().Equal((byte)0);

            chunk = channel.Take(16).ToArray();
            chunk.Length.Should().Equal(16);
            chunk[0].Should().Equal((byte)16);

            chunk = channel.Take(16).ToArray();
            chunk.Length.Should().Equal(10);
            chunk[0].Should().Equal((byte)32);

            inStream.Position = 0;

            var chunks = inStream.ToEnumerable().Window(16).ToArray();
            chunks.Length.Should().Equal(3);
            chunks[0].Length.Should().Equal(16);
            chunks[1].Length.Should().Equal(16);
            chunks[2].Length.Should().Equal(10);
        }
    }
}

Q1 in books

2011-04-06T22:52:00.002+01:00

More fiction reading than I usually do these days, mainly from the grotty weather and having spotted a bunch of titles of interest to add to the Christmas list:

The Quantum Thief by Hannu Rajaniemi

Somewhat reminiscent of early Roger Zelazny -- definitely not the Egan-esque fare suggested by other reviews -- as if Zelazny had been handed Schroeder's Lady of Mazes and a genre trope update for the next 40-odd years when he was starting to write Today We Choose Faces.

Good, not great.

Kraken by China Miéville

After The City and the City, doing yet another magical London seems rather a return to the formulaic. At least he resisted -- given the title and the main character's name -- doing any "Billy the Squid" jokes.

Only pedestrian by comparison with his other works.

The Quiet War/Gardens of the Sun by Paul McAuley

You just have to take the next couple of hundred years future history in this set-up as a given, to provide the backdrop for a high-tech, solar-system spanning tale of Byzantine politics, interspersed with travelogue, the latter possibly becoming a bit repetitive each time we are shown a new piece of airless real-estate.

The ending is fortunately not telegraphed -- whether the not-so-bad guys or the greasy-pole climbers will win out is finely balanced almost all the way.

Wayland's Principia by Richard Garfinkle

This time, he takes on the old staple of First Contact, weaving the alien into territory more usually seen in myth -- it's all about its construction of aliens and where the not-human becomes a monster, and how the gulf might be bridged.

As such it's not set up as something that can have a tidy end, but rather fades out into the future, the narrative sort of pulling back and speeding up in the last few pages, until it rather petered out in the end. A good and interesting book, but not, I feel, his best work.

Full Spring

2011-04-06T22:30:00.002+01:00

After several weeks where the weather has been generally miserable -- at best leaden skies and wind, though not so bad as to stop me putting in a a few days of cycling to work -- and unexpected fire-fighting at work has been sapping my energies, today was, after a dull start, full of promise of summer. Let's hope it doesn't turn into one of those years where Q2 is put on back-to-front.

In the garden, the forsythia is finishing, the earliest tulips have finished, but the daffs, grape hyacinths and main tulips are in full flower; while the primroses carry on. And a fair number of new plants from last year failed to overwinter -- one of the lavenders, most of the geraniums, despite the latter being in a notionally frost-free greenhouse.

Still, the weather was mild enough weekend before last to take the surviving plants out, and fumigate the greenhouse; and I could then use the space to dry out the lawn weed-and-feed that had gotten damp in the shed over the winter. So the lawn dressing belatedly happened last weekend, a couple of weeks later than usual. And now the first of the orders of plants arrived, so there are tomatoes in pots, and chilis and overwintered perpetual spinach in the greenhouse; with salad leaves and basil planted as seeds.

Out in the rest of the garden, the bed for annuals has been dug, raked & weeded, so is ready for scattering seed, sunflowers are in a seed-tray, waiting to germinate, and the green bin full to the brim again, with over a week to the next collection -- showing how suddenly I have to make up for lost time (some years gardening started in mid-February, so I'm at least a month behind, plus having still to catch up from the abruptly truncated autumn).

Using LINQ/Extension methods with IronPython 2.7's clr.ImportExtensions method

2011-04-06T22:23:00.002+01:00

Woefully underdocumented on the internet, I worked this one out by analogy to the IronRuby using_clr_extensions method:

>& 'C:\Program Files\IronPython 2.7\ipy.exe'
IronPython 2.7 (2.7.0.40) on .NET 4.0.30319.225
Type "help", "copyright", "credits" or "license" for more information.
>>> import clr
>>> clr.AddReference("System.Core")
>>> from System.Collections.Generic import List
>>> dir(List)
['Add', 'AddRange', 'AsReadOnly', 'BinarySearch', 'Capacity', 'Clear', 'Contains', 'ConvertAll', 'CopyTo', 'Count', 'Enu
merator', 'Equals', 'Exists', 'Find', 'FindAll', 'FindIndex', 'FindLast', 'FindLastIndex', 'ForEach', 'GetEnumerator', '
GetHashCode', 'GetRange', 'GetType', 'IndexOf', 'Insert', 'InsertRange', 'IsReadOnly', 'IsSynchronized', 'Item', 'LastIn
dexOf', 'MemberwiseClone', 'ReferenceEquals', 'Remove', 'RemoveAll', 'RemoveAt', 'RemoveRange', 'Reverse', 'Sort', 'Sync
Root', 'ToArray', 'ToString', 'TrimExcess', 'TrueForAll', '__add__', '__class__', '__contains__', '__delattr__', '__doc_
_', '__format__', '__getattribute__', '__getitem__', '__hash__', '__init__', '__iter__', '__len__', '__new__', '__reduce
__', '__reduce_ex__', '__repr__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__']
>>> import System
>>> clr.ImportExtensions(System.Linq)
>>> dir (List)
['Add', 'AddRange', 'Aggregate', 'All', 'Any', 'AsEnumerable', 'AsParallel', 'AsQueryable', 'AsReadOnly', 'Average', 'Bi
narySearch', 'Capacity', 'Cast', 'Clear', 'Concat', 'Contains', 'ConvertAll', 'CopyTo', 'Count', 'DefaultIfEmpty', 'Dist
inct', 'ElementAt', 'ElementAtOrDefault', 'Enumerator', 'Equals', 'Except', 'Exists', 'Find', 'FindAll', 'FindIndex', 'F
indLast', 'FindLastIndex', 'First', 'FirstOrDefault', 'ForEach', 'GetEnumerator', 'GetHashCode', 'GetRange', 'GetType',
'GroupBy', 'GroupJoin', 'IndexOf', 'Insert', 'InsertRange', 'Intersect', 'IsReadOnly', 'IsSynchronized', 'Item', 'Join',
 'Last', 'LastIndexOf', 'LastOrDefault', 'LongCount', 'Max', 'MemberwiseClone', 'Min', 'OfType', 'OrderBy', 'OrderByDesc
ending', 'ReferenceEquals', 'Remove', 'RemoveAll', 'RemoveAt', 'RemoveRange', 'Reverse', 'Select', 'SelectMany', 'Sequen
ceEqual', 'Single', 'SingleOrDefault', 'Skip', 'SkipWhile', 'Sort', 'Sum', 'SyncRoot', 'Take', 'TakeWhile', 'ToArray', '
ToDictionary', 'ToList', 'ToLookup', 'ToString', 'TrimExcess', 'TrueForAll', 'Union', 'Where', 'Zip', '__add__', '__clas
s__', '__contains__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__getitem__', '__hash__', '__init__',
 '__iter__', '__len__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setitem__', '__sizeof__'
, '__str__', '__subclasshook__']

Uninstalling the uninstallable -- old, dead Java Consoles in Firefox

2011-03-01T20:55:00.003Z

There they are, every time. You say no to the Yahoo! toolbar, but each time you update your Java runtine, your extensions list accumulates another disabled Java Console 6.0.xx that won't uninstall through Firefox.

To get rid of an old 6.0.xx extension, go to C:\Program Files\Mozilla Firefox\extensions and just delete the folder called {CAFEEFAC-0016-0000-00xx-ABCDEFFEDCBA}, and you're sorted.

You can use the same trick to get rid of other extensions that won't take a hint (e.g. the Skype extension). The folder is another {Guid} -- look in the chrome.manifest file inside the folder for one containing appropriate names; or check the date of the folder against the date you last installed/updated.

Book — Summer of the Ubume by Natsuhiko Kyogoku

2011-02-27T11:03:00.002Z

The first novel by Natsuhiko Kyogoku, and the one to which Mouryou no Hako is a sequel.

The prose is rendered nicely into English, while retaining some of the stylistics tics of the original (no sentence broken between pages, at least for the first 200-odd pages); there are a few translators notes where allusions are made to various folk-tales. The most obtrusive translation quirks are where dates are concerned

The book he was reading was something from the Edo Period [1603-1868]...

but those are quickly past after the opening chapter.

Many of the cast of Mouryou no Hako appear in this first novel -- many more than I had expected. And Kyogoku seems to have a thing about bizarre hospitals.

By the end, this is clearly a strong first novel -- and it's also clear why the next book he wrote about the same cast was the one that got the adaptations : this one relied just a bit too much on the power of coincidence (three different characters suffering the same hysterical blindness, for one), and on having some of the main cast involved at some point in the past with one or other of the major players in the mystery.

Still, if you enjoyed the Mouryou no Hako anime, this is the nearest thing you'll get to having another fix of the same in English language.

Signs of Spring

2011-02-27T10:57:00.002Z

Sunlight through the curtains and birdsong woke me at just after 07:00 today. In the garden, the snowdrops are fading after having gone strong all month, and the forsythia has been out for over a week. Purple crocuses are joining the yellow ones; and the primroses in shadier parts are flowering, following the ones in the sunny bed by the greenhouse wall, which have been out for weeks.

Monday, as most of the office were out karting, I took the chance to cycle to work, and bunk off ahead of dusk and rain; but it'll be a couple of weeks yet before starting to do it around the normal working day.

Mono 2.10, XBuild, F# -- first attempts

2011-02-19T22:24:00.006Z

Following the announcement that F# and the Iron languages are being bundled with Mono 2.10 for Mac and Linux, I thought it worth taking the system for a spin on Windows as well. This is a record of my experiments, and are nowhere near a complete HOW-TO as yet (they more closely resemble a HOW NOT TO at the moment).

The easy bit is as per Zor's blog, here --

copy C:\program files\MSBuild\FSharp to C:\Program Files\Mono-2.10\lib\mono\xbuild
Get the mono strong-name key https://github.com/mono/mono/blob/master/mcs/class/mono.snk
Now this is voodoo -- without going & 'C:\Program Files\Mono-2.10\bin\sn.bat' -R .\FSharp.Core.dll .\mono.snk on a copy of the F# runtime, the GACing step fails : yet the operation doesn't actually change the embedded key (going as far as ILDasm/ILAsm seems a bit extreme!)
As administrator, & 'C:\Program Files\Mono-2.10\bin\mono.exe' 'C:\Program Files\Mono-2.10\lib\mono\2.0\gacutil.exe' -i '.\FSharp.Core.dll', and then copy the original Fsharp.core.sigdata and .optdata alongside
in C:\Program Files\Mono-2.10\lib\mono\3.5\Microsoft.Common.targets undo the commenting out of _ComputeNonExistentFileProperty
In the copy of the Microsoft.Sharp.targets file, comment out the <ItemGroup Remove="..."> clauses after Fsc task in the CoreCompile target, and the two in CreateManifestResourceNames (might there be a CreateItem equivalent? Haven't tried yet).

At this point, pure F# projects for VS2008 will build happily, so long as you don't have any fancy modern task attributes in BeforeBuild or AfterBuild (e.g. Error executing task Copy: Task does not have property "OverwriteReadOnlyFiles" defined if you try to use that).

Where I have yet to manage to persuade the build process to work is for projects with embedded resources (such as Glade XML definitions for a UI). These yield up

        Target CopyNonResxEmbeddedResources:
C:\PROGRA~1\Mono-2.10\lib\mono\3.5\Microsoft.Common.targets: error : 
You must specify DestinationFolder or DestinationFiles attribute.
        Task "Copy" execution -- FAILED

which may possibly relate to some of the <ItemGroup Remove="..."> operations not being carried out. Certainly, some more MSBuild hackery will be required to track down what is going on and make sure that both sides of the copy operation match up -- so that either the source is empty, and no copy is attempted, or so that the destination doesn't end up null.

Support for VS2010 projects isn't there yet -- after sn -R (with equally little effect on the strong-name given by sn -Tp) and GACing the 4.0 version of FSharp.Core.dll, trying to build a VS2010 project yields up the cryptic message, sometimes as an error, sometimes as a warning, that "Operation is not valid due to the current state of the object" -- even with /verbosity:diagnostic turned on no more detail is divulged. And having the 4.0 version GAC'd interferes with the 2.0 version in the GAC -- the later version is selected by preference and then fails for being incompatible with the 2.0-level mscorlib when building a vanilla VS2008 project with just a simple reference (without explicit version information like <Reference Include="FSharp.Core" />) to the F# runtime.

ILSpy and F#

2011-02-18T07:22:00.005Z

Suddenly a new contender in the post-Reflector age -- ILSpy, emerging from the SharpDevelop stable.

So I decided to give it a whirl with the same F# code as with the simple tests I made last week with Cecil.Decompiler -- a few extension methods for Option

module Augment =   

  type Microsoft.FSharp.Core.Option<'T> with
      static member filter (f : 'T -> bool) (x : option<'T>) =
        match x with
        | Some v when f(v) -> Some v
        | _ -> None
        
      static member getOrElse (fallback : 'T) (x : option<'T>) = defaultArg x fallback

      static member select (f : 'T -> bool) (x : 'T) =
        if f(x) then Some x
        else None
        
      static member nullable (x : 'a when 'a : null) : option<'a> =
        if x <> null then Some x
        else None

The first thing I notice is that I can't copy and paste code from the decompilation screen -- I have to save it, and which point I get a button to open Explorer in the same directory as I saved. The C# code generated looks sane

public static FSharpOption<T> Option`1.select.Static(FSharpFunc<T, bool> f, T x)
{
 if (f.Invoke(x))
 {
 }
 else
 {
  goto Block_3;
 }
 return FSharpOption<T>.Some(x);
 Block_3:
 return null;
}

Unlike Cecil, this gets the F# branching right in representing the highlighted section

        IL_0000: nop
        IL_0001: ldarg.0
        IL_0002: ldarg.1
        IL_0003: callvirt !1 Microsoft.FSharp.Core.FSharpFunc`2<T,System.Boolean>::Invoke(!0)
        IL_0008: brfalse.s IL_000c
        IL_000a: br.s IL_000e
        IL_000c: br.s IL_0015
        IL_000e: ldarg.1
        IL_000f: call Microsoft.FSharp.Core.FSharpOption`1<!0> Microsoft.FSharp.Core.FSharpOption`1<T>::Some(!0)
        IL_0014: ret
        IL_0015: ldnull
        IL_0016: ret

It also handles compiler generated temporaries just fine -- albeit with the same 'C'-style if

public static FSharpOption<T> Option`1.filter.Static(FSharpFunc<T, bool> f, FSharpOption<T> x)
{
 FSharpOption<T> fSharpOption = x;
 if (fSharpOption)
 {
  FSharpOption<T> fSharpOption2 = fSharpOption;
  T t = fSharpOption2.Value;
  if (f.Invoke(t))
  {
   goto Block_4;
  }
 }
 return null;
 Block_4:
 return FSharpOption<T>.Some(fSharpOption2.Value);
}

And -- bonus -- while Cecil choked on this C# code

private static void DumpAssembly(string path)
  {
   var assembly = AssemblyDefinition.ReadAssembly(path);
   var pdbpath = Path.ChangeExtension (path, ".pdb");
   var provider = new PdbReaderProvider();
            var reader = provider.GetSymbolReader(assembly.MainModule, pdbpath);
            assembly.MainModule.ReadSymbols(reader);

   assembly.MainModule.Types.SelectMany(x => x.Methods).ToList().ForEach(Decompile);    
  }

ILSpy comes up with

private static void DumpAssembly(string path)
{
 AssemblyDefinition assemblyDefinition = AssemblyDefinition.ReadAssembly(path);
 string text = Path.ChangeExtension(path, ".pdb");
 ISymbolReader symbolReader = new PdbReaderProvider().GetSymbolReader(assemblyDefinition.MainModule, text);
 assemblyDefinition.MainModule.ReadSymbols(symbolReader);
 var arg_5F_0 = assemblyDefinition.MainModule.Types;
 if (!Program.CS$<>9__CachedAnonymousMethodDelegate1)
 {
  Program.CS$<>9__CachedAnonymousMethodDelegate1 = delegate(TypeDefinition x)
  {
   IEnumerable<MethodDefinition> enumerable = x.Methods;
   return enumerable;
  }
  ;
 }
 Enumerable.ToList<MethodDefinition>(Enumerable.SelectMany<TypeDefinition, MethodDefinition>(arg_5F_0, Program.CS$<>9__CachedAnonymousMethodDelegate1)).ForEach(new Action<MethodDefinition>(Program.Decompile));
}

which leaks some of the internal plumbing -- delegate caching; how extension methods are actually handled -- but does at least deliver.

Scripting the Win32 API - F# and IronPython FFI

2011-02-14T22:26:00.002Z

Mainly a worked example for reminding me of how the syntax goes, reimplementing sn -k in F# and IronPython:

open System
open System.Diagnostics
open System.IO
open System.Runtime.InteropServices

#nowarn "51"
// "The use of native pointers may result in unverifiable .NET IL code" --
// for arguments &&keyBlob, &&generatedSize in the call to StrongNameKeyGenEx

module NativeMethods =
    [<DllImport("mscoree.dll")>]
    extern int StrongNameErrorInfo()
    
    [<DllImport("mscoree.dll")>]
    extern bool StrongNameKeyGenEx(
                  [<MarshalAs(UnmanagedType.LPWStr)>]String wszKeyContainer,
                    int dwFlags, int dwKeySize, IntPtr* ppbKeyBlob, int* pcbKeyBlob)
                  
    [<DllImport("mscoree.dll")>]
    extern void StrongNameFreeBuffer(IntPtr pbMemory)

let GetLastStrongNameError() =
    Marshal.GetExceptionForHR(NativeMethods.StrongNameErrorInfo()).Message

let GenerateKeyPair (keyK : int) =
    if keyK < 0 || keyK > 2 then
        raise (new ArgumentOutOfRangeException("keyK", keyK, "Invalid Key Size -- should be 1 or 2"))
               
    // variables that hold the unmanaged key
    let mutable keyBlob = IntPtr.Zero
    let mutable generatedSize = 0

    // create the key
    let createdKey = NativeMethods.StrongNameKeyGenEx(
                                   null, 0, 1024 * keyK,
                                   &&keyBlob, &&generatedSize)
    try
       // if there was a problem, translate it and report it
       if not createdKey || keyBlob = IntPtr.Zero then
           raise (new InvalidOperationException(GetLastStrongNameError()))

       // make sure the key size makes sense
       if generatedSize <= 0 || generatedSize > Int32.MaxValue then
           raise (new InvalidOperationException("InternalError"))

       // get the key into managed memory
       let key = Array.create generatedSize 0uy
       Marshal.Copy(keyBlob, key, 0, generatedSize)
       key
    finally
       // release the unmanaged memory the key resides in
       if keyBlob <> IntPtr.Zero then
           NativeMethods.StrongNameFreeBuffer(keyBlob)
           
// path to generate new string name key
let keypath = fsi.CommandLineArgs.[1]

// write the key to the specified file if it is not already present
if not (File.Exists(keypath)) then
    let key = GenerateKeyPair 1
    use snkStream = new FileStream(keypath, FileMode.Create, FileAccess.Write)
    use snkWriter = new BinaryWriter(snkStream)
    snkWriter.Write(key)

This about halfway between how you'd just call the APIs naturally in C++/CLI, and the full process of C# P/Invoke. You do need to do your own extern method declarations, whereas IronPython is something else again:

from System import Int32, Byte, IntPtr
import System
from System.IO import *
from System.Runtime.InteropServices import Marshal
from ctypes import *
import sys

mscoree = windll.mscoree

def GetLastStrongNameError():
    error = Marshal.GetExceptionForHR(mscoree.StrongNameErrorInfo())
    return error.Message

def GenerateKeyPair (keyK):
    if keyK < 0 or keyK > 2 :
        raise System.ArgumentOutOfRangeException(
        "keyK", keyK, "Invalid Key Size -- should be 1 or 2")
               
    # variables that hold the unmanaged key
    keyBlob = c_void_p()
    generatedSize = c_uint(0)

    # create the key
    createdKey = mscoree.StrongNameKeyGenEx(
                                   None, 
                                   c_uint(0),
                                   c_uint(1024 * keyK),
                                   byref(keyBlob),
                                   byref(generatedSize))

    try:
       # if there was a problem, translate it and report it
       if not createdKey or keyBlob.value == 0:
           raise  System.InvalidOperationException(GetLastStrongNameError())

       #make sure the key size makes sense
       generatedSize = generatedSize.value
       if generatedSize <= 0 or generatedSize > Int32.MaxValue:
           raise System.InvalidOperationException("InternalError")

       # get the key into managed memory
       key = System.Array.CreateInstance(Byte, generatedSize)
       Marshal.Copy(IntPtr(keyBlob.value), key, 0, generatedSize)
       return key
    finally:
       # release the unmanaged memory the key resides in
       if keyBlob.value != 0:
           mscoree.StrongNameFreeBuffer(keyBlob)
           
# write the key to the specified file if it is not already present
keypath = sys.argv[1]
if not (File.Exists(keypath)):
    print 'Generating key at %s' % (keypath)
    key = GenerateKeyPair (1)
    # should do this using a 'with'
    snkStream = FileStream(keypath, FileMode.Create, FileAccess.Write)
    snkWriter = BinaryWriter(snkStream)
    snkWriter.Write(key)
    snkWriter.Close()

where it was easier to fudge the pointer value for the key buffer into an IntPtr than to try and dereference it via ctypes and make a manual copy.

The code here is an adapted subset of a managed API for strong-name keys; the use-case being a contingent key generation as part of a build process.

Signs of Spring

2011-02-12T23:52:00.004Z

After the front-loaded winter in November and December, mild and often wet weather has brought the snowdrops to their height now, and in recent days crocuses and early irises too; even in sunny spots, early primroses, while the viburnum continues.

I cycled into town today to give my newly rebuilt bike (a full 3 star service from the new bike shop near work) a try-out; and having put on a winter-weight jacket was quite hot by the time I got there. So I stuffed the jacket in one pannier and was nigh-on the only person around with T-shirt and shades -- heavy coats and jackets were everywhere. Indeed so mild it was that I was comfortable cycling home without putting the jacket back on as the sun occasionally went behind the first real fair-weather cumulus I've spotted this year.

Other notes -- 6 months with the new car and a whisker under 2200 miles done on it, allowing for the clock showing 14 when I took delivery (from parking at home a fortnight ago with exactly 2000 showing).

MSBuild -- getting the assemblies you depend on from other projects

2011-02-12T23:42:00.002Z

So in one project, you'd like to do something where you automatically pick up the other assemblies you've built in projects you depend on -- an obvious case in point being running some automated test based on the dependent projects. Just going for the list of references will pick you up a whole raft of system assemblies that you can't get rid of in .net4; so you need to turn the project references into the output names.

For that we have the MSBuild task and the GetTargetPath target, thus:

  <PropertyGroup>
    <TargetList>@(Targets->'%(filename)%(extension)', '; ')</TargetList>
  </PropertyGroup>
  <Target Name="BeforeBuild">
     <MSBuild
    Projects="@(ProjectReference)"
    Targets="GetTargetPath">
    <Output TaskParameter="TargetOutputs" ItemName="Targets" />
    </MSBuild>
    <Message Text="My reference list is $(TargetList)" />
  </Target>

which is a trivial example where we just echo the values to the console, like

BeforeBuild:
  My reference list is AnotherAssembly.dll; NewAssembly.dll

Getting the public key and token from a strong-name key

2011-02-11T00:08:00.002Z

I hope we're all familiar with sn -Tp to get the public key details from an assembly -- here using an example where the key is public and anyone can reproduce the results:

>sn -Tp .\Mono.Cecil.dll

Microsoft (R) .NET Framework Strong Name Utility  Version 3.5.30729.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Public key is
002400000480000094000000060200000024000052534131000400000100010079159977d2d03a
8e6bea7a2e74e8d1afcc93e8851974952bb480a12c9134474d04062447c37e0e68c080536fcf3c
3fbe2ff9c979ce998475e506e8ce82dd5b0f350dc10e93bf2eeecf874b24770c5081dbea7447fd
dafa277b22de47d6ffea449674a4f9fccf84d15069089380284dbdd35f46cdff12a1bd78e4ef00
65d016df

Public key token is 0738eb9f132ed756

But if you haven't built anything yet, you might still want these values from a string-name key pair (e.g. to put in an InternalsVisibleTo attribute on the first assembly at the bottom of the calling stack for the unit tests that are to come). So here's how

open System
open System.IO
open System.Reflection
open System.Security.Cryptography

let key = fsi.CommandLineArgs.[1]
let stream = new System.IO.FileStream(key, System.IO.FileMode.Open, System.IO.FileAccess.Read)
let pair = new StrongNameKeyPair(stream)

// get the public key token as 8 bytes from the end of a SHA1 hash of the key material
let hash = new System.Security.Cryptography.SHA1CryptoServiceProvider()
let token = hash.ComputeHash(pair.PublicKey)
                |> Array.rev
                |> Seq.take 8
                
  
let patchArray s =
  s
  |> Seq.map (fun (x:byte) -> x.ToString("X2").ToLower() )
  |> Seq.toList

let rec chunkArray (s:string list) =
  let chunk = 39
  s
  |> Seq.truncate chunk
  |> Seq.iter Console.Write
  Console.WriteLine(String.Empty)
  if chunk > (List.length s ) then ()
  else  s
        |> Seq.skip chunk
        |> Seq.toList
        |> chunkArray

let dumpArray s =
  s
  |> patchArray
  |> chunkArray 
  
Console.WriteLine("Public key is")
pair.PublicKey
|> dumpArray
Console.WriteLine(String.Empty)

Console.Write("Public key token is ")
token
|> dumpArray
Console.WriteLine(String.Empty)

so now we can do

>& 'C:\Program Files\FSharp-2.0.0.0\bin\fsi.exe' .\sntp.fsx .\mono.snk
Public key is
002400000480000094000000060200000024000052534131000400000100010079159977d2d03a
8e6bea7a2e74e8d1afcc93e8851974952bb480a12c9134474d04062447c37e0e68c080536fcf3c
3fbe2ff9c979ce998475e506e8ce82dd5b0f350dc10e93bf2eeecf874b24770c5081dbea7447fd
dafa277b22de47d6ffea449674a4f9fccf84d15069089380284dbdd35f46cdff12a1bd78e4ef00
65d016df

Public key token is 0738eb9f132ed756

and get the values we want. As most of the time you'll want the public key as a continuous string, much of the format nonsense in the script can be dispensed with.

Cecil.Decompiler and F#

2011-02-10T07:00:00.004Z

In this new post-Reflector age, I thought I'd have a look-see how the main competition worked on F#. So I whipped up a simple driver for Cecil.Decompiler:

  private static void Decompile(MethodDefinition m)
  {
   try {
    Formatter.WriteMethodBody(Console.Out, m);

   var lang = CSharp.GetLanguage(CSharpVersion.V3);
   var writer = lang.GetWriter(new PlainTextFormatter(Console.Out));
   writer.Write(m);
   } catch (Exception e) {
    Console.WriteLine(m.FullName);
    Console.WriteLine(e.Message);
   }
  }
  
  private static void DumpAssembly(string path)
  {
   var assembly = AssemblyDefinition.ReadAssembly(path);
   var pdbpath = Path.ChangeExtension (path, ".pdb");
   var provider = new PdbReaderProvider();
            var reader = provider.GetSymbolReader(assembly.MainModule, pdbpath);
            assembly.MainModule.ReadSymbols(reader);

   assembly.MainModule.Types.SelectMany(x => x.Methods).ToList().ForEach(Decompile);    
  }

and tried it on an assembly of some fairly simple F# -- a few extension methods for Option

module Augment =   

  type Microsoft.FSharp.Core.Option<'T> with
      static member filter (f : 'T -> bool) (x : option<'T>) =
        match x with
        | Some v when f(v) -> Some v
        | _ -> None
        
      static member getOrElse (fallback : 'T) (x : option<'T>) = defaultArg x fallback

      static member select (f : 'T -> bool) (x : 'T) =
        if f(x) then Some x
        else None
        
      static member nullable (x : 'a when 'a : null) : option<'a> =
        if x <> null then Some x
        else None

and we get things like

public static FSharpOption<T> Option`1.select.Static(FSharpFunc<T, bool> f, T x)
{
        if (f.Invoke(x))
        {
        }
        return FSharpOption<T>.Some(x);
        return null;
}

This is better than Reflector was when I first tried it about 18 months ago -- F#'s cluster of branch instructions

        IL_0000: nop
        IL_0001: ldarg.0
        IL_0002: ldarg.1
        IL_0003: callvirt !1 Microsoft.FSharp.Core.FSharpFunc`2<T,System.Boolean>::Invoke(!0)
        IL_0008: brfalse.s IL_000c
        IL_000a: br.s IL_000e
        IL_000c: br.s IL_0015
        IL_000e: ldarg.1
        IL_000f: call Microsoft.FSharp.Core.FSharpOption`1<!0> Microsoft.FSharp.Core.FSharpOption`1<T>::Some(!0)
        IL_0014: ret
        IL_0015: ldnull
        IL_0016: ret

caused the old Reflector to just crash.

F#'s habit of lacing in temporaries did flummox the decompilation a bit:

public static FSharpOption<T> Option`1.filter.Static(FSharpFunc<T, bool> f, FSharpOption<T> x)
{
        if (x)
        {
                FSharpOption<T>  = ;
                T v = .get_Value();
                if (f.Invoke(v))
                {
                }
                T v = .get_Value();
                return FSharpOption<T>.Some(v);
        }
        return null;
}

compared with the current Reflector's take of

public static FSharpOption<T> Option`1.filter.Static<T>(FSharpFunc<T, bool> f, FSharpOption<T> x)
{
    FSharpOption<T> option = x;
    if (option != null)
    {
        FSharpOption<T> option2 = option;
        T v = option2.get_Value();
        if (f.Invoke(v))
        {
            return FSharpOption<T>.Some(option2.get_Value());
        }
    }
    return null;
}

MonoDevelop 2.4.2.'s Assembly Browser gets it a bit better than the raw decompiler, though it still has the same 'C'-style if in there:

public static FSharpOption<T> Option`1.filter.Static(FSharpFunc<T, bool> f, FSharpOption<T> x)
{
 if (x)
 {
  FSharpOption<T> V_1 = V_0;
  T V_2 = V_1.get_Value();
  if (f.Invoke(V_2))
  {
  }
  T V_3 = V_1.get_Value();
  return FSharpOption<T>.Some(V_3);
 }
 return null;
}

Amusingly, I was able to get the decompilation to throw by feeding the little driver program into itself -- the DumpAssembly method would not turn back into C#! MonoDevelop silently refused to load the contents of the whole assembly.

Getting the baked-in .pdb location from an assembly with Mono.Cecil 0.9.4

2011-02-06T14:35:00.002Z

Because it isn't always as simple as

  public static string GetPdbFileName (string assemblyFileName)
  {
   return Path.ChangeExtension (assemblyFileName, ".pdb");
  }

as is done in Mono.Cecil.Pdb.PdbHelper, if assemblies and symbols have been moved to separate locations during a build.

The tools are there -- we just need to get the debug data from the PE image if it's present, skip the first 24 bytes, and interpret the rest as a string. Alas, all the are annoyingly just slightly encapsulated from us. But never mind! Reflection gets us there without having to negotiate a patch or make a fork:

        /// <summary>
        /// Violate Cecil encapsulation to get the PDB path -- a candidate
        /// for another method on ModuleDefinition
        /// </summary>
        /// <param name="assembly">The assembly to find the .pdb path for</param>
        /// <returns>The path (null if the operation fails)</returns>
        public static string GetPdbFromImage(AssemblyDefinition assembly)
        {
            var m = assembly.MainModule;
            var imageField = typeof(ModuleDefinition).GetField("Image", BindingFlags.Instance | BindingFlags.NonPublic);
            var image = imageField.GetValue(m);
            var getDebugHeaderInfo = image.GetType().GetMethod("GetDebugHeader");
            byte[] result = null;
            var args = new object[] { result };
    
            try 
            {
                getDebugHeaderInfo.Invoke(image, args);
                var SizeOfDebugInfo = 0x18;
                result = args[0] as byte[];
                if (null == result)
                {
                    return null;
                }

                var byteValue = result.Skip(SizeOfDebugInfo).
                    TakeWhile(x => x != 0).ToArray();

                if (byteValue.Length > 0)
                {
                    // UTF-8 encoding works for an assembly named GetÞePdbLocation.
                    return Encoding.UTF8.GetString(byteValue);
                }
            }
            catch (TargetInvocationException)
            {}

            return null;
        }

Monad.Zero -- doing it right

2011-01-19T22:26:00.005Z

For my own benefit (see title of blog) as much as anything else. When I wrote

Rule 1 of monadic parsers is -- if you return Monad.Zero, you're probably doing it wrong... It's likely that you mean monadic return of an empty list.

it's because there's a distinction between positively returning an empty result (i.e. successfully doing nothing) and returning nothing at all (not succeeding in what you were trying to do) -- as the F# documentation for Zero puts it

Called for empty else branches of if...then expressions in computation expressions.

So, for a parser monad Zero is the function that returns a parser that takes no input and always fails:

    public static Parser<TInput, TOutput> Zero<TInput, TOutput>()
    {
        Operation<TInput, TOutput> bound = cs =>
        {
            return new List<KeyValuePair<TOutput, IEnumerable<TInput>>>();
        };
        return bound.ToParser();
    }

for the Enumerable or Seq monad, it's the empty sequence, for Maybe its None and so forth -- basically the only thing you can return if you don't have any value to return.

And for individual leaf parsers -- the functions that define an Operation<TInput, TOutput> and return them wrapped as a Parser<TInput, TOutput>, those Operation<TInput, TOutput>s should return the same non-value when faced with a parse failure.

By contrast, when composing a parser that may try to consume some of the input, but fail, and you want to not fail the whole parse, you need a parser that successfully matches nothing (and consumes nothing) as an alternative that provides a success value -- and that is not Zero, but rather Return of the non-value of the appropriate type. So, for example, the parser combinator Many:

        public static Parser<TInput, List<TOutput>> Many<TInput, TOutput>(Parser<TInput, TOutput> p)
        {
            var arg1 = Many1(p);
            var arg2 = ParserMonad.Return<TInput, List<TOutput>>(new List<TOutput>());
            return Combinators.Oppp(arg1, arg2);
        }

is "Consume as many blocks of p as you can -- or, if that fails, do nothing"; by contrast, the combinator Sat fails if the predicate doesn't match:

        public static Parser<TInput, TOutput> Sat<TInput, TOutput>(Predicate<TInput> p, Func<TInput, TInput, TOutput> select)
        {
            return Item<TInput, TInput>(x => x).SelectMany(c =>
            {
                if (p(c))
                    return ParserMonad.Return<TInput, TInput>(c);
                else
                    return ParserMonad.Zero<TInput, TInput>();
            }, select);
        }

So in that case it says Zero and means Zero because that's a failure to satisfy the parser -- here in the equivalent the F# computation expression we don't even have to write the else branch; it's filled in for us by the compiler.

And that's really the one subtlety to be borne in mind -- otherwise you end up with an almost intractable debugging issue when your parser fails unexpectedly, partway through consuming valid input.

Monadic Parsers in C#

2011-01-18T20:02:00.003Z

Updated 17-Jan -- because I realised I'd gotten Input and Output the wrong way around; too easy to do when you're using the same type in each case.

Updated 18-Jan -- In which we bug-fix the heck out of the previous two iterations, and pretty much completely rewrite this post. Rule 1 of monadic parsers is -- if you return Monad.Zero, you're probably doing it wrong... It's likely that you mean monadic return of an empty list.

Following on from Mike Hadlow's recent series of posts on monads in C#, and the "Aha!" moment that SelectMany actually means Bind, in the same way that Select means map, I thought I'd take it for a spin on something more than just the usual Identity, Maybe and State monads.

For a long while now there have been things like illustrative F# ports of a Haskell monadic parser -- so why not, I thought, try to emulate that in C#?

So we have a type which looks like this (abusing KeyValuePair as a 2-tuple for .net 3.5 compatibility) --

    public delegate List<KeyValuePair<TOutput, IEnumerable<TInput>>> Operation<TInput, TOutput>(IEnumerable<TInput> seq);

    public struct Parser<TInput, TOutput>
    {
        public Operation<TInput, TOutput> Op;
    }

which is a monad upon the return type; the input stream parametrization being orthogonal to this.

The monad itself (and here I've reverted to using Bind for the simple case analogous to the F# case) looks like

    public static class ParserMonad
    {
        public static Parser<TInput, TOutput> ToParser<TInput, TOutput>(this Operation<TInput, TOutput> value)
        {
            return new Parser<TInput, TOutput> { Op = value };
        }

        public static Parser<TInput, UOutput> Bind<TInput, TOutput, UOutput>(
            this Parser<TInput, TOutput> value,
            Func<TOutput, Parser<TInput, UOutput>> operation)
        {
            Operation<TInput, UOutput> bound = cs =>
            {
                var r = value.Op(cs);
                var rr = r.Select(kv => operation(kv.Key).Op(kv.Value));
                return rr.SelectMany(x => x).ToList();
            };

            return bound.ToParser();
        }

        public static Parser<TInput, VOutput> SelectMany<TInput, TOutput, UOutput, VOutput>(this Parser<TInput, TOutput> value,
            Func<TOutput, Parser<TInput, UOutput>> operation,
            Func<TOutput, UOutput, VOutput> select)
        {
            return value.Bind(t =>
                    operation(t).Bind(u => 
                        ParserMonad.Return<TInput, VOutput>(select(t, u))
                        ));
        }

        public static Parser<TInput, TOutput> Return<TInput, TOutput>(TOutput value)
        {
            Operation<TInput, TOutput> bound = cs =>
            {
                var tmp = new KeyValuePair<TOutput, IEnumerable<TInput>>(value, cs);
                return new List<KeyValuePair<TOutput, IEnumerable<TInput>>> { tmp };
            };
            return bound.ToParser();
        }

        public static Parser<TInput, TOutput> Zero<TInput, TOutput>()
        {
            Operation<TInput, TOutput> bound = cs =>
            {
                return new List<KeyValuePair<TOutput, IEnumerable<TInput>>>();
            };
            return bound.ToParser();
        }
    }

Here ToParser is the convenience function-to-monad constructor; and Return is what lifts an output value into the monad -- creating a parser that places that output value into the output stream.

The combinators assemble all the parsers from the atomic units -- some here are more specialized than others.

    public static class Combinators
    {
        /// Non-deterministic choice -- The (++) operator
        /// is a (non-deterministic) choice operator for parsers. The parser p ++ q applies
        /// both parsers p and q to the argument string, and appends their list of results.
        public static Parser<TInput, TOutput> Opp<TInput, TOutput>(Parser<TInput, TOutput> p, Parser<TInput, TOutput> q)
        {
            Operation<TInput, TOutput> bound = cs =>
            {
                return p.Op(cs).Concat(q.Op(cs)).ToList();
            };
            return bound.ToParser();
        }

        /// a deterministic)choice operator (+++) that has
        /// the same behaviour as (++), except that at most one result is returned:
        public static Parser<TInput, TOutput> Oppp<TInput, TOutput>(Parser<TInput, TOutput> p, Parser<TInput, TOutput> q)
        {
            Operation<TInput, TOutput> bound = cs =>
            {
                var sum = Opp(p, q).Op(cs);
                if (sum.Any())
                {
                    return new List<KeyValuePair<TOutput, IEnumerable<TInput>>> { sum.First() };
                }

                return new List<KeyValuePair<TOutput, IEnumerable<TInput>>>();
            };

            return bound.ToParser();
        }

        /// Matches 1 or more applications of parser p
        /// f# prototype is parser['a, 'b] -> parser['a list, 'b]
        public static Parser<TInput, List<TOutput>> Many1<TInput, TOutput>(Parser<TInput, TOutput> p)
        {
            var parser = p.Bind(x =>
            {
                var manyp = Many(p);
                var outer = manyp.Bind(xs =>
                {
                    var tmp = new List<TOutput> { x };
                    var result = tmp.Concat(xs).ToList();
                    return ParserMonad.Return<TInput, List<TOutput>>(result);
                });
                return outer;
            });

            return parser;
        }

        /// Matches 0 or more applications of parser p
        /// f# prototype is parser['a, 'b] -> parser['a list, 'b]
        public static Parser<TInput, List<TOutput>> Many<TInput, TOutput>(Parser<TInput, TOutput> p)
        {
            var arg1 = Many1(p);
            var arg2 = ParserMonad.Return<TInput, List<TOutput>>(new List<TOutput>());
            return Combinators.Oppp(arg1, arg2);
        }

        /// Apply a parser
        public static Parser<TInput, TOutput> Apply<TInput, TOutput>(Parser<TInput, TOutput> p)
        {
            var parser = p.Bind(x => ParserMonad.Return<TInput, TOutput>(x));
            return parser;
        }

        /// A parser which successfully consumes the first element
        /// of the input stream if it is is non-empty, and fails otherwise.
        /// f# prototype is (unit ->) parser[char, string]
        public static Parser<TInput, TOutput> Item<TInput, TOutput>(Func<TInput, TOutput> select)
        {
            Operation<TInput, TOutput> bound =
                    cs =>
                    {
                        if (cs.Any())
                        {
                            return new List<KeyValuePair<TOutput, IEnumerable<TInput>>> 
                            { 
                                new KeyValuePair<TOutput, IEnumerable<TInput>>(select(cs.First()), cs.Skip(1).ToList()) 
                            };
                        }

                        return new List<KeyValuePair<TOutput, IEnumerable<TInput>>>();
                    };

            return bound.ToParser();
        }

        /// A parser that consumes a single token if it matches the predicate
        /// or fails otherwise
        /// f# prototype is (char->bool) -> parser[char, string]
        public static Parser<TInput, TOutput> Sat<TInput, TOutput>(Predicate<TInput> p, Func<TInput, TInput, TOutput> select)
        {
            var parser = Item<TInput, TInput>(x => x).SelectMany(c =>
            {
                if (p(c))
                {
                    return ParserMonad.Return<TInput, TInput>(c);
                }
                else
                {
                    return ParserMonad.Zero<TInput, TInput>();
                }
            }, select);

            return parser;
        }

        /// A parser for specific characters
        /// f# prototype is char -> parser[char, string]
        public static Parser<char, TOutput> Char<TOutput>(char c, Func<char, char, TOutput> select)
        {
            return Sat(s => s == c, select);
        }

        /// Parse a specific string
        /// f# prototype is string -> parser[char list, string]
        public static Parser<char, List<TOutput>> StringP<TOutput>(string input, Func<char, char, TOutput> select)
        {
            if (input.Any())
            {
                var parser = Char<TOutput>(input.ToCharArray()[0], select).Bind(c =>
                {
                    var p2 = StringP(input.Substring(1), select);
                    var inner = p2.Bind(cs =>
                    {
                        var tmp = new List<TOutput> { c };
                        var result = tmp.Concat(cs).ToList();
                        return ParserMonad.Return<char, List<TOutput>>(result);
                    });
                    return inner;
                });

                return parser;
            }
            else
            {
                return ParserMonad.Return<char, List<TOutput>>(new List<TOutput>());
            }
        }
           
        /// Parse a string of spaces.
        /// f# prototype is (unit ->) parser[char list, string]
        public static Parser<char, List<TOutput>> Space<TOutput>(Func<char, char, TOutput> select)
        {
            return Many(Sat(System.Char.IsWhiteSpace, select));
        }

        /// Parse a token using a parser p, throwing away any trailing space.
        /// f# prototype is parser['a, string] -> parser['a, string]
        public static Parser<char, TOutput> Token<TOutput>(Parser<char, TOutput> p, Func<char, char, TOutput> select)
        {
            var parser = p.Bind(x =>
                        Space(select).Bind(xs =>
                        {
                            return ParserMonad.Return<char, TOutput>(x); 
                        }
                ));
            return parser;
        }

        /// Parse a symbolic token:
        /// f# prototype is string -> parser[char list, string]
        public static Parser<char, List<TOutput>> Symbol<TOutput>(string symbol, Func<char, char, TOutput> select)
        {
            Parser<char, List<TOutput>> inner = StringP<TOutput>(symbol, select);
            return Token<List<TOutput>>(inner, (x, y) => new List<TOutput> { select(x,y) } );
        }

        /// Parse repeated applications of a parser p, separated by applications of a parser
        /// op whose result value is an operator that is assumed to associate to the left,
        /// and which is used to combine the results from the p parsers.
        public static Parser<TInput, TOutput> Chain1<TInput, TOutput> (
            Parser<TInput, TOutput> p,
            Parser<TInput, Func<TOutput, TOutput, TOutput>> op)
        {
            return p.Bind(x => Chain1Rest(x, p, op));
        }

        /// Helper method for Chain1
        public static Parser<TInput, TOutput> Chain1Rest<TInput, TOutput> (
            TOutput a,
            Parser<TInput, TOutput> p,
            Parser<TInput, Func<TOutput, TOutput, TOutput>> op)
        {
            var parser = op.Bind(f => 
                p.Bind(b =>
                    Chain1Rest(f(a, b), p, op)
                ));
            var stop = ParserMonad.Return<TInput, TOutput>(a);

            return Oppp(parser, stop);
        }

//F# Monadic Parser - Calculator Example :))) 

// Grammar
//expr ::= expr addop term | term
//term ::= term mulop factor | factor
//factor ::= digit | ( expr )
//digit ::= 0 | 1 | : : : | 9
//addop ::= + | -
//mulop ::= * | /

        public static Parser<char, Func<double, double, double>> AddOp()
        {
            var plus = Symbol<char>("+", (x, y) => x).Bind(x =>
                ParserMonad.Return<char, Func<double, double, double>>(
                (a, b) => a + b));
            var minus = Symbol<char>("-", (x, y) => x).Bind(x =>
                ParserMonad.Return<char, Func<double, double, double>>(
                (a, b) => a - b));
            return Oppp(plus, minus);
        }

        public static Parser<char, Func<double, double, double>> MulOp()
        {
            var plus = Symbol<char>("*", (x, y) => x).Bind(x =>
                ParserMonad.Return<char, Func<double, double, double>>(
                (a, b) => a * b));
            var minus = Symbol<char>("/", (x, y) => x).Bind(x =>
                ParserMonad.Return<char, Func<double, double, double>>(
                (a, b) => a / b));
            return Oppp(plus, minus);
        }

        public static Parser<char, double> Digit()
        {
            var sat = Sat<char, char>(System.Char.IsDigit, (x, y) => x);
            return Token<char> (sat, (x,y) => x).Bind<char, char, double>
                (x => ParserMonad.Return<char, double>((double)(x - '0')));
        }

        public static Parser<char, double> Expr()
        {
            return Chain1<char, double>(Term(), AddOp());
        }

        public static Parser<char, double> Term()
        {
            return Chain1<char, double>(Factor(), MulOp());
        }

        public static Parser<char, double> Factor()
        {
            var paren = Symbol("(", (x, y) => x).
                        Bind(x => Expr().
                        Bind(n => Symbol(")", (x2, y2) => x2).
                        Bind(z => ParserMonad.Return<char, double>(n))));
            return Oppp(Digit(), paren);
        }
    }

And we can top this off with an example

        public static void Main()
        {
            var restToken = Combinators.Sat<string, int>(x =>
            {
                return x.Length > 2;
            },
            (x,y) => {
                return Math.Max(x.Length, y.Length);
            });
            var tail = Combinators.Many(restToken);
            var data = new List<string> { "the", "quick", "brown", "fox" };

            var results = Combinators.Apply(tail).Op(data);

            if (results.Any())
            {
                results.ForEach(x => Console.WriteLine(
                    "key={0}; value={1}",
                    "[" + string.Join("; ", x.Key.Select(n=>n.ToString()).ToArray()) + "]",
                    "[" + string.Join("; ", x.Value.ToArray()) + "]"));
            }
            else
            {
                Console.WriteLine("empty");
            }

            var calc = "5 * (3 + 4)";
            var summed = Combinators.Apply(Combinators.Expr()).Op(calc.ToCharArray());
            if (summed.Any())
            {
                summed.ForEach(x => Console.WriteLine(
                    "key={0}; value={1}",
                    "[" + x.Key + "]",
                    "[" + string.Join("; ", x.Value.Select(z => z.ToString()).ToArray()) + "]"));
            }
            else
            {
                Console.WriteLine("empty");
            }
        }

which produces a successfully parsed output

key=[3; 5; 5; 3]; value=[]
key=[35]; value=[]

So what lessons do I draw from this exercise?

The big one is that old saying about needing to be twice as clever to debug code as to write it; and code that works with continuation passing can get pretty darn clever. Or, as Eric Lippert put it (in the context of writing asynchronous code in this style in C#):

Remember what I was saying about the pros and cons of CPS?

PRO: Arbitrarily complex and interesting control flows can be built out of simple parts – check.
CON: The reification of control flow via continuations is hard to read and hard to reason about – check.
CON: The code that represents the mechanisms of control flow completely overwhelms the meaning of the code – check.
CON: The transformation of ordinary code control flow into CPS is the kind of thing that compilers are good at, and almost no one else is – check.

The parts are indeed individually simple enough; but all the rest is sadly true. We can see how clever, what the actual power-to-weight ratio is, or how much what actually gets executed differs from what it superficially looks like we've written, by replacing the initialization in ToParser with

{ Op = x => {
                max = Math.Max(max, new System.Diagnostics.StackTrace().GetFrames().Length);
                return value(x); 
            } }

for a static integer value max in the class. Running the test example, we see that the stack depths achieved by these simple parsers are 64 and 95 respectively -- causes and effects are remote from one another (with the side effect that the only code not covered in this belongs to the subtract and divide operator parsers, and the failure handlers in the main program.

Coupled with this is the difficulty of doing anything meaningful in the way of fine-grained TDD in the bootstrap phase -- the individual pieces are not parsers, are comparatively trivial, and even those return functions operating on functions; it's in combination that they do their magic, and there's a lot of code needed just to read one token out of a stream.

Having the F# equivalent to work with -- to provide guidance of what types all the individual pieces were, and then in essence manually de-sugaring the computation expression notation -- was invaluable. But if you have the F# version, there is little need for the transposition except as a five-finger exercise like this.

On the positive side, though, once you have the toolkit like this and it handles even the simple cases, that probably means that it is robust; and then it can pretty much be taken as a black box, for those times when you absolutely have to do it in C# rather than F#.

More Garden Rubbish

2011-01-16T22:27:00.002Z

Insanely mild weather today meant that when I slipped out in bare feet and T-shirt to check the greenhouse, I ended up doing a bit more pottering around in the garden to catch up with some of the tidying from the autumn tasks without feeling the need to wear more.

I did put on a shirt and sandals for the catch-up pruning of the apple trees and digging over the beds where the annual flowers and the broccoli, which didn't survive the harsh weather in Nov-Dec, had been.

Using F# 2.0 Powerpack ArgParser from C# -- (ii)

2011-01-11T23:00:00.004Z

Having resolved the issue of why the "--" separator wasn't triggering (PowerShell was swallowing it before it got to my code) that was stumping me in the previous post, I put together a little fluent wrapper to make this a little easier to consume from C#. The code that isolates the F# types looks like this:

namespace Tinesware.ArgParser
{
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using Microsoft.FSharp.Core;
    using Microsoft.FSharp.Text;

    /// <summary>
    /// Assembles the various command line options and then calls the parser
    /// </summary>
    public class Options
    {
        /// <summary>
        /// Option values
        /// </summary>
        private List<ArgInfo> options = new List<ArgInfo>();

        /// <summary>
        /// Callbacks for boolean values
        /// </summary>
        private List<Flag> flags = new List<Flag>();

        /// <summary>
        /// Initializes a new instance of the Options class.
        /// </summary>
        public Options()
        {
        }

        /// <summary>
        /// An option with no argument
        /// </summary>
        /// <param name="option">The option string</param>
        /// <param name="callback">The callback for this option</param>
        /// <param name="usage">Usage comment</param>
        /// <returns>This object (fluent interface)</returns>
        public Options Unit(string option, Action callback, string usage)
        {
            var type = ArgType.Unit(FuncConvert.ToFSharpFunc<Unit>(x => callback()));
            this.options.Add(new ArgInfo(option, type, usage));
            return this;
        }

        /// <summary>
        /// An option with a string argument
        /// </summary>
        /// <param name="option">The option string</param>
        /// <param name="callback">The callback for this option</param>
        /// <param name="usage">Usage comment</param>
        /// <returns>This object (fluent interface)</returns>
        public Options String(string option, Action<string> callback, string usage)
        {
            var type = ArgType.String(FuncConvert.ToFSharpFunc<string>(callback));
            this.options.Add(new ArgInfo(option, type, usage));
            return this;
        }

        /// <summary>
        /// An option with an integer argument
        /// </summary>
        /// <param name="option">The option string</param>
        /// <param name="callback">The callback for this option</param>
        /// <param name="usage">Usage comment</param>
        /// <returns>This object (fluent interface)</returns>
        public Options Int(string option, Action<int> callback, string usage)
        {
            var type = ArgType.Int(FuncConvert.ToFSharpFunc<int>(callback));
            this.options.Add(new ArgInfo(option, type, usage));
            return this;
        }

        /// <summary>
        /// An option with a double precision argument
        /// </summary>
        /// <param name="option">The option string</param>
        /// <param name="callback">The callback for this option</param>
        /// <param name="usage">Usage comment</param>
        /// <returns>This object (fluent interface)</returns>
        public Options Float(string option, Action<double> callback, string usage)
        {
            var type = ArgType.Float(FuncConvert.ToFSharpFunc<double>(callback));
            this.options.Add(new ArgInfo(option, type, usage));
            return this;
        }

        /// <summary>
        /// An end-of-arguments argument, usually "--"
        /// </summary>
        /// <param name="option">The option string, usually "--"</param>
        /// <param name="callback">The callback for the remaining arguments</param>
        /// <param name="usage">Usage comment</param>
        /// <returns>This object (fluent interface)</returns>
        public Options Rest(string option, Action<string> callback, string usage)
        {
            var type = ArgType.Rest(FuncConvert.ToFSharpFunc<string>(callback));
            this.options.Add(new ArgInfo(option, type, usage));
            return this;
        }

        /// <summary>
        /// An argument indicating true
        /// </summary>
        /// <param name="option">The option string</param>
        /// <param name="callback">The callback for the remaining arguments</param>
        /// <param name="usage">Usage comment</param>
        /// <returns>This object (fluent interface)</returns>
        public Options Set(string option, Action<bool> callback, string usage)
        {
            var flag = new Flag 
            { 
                Reference = new FSharpRef<bool>(false), 
                Callback = callback 
            };

            var type = ArgType.Set(flag.Reference);
            this.options.Add(new ArgInfo(option, type, usage));
            this.flags.Add(flag);
            return this;
        }

        /// <summary>
        /// An argument indicating false
        /// </summary>
        /// <param name="option">The option string</param>
        /// <param name="callback">The callback for the remaining arguments</param>
        /// <param name="usage">Usage comment</param>
        /// <returns>This object (fluent interface)</returns>
        public Options Clear(string option, Action<bool> callback, string usage)
        {
            var flag = new Flag 
            { 
                Reference = new FSharpRef<bool>(true), 
                Callback = callback 
            };

            var type = ArgType.Set(flag.Reference);
            this.options.Add(new ArgInfo(option, type, usage));
            this.flags.Add(flag);
            return this;
        }

        /// <summary>
        /// Parse the environment command line with no default handler or usage comment
        /// </summary>
        public void Parse()
        {
            this.Parse(null, null);
        }

        /// <summary>
        /// Parse the environment command line with no default handler
        /// </summary>
        /// <param name="usage">usage comment</param>
        public void Parse(string usage)
        {
            this.Parse(null, usage);
        }

        /// <summary>
        /// Parse the environment command line with no usage comment
        /// </summary>
        /// <param name="defaultHandler">default handler for unmatched arguments</param>
        public void Parse(Action<string> defaultHandler)
        {
            this.Parse(defaultHandler, null);
        }

        /// <summary>
        /// Parse the environment command line
        /// </summary>
        /// <param name="defaultHandler">default handler for unmatched arguments</param>
        /// <param name="usage">usage comment</param>
        public void Parse(Action<string> defaultHandler, string usage)
        {
            var temp = usage ?? string.Empty;

            var theUsage = string.IsNullOrEmpty(temp.Trim()) ?
                FSharpOption<string>.None :
                new FSharpOption<string>(usage);

            var theDefault = defaultHandler == null ?
                FSharpOption<FSharpFunc<string, Unit>>.None :
                new FSharpOption<FSharpFunc<string, Unit>>(
                    FuncConvert.ToFSharpFunc<string>(defaultHandler));

            ArgParser.Parse(this.options, theDefault, theUsage);

            this.flags.ForEach(x => x.Callback(x.Reference.Value));
        }

        /// <summary>
        /// Report the command usage
        /// </summary>
        public void Usage()
        {
            this.Usage(null);
        }

        /// <summary>
        /// Report the command usage
        /// </summary>
        /// <param name="remark">Initial comment</param>
        public void Usage(string remark)
        {
            var temp = remark ?? string.Empty;

            var theUsage = string.IsNullOrEmpty(temp.Trim()) ?
                FSharpOption<string>.None :
                new FSharpOption<string>(remark);

            ArgParser.Usage(this.options, theUsage);
        }

        /// <summary>
        /// Binds a ref bool and a callback
        /// </summary>
        private struct Flag
        {
            /// <summary>
            /// A reference cell with a boolean value
            /// </summary>
            public FSharpRef<bool> Reference;

            /// <summary>
            /// A post-parse callback
            /// </summary>
            public Action<bool> Callback;
        }
    }
}

The calling code, for the same example as before looks like

namespace fsgetopt
{
    class Program
    {
        static void Main(string[] args)
        {
            Action<string> compile = (s => Console.WriteLine("Compiling {0}...", s));
            var outputName = "a.out";
            var verbose = false;
            var warningLevel = 0;
            var rest = new List<string>();
          
            var options = new Tinesware.ArgParser.Options().
                String("-o",
                    x => outputName = x,
                    "Name of the output").
                Set("-v",
                    x => verbose = x,
                    "Display additional information").
                Int("--warn",
                    x => warningLevel = x,
                    "Set warning level").
                Rest("--",
                    x => rest.Add(x),
                    "Stop parsing command line");
          
            options.Parse(compile, "Usage options are:");
            
            Console.WriteLine("outputName = {0}", outputName);
            Console.WriteLine("Verbose = {0}", verbose.Value);
            Console.WriteLine("Warning level = {0}", warningLevel);
            Console.WriteLine("rest = {0}", rest.Count);
        }
    }
}

which is much clearer.

Now edited for FxCop.

Garden Rubbish and Meaningless Numbers

2011-01-10T22:02:00.004Z

You can tell quite how early -- late November -- the big freeze set in, as for the last couple of weekends I was doing things like raking leaves off the lawn, as well as other winter chores like pruning roses and clearing other dead foliage from expected dieback (enough to fill the green bin). The cold was also enough that inside the greenhouse, even under a further bubble-wrap tent with a heater set to frost-free, some of the pelargoniums got scorched -- not something I've had happen before in a winterised greenhouse.

In the weekend clearout, I got rid of the last tomato plants from the tent, harvesting one last tomato. Sunday I baked a crumble and almost used up the last of the apples -- leaving two Charles Ross picked and packed remained in a box of fruit that had mostly spontaneously self-destructed; and a large windfall Bramley (we ate the former raw and baked the latter for dessert today). Many of the picked-as-ripe Bramleys also self-destructed from within in the last couple of weeks; the windfalls were just as durable as the cosseted ones.

And some numbers -- at 08:39-08:40 on 21-Oct-10, the digital clock and odometer coincided; as again, cheating at 09:50 BST on 2-Nov; not cheating at 17:43-40 on 6-Jan-11; 1000 miles on 4-Nov. The bike read 1033.9 miles at the end of the year (and also on the 6 month mark of having the bike computer, 7-Jan).

Using F# 2.0 Powerpack ArgParser from C#

2011-01-10T21:31:00.005Z

This is an update of the technique mentioned in an antique post by Robert Pickering which was referenced by Laurent Le Brun last summer. I've taken Laurent's example of F# usage and ported it to C# in the most direct fashion:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.FSharp.Core;
using Microsoft.FSharp.Text;

namespace fsgetopt
{
    class Program
    {
        static void Main(string[] args)
        {
            Action<string> compile = (s => Console.WriteLine("Compiling {0}...", s));
            var outputName = "a.out";
            var verbose = new FSharpRef<bool>(false);
            var warningLevel = 0;
            ArgInfo[] specs = 
            {
                new ArgInfo("-o", 
                    ArgType.String (FuncConvert.ToFSharpFunc<string> (x => outputName=x)), 
                    "Name of the output"),
                new ArgInfo("-v", 
                    ArgType.Set (verbose), 
                    "Display additional information"),
                new ArgInfo("--warn", 
                    ArgType.Int (FuncConvert.ToFSharpFunc<int> (x => warningLevel=x)), 
                    "Set warning level"),
                new ArgInfo("--", 
                    ArgType.Rest (FuncConvert.ToFSharpFunc<string> (x => Console.WriteLine("rest has {0}", x))), 
                    "Stop parsing command line"),
            };

            var after = FuncConvert.ToFSharpFunc<string>(compile);
            var opt1 = new FSharpOption<FSharpFunc<string, Unit>>(after);
            var opt2 = new FSharpOption<string>("Usage options are:");
            ArgParser.Parse(specs, opt1, opt2);

            Console.WriteLine("outputName = {0}", outputName);
            Console.WriteLine("Verbose = {0}", verbose.Value);
            Console.WriteLine("Warning level = {0}", warningLevel);
        }
    }
}

It is possible to streamline this to hide most of the messy type names inside utilities.

One thing I've not managed to do is get the lambda associated with the ArgType.Rest to fire, even after setting the default handler opt1 to FSharpOption<T>.None. That -- the default handler -- gets fired every time an otherwise unmatched argument is encountered; the ArgType.Rest handler looks from the code like it should "just work" so I'm a bit baffled ATM.

LATER: Mystery solved

Of course, I was testing this at a PowerShell prompt -- and the shell was swallowing the unquoted "--" string : there was no "rest" to operate on.

2010 in media

2010-12-29T17:22:00.003Z

Film

A very disappointing year, with a totally dismal Film Festival, and otherwie only the Chaumet/Tati L'Illusioniste really standing out enough to tempt me to the cinema. I would have gone to see The Story of Kells as part of the festival had it not been in a no-unaccompanied-adult showing; but wasn't motivated to see it when it came around later. And similarly, previous commitments and bad weather relieved me of having to decide whether to make the effort to fit in Rare Exports.

Books

I actually read a few this year that weren't software related. Excluding the holiday reading potboilers (Lindsey Davis' latest Falco, some by the numbers sci-fi,...)

Miéville's Un Lun Dun was a young people's Neverwhere -- some nice ideas, but it didn't quite cohere in the end. And I'm sufficiently bourgeois that the mundane London in the book was a more alien world than Un Lun Dun itself.
Also by Miéville, The City and the City is the first Hugo winner I've read before the award since A Fire Upon The Deep. An interesting conceit of cities wrapped around a police procedural.
Reynold's Terminal World was a vast improvement over his previous House of Suns, being a fairly straightforward adventure yarn in a setting where Vingean Zones of Thought happen on a scale of miles rather than kiloparsecs.
I also finally got around to Nahoko Uehashi's Moribito: Guardian of the Spirit -- and was surprised quite how much the anime had expanded upon the rather short and sparse narrative.

And while I'm here, probably worth warning people about the two weakest of the computing texts I picked up this year, both in the Wrox Professional series:

Professional Iron Python was a very strange book, and it did cover some obscure points of .net/Windows programming; but for IronPython, not good. It read like a draft that had been put together while the language was still in its pre-beta stages and then hurriedly given a Visual Studio 2010/.net 4 gloss. In particular the author repeatedly makes the incorrect assertion that IronPython statements cannot span multiple lines -- which leaves me dubious as to the rest of the content I've not independently verified. Get IronPython in Action instead.
Professional F#2.0 just approaches being content free. It covers very little of the language (no computation expressions, let alone anything like quotations), giving more emphasis to the imperative rather than functional aspects where it does. Get the O'Reilly Programming F# instead.

Anime

2010 was the year where Crunchyroll reached a sufficient point that -- whatever the sordid history of the operation might be -- it was possible to watch a fair selection of series without resorting to fansubs. A few series that I might have watched, or at least sampled (House of Five Leaves, Tatami Galaxy) were streamed by Funimation -- but they don't seem to have cottoned on that the world is more than just Japan and the US.

Going by the metric that "/a/ doesn't talk about good anime" -- where it is obvious whether a title that is not talked about is good or bad -- House of Five Leaves, from a manga by the same author as Ristorante Paradiso is the only one that I feel any regrets about missing.

Apart from titles already blogged, I still have the last episode of Virgin Pomegranate Monster to watch, as well as the concluding cours for Letter Bee REVERSE and Super Robot Wars OG : The Inspector.

And of course no discussion of 2010 in anime would be complete without a mention of the new GAINAX series, Panty & Stocking with Garterbelt, a title that I watched just enough of to feel I could add it to my MAL dropped list (i.e. more than one episode).

A combination of all that is wrong with American cartoons multiplied by "LOL, Japan!", this was the disappointing result of giving a studio of talented animators full freedom to have fun and do what they wanted. Alas, what they wanted to do was act like retarded 12-year-olds -- so if you've grown out of finding messy bodily functions funny, or haven't seen any of the stuff that they're lampooning, and don't get your jollies from hearing cute Japanese girls swearing a blue streak, it ends up by being the saddest waste of talent I've seen in many a year. And another case where I ignored the implication "it's being talked about, ergo it isn't very good", because I couldn't believe it was really that bad.

Alas, it was. In fact, watching the audience reactions as they were trolled mightily at every turn with misleading hints about forthcoming episodes, and desperately tried to make it all make sense, all the way up to the sudden ending with "To be continued in next season" (when Gainax have never done a second season of any of their original anime) was far, far more entertaining than the series itself.

“Hello, OTP!” from F# revisited

2010-12-28T20:40:00.004Z

That previous F# code was just too ugly to leave as it was; so after a little bit of work, I've factored out a general purpose wrapper to convert the Otp.Erlang types into values of a discriminated union on the F# side of the fence. Like this we can directly pattern match, without the descent into massive nesting that we had before, and keep the Otp.Erlang types encapsulated into wrappers for method calls on the Otp types.

With this addition the main program above can be revised to become

namespace ErsharpClient

open System
open Otp
open Tinesware.Unfurling

module Main =
         
  let ConnectTo serverNode cookie =
    let clientNode = new OtpSelf("clientnode", cookie)
    let serverNode = new OtpPeer(serverNode)
    clientNode.connect(serverNode)
    
  let ReceiveRPC (connection : OtpConnection) =
    Wrapper.ToTerm <| connection.receiveRPC()
    
  let MakeRPC (connection : OtpConnection) (moduleName:string) (functionName:string) (arguments: Term seq) =
     connection.sendRPC(moduleName, functionName, arguments |> Seq.map Wrapper.ToOtpObject |> Seq.toArray)
     ReceiveRPC connection
    
  let StartGenServer (connection : OtpConnection)  =
    match MakeRPC connection "mathserver" "start_link" [] with
    | Term.Tuple [ Term.Atom "ok" ; pid ] -> pid
    | Term.Tuple [ Term.Atom "error"; Term.Tuple [ Term.Atom "already_started" ; pid2 ]] -> pid2
    | start -> raise (new System.InvalidOperationException(start.ToString()))
    
  [<EntryPoint>]
  let Main arguments =
    let connection = ConnectTo "servernode@YourNodeNameHere" "cookie"
    let pid = StartGenServer connection
    
    let args = [6I; 9I]
               |> Seq.map Term.Integer
    
    match MakeRPC connection "mathserver" "multiply" args with
    | Term.Tuple [ Term.Atom "ok" ; Term.Integer value ] -> 
      Console.WriteLine("Return Value:" + value.ToString())
    | result -> Console.WriteLine("Failure:" + result.ToString())
    
    0

which looks a lot more like the language we're trying to write in. All the magic happens in this file

namespace Tinesware.Unfurling

open Otp

type public Pid = { Node     : string;
                    Id       : int;
                    Serial   : int;
                    Creation : int }
             
type public Port = { Node     : string;
                     Id       : int;
                     Creation : int }
              
type public Ref = { Node     : string;
                    Creation : int;
                    Ids      : int list }             
              
type public Term =
    | Double of float
    | Integer of bigint
    | List of Term list
    | Tuple of Term list
    | Atom of string
    | Binary of byte[]
    | Pid of Pid
    | Port of Port
    | Ref of Ref

module Wrapper =
    
    let rec ToTerm (term:Erlang.Object) =
        match term with
        | :? Erlang.Atom as atom -> Term.Atom <| atom.atomValue()
        | :? Erlang.Binary as binary -> Term.Binary <| binary.binaryValue()
        | :? Erlang.Double as double -> Term.Double <| double.doubleValue()
        | :? Erlang.List as list -> 
          let contents = list.elements()
                         |> Seq.map ToTerm
                         |> Seq.toList
          Term.List contents
        | :? Erlang.Long as long -> Term.Integer <| bigint (long.longValue())
        | :? Erlang.Pid as pid -> Term.Pid {Node = pid.node(); Id = pid.id(); Serial = pid.serial(); Creation = pid.creation()}
        | :? Erlang.Port as port -> Term.Port {Node = port.node(); Id = port.id(); Creation = port.creation()}
        | :? Erlang.Ref as ref -> Term.Ref {Node = ref.node(); Creation = ref.creation(); Ids =  ref.ids() |> Array.toList }
        | :? Erlang.Tuple as tuple ->
          let contents = tuple.elements()
                         |> Seq.map ToTerm
                         |> Seq.toList
          Term.Tuple contents
          
        | _ -> raise <| new System.InvalidOperationException(term.ToString())
        
    let rec ToOtpObject (term : Term) =
      match term with
      | Term.Atom atom -> new Erlang.Atom(atom) :> Erlang.Object
      | Term.Binary binary -> new Erlang.Binary(binary) :> Erlang.Object
      | Term.Double double -> new Erlang.Double(double) :> Erlang.Object
      | Term.Integer long -> new Erlang.Long(int64 long) :> Erlang.Object
      | Term.List list -> new Erlang.List( list 
                                           |> Seq.map ToOtpObject
                                           |> Seq.toArray ) :> Erlang.Object
      | Term.Pid pid -> new Erlang.Pid(pid.Node, pid.Id, pid.Serial, pid.Creation) :> Erlang.Object
      | Term.Port port -> new Erlang.Port(port.Node, port.Id, port.Creation) :> Erlang.Object
      | Term.Ref ref -> new Erlang.Ref(ref.Node, ref.Ids |> List.toArray, ref.Creation) :> Erlang.Object
      | Term.Tuple tuple -> new Erlang.Tuple ( tuple
                                           |> Seq.map ToOtpObject
                                           |> Seq.toArray ) :> Erlang.Object

Of course, as OTP.Net is some revs behind the current Erlang distro's JInterface, this type mapping is incomplete (no BitStr or Fun types) compared with what we could be using. The main advantage is that it works right now.

“Hello, OTP!” from F#

2010-12-28T16:14:00.002Z

Nothing earth-shattering here, just recording the results of a little bit of playing around with OTP.Net and F#. This is all building on the original work done elsewhere Integrating .NET and Erlang using OTP.NET and Integrating F# and Erlang Using OTP.NET, as a little bit of a five-finger exercise using the OTP framework as now explained in Erlang and OTP in Action.

So, simple gen_server to export the same sort of API as the mathserver example in the former:

%% ---------------------------------------------------------------------
%% File: mathserver.erl
%%
%% This is a simple implementation of a gen_server callback module.

-module(mathserver).

-behaviour(gen_server).

-export([init/1, handle_call/3, handle_cast/2, handle_info/2,
         terminate/2, code_change/3]).
         
-export([
            start_link/0,
            stop/0,
            multiply/2
            ]).

-define(SERVER, ?MODULE).

-record(state, { }).
%%%===================================================================
%%% API
%%%===================================================================


%%--------------------------------------------------------------------
%% @doc Starts the server.
%%
%% @spec start_link() -> {ok, Pid}
%% where
%% Pid = pid()
%% @end
%%--------------------------------------------------------------------
start_link() ->
    gen_server:start_link({local, ?SERVER}, ?MODULE, [], []).
    
%%--------------------------------------------------------------------
%% @doc Stops the server.
%% @spec stop() -> ok
%% @end
%%--------------------------------------------------------------------
stop() ->
    gen_server:cast(?SERVER, stop).

%%--------------------------------------------------------------------
%% @doc multiplies two integers.
%% @spec multiply(X::integer(), Y::integer()) -> {ok, Product}
%% where
%% Product = integer()
%% @end
%%--------------------------------------------------------------------
multiply(X, Y) ->
    gen_server:call(?SERVER, {multiply, {X, Y}}).


%%%===================================================================
%%% gen_server callbacks
%%%===================================================================


init([]) ->
    {ok, #state{}}.

handle_call({multiply, {X, Y}}, _From, State) ->
    Reply = {ok, multiply_impl(X,Y)},
    {reply, Reply, State};
handle_call(_Request, _From, State) ->
    Reply = ok,
    {reply, Reply, State}.

handle_cast(_Msg, State) ->
    {noreply, State}.

handle_info(_Info, State) ->
    {noreply, State}.

terminate(_Reason, _State) ->
    ok.

code_change(_OldVsn, State, _Extra) ->
    {ok, State}.

%%%===================================================================
%%% API internals
%%%===================================================================

multiply_impl(First, Second) ->
    First * Second.

Compile this using erlc and start werl -sname servernode -setcookie cookie. Meanwhile the matching F# program to build and run is

namespace ErsharpClient

open System
open Otp

module Main =

  let (|Atom|_|) text (term : Erlang.Object) =
   match term with
   | :? Erlang.Tuple as tuple ->
    let terms = tuple.elements()
    if terms.Length <> 2 then
      None
    else match terms.[0] with
         | :? Erlang.Atom as atom ->
           if atom.atomValue() = text then Some(terms.[1])
           else None
         | _ -> None
   | _ -> None
         
  let ConnectTo serverNode cookie =
    let clientNode = new OtpSelf("clientnode", cookie)
    let serverNode = new OtpPeer(serverNode)
    clientNode.connect(serverNode)
    
  let StartGenServer (connection : OtpConnection)  =
    connection.sendRPC("mathserver", "start_link", Array.empty<Otp.Erlang.Object>)
    let start = connection.receiveRPC()
    match start with
    | Atom "ok" pid -> pid
    | Atom "error" value -> 
      match value with
      | Atom "already_started" pid2 -> pid2
      | _ -> raise (new System.InvalidOperationException(start.ToString()))
    | _ -> raise (new System.InvalidOperationException(start.ToString()))
    

  [<EntryPoint>]
  let Main arguments =
    let connection = ConnectTo "servernode@YourNodeNameHere" "cookie"
    let pid = StartGenServer connection
    
    let args = [| new Otp.Erlang.Long(6L); new Otp.Erlang.Long(9L)|] :  Otp.Erlang.Object array
    connection.sendRPC("mathserver", "multiply", args);
    let result = connection.receiveRPC()
    
    match result with
    | Atom "ok" value -> Console.WriteLine("Return Value:" + value.ToString())
    | _ -> Console.WriteLine("Failure:" + result.ToString())
    
    0

Matching the dynamic types possible with Erlang, and the F# way with type hierarchies, makes this code somewhat ugly (as in multiply nested decision points and the start of the arrow anti-pattern) when trying to unpick the return values, especially trying to decode the possible return values when requesting that the far end server start up.

Some of the repeated noise has been factored out into the Atom active pattern for dealing with {Reason, Payload}, but the underlying data model in OTP.Net is simply not very F# friendly.

Anime — Giant Killing

2010-12-28T11:00:00.004Z

When this title was announced for the spring season, my reaction was "Oh, football..." and pass it by -- but towards the end of its airing, it had become one of those series that hardly ever got talked about, but always positively when it was mentioned. So, with a fairly thin line-up for autumn, I went back and gave the first episode a try...

And it has turned out to be my pick for the best series of the year, by being so much not what the bulk of anime has become in recent years : the characters are adults, there's no romance, or borderline smut, or cute girls doing things cutely. It also confirms the trend that series where the characters have perceptible noses are ones worth investigating.

The story opens when East Tokyo United, a team in a prolonged slump, bring back a former star player, Takeshi Tatsumi, from England, where he as been coaching amateur (or at least not full-time professional) sides to good effect -- like taking them to the 4th round of the FA Cup. This return as coach after having abandoned the team years before rankles amongst some of the players and many of the fans; the more so when he finally shows up at training, and chooses a starting side mainly from the reserves.

Slowly, with a mixture of shrewd insight, and complete irreverence to the way things are done on and off the pitch (like sloping out of a press conference for an informal chat with the manager of the national team), Tatsumi starts to bring the team together. At last ETU notches up a few wins, and then in the climax of the series, have to face the Osaka Gunners, who in their previous match steamrollered a stronger side than ETU 8-0, in a match that actually spreads over more than 90 minutes of air time.

The manga continues; we have only come half-way through the new season, and there isn't a plot-wrapped-up end. If there were to be a second season, I'd watch it.

Links for 25-Dec Bumper catch-up edition

2010-12-25T15:17:00.002Z

2010 having spanned the run from first integration point through to RTM and GA for XenDesktop 5, I was being increasingly preoccupied as the year went on; and blogging of any form was one of the main casualties. Which means I have a not-so-little list of backlog of things I want to keep track of from the last couple of months:

F#

.net General

Creating Zip files with System.IO.Packaging namespace
Kill That Util Class! with C# extension methods
Debugging with Async -- C#5 CTP
Dynamic Language Integration in a C# World
Algorithmia Source Code released on CodePlex
Mono: What we are Cooking (9-Dec-10)
How to: Determine which Microsoft .Net Framework version and service pack installed?
Decoding clr20r3 .NET exception – using mono cecil
C# Closure Constructors -- from the original Javascript

Build Process

Misc.

F# GUI plumbing with reactive Events

2010-12-25T12:09:00.007Z

My most recent "Aha!" moment with the language is to finally wrap my head around the standard Event module -- a sort of lightweight subset of the Reactive Extensions for .net -- while starting to do some more GUI programming in F#, which is making belated use of the GTK#/Glade spikes I made about a year ago.

It may not make much difference in the simplest cases where

ok_btn.Clicked += new EventHandler(OnButtonClick);

naively went to

ok_btn.Clicked.AddHandler(fun source arguments -> OnButtonClick source arguments)

but which is equivalent to, using the Event module,

ok_btn.Clicked
   |> Event.add (OnButtonClick ok_btn)

At this point the main difference is the change in the function signature -- first, it is a real F# function rather than being forced to use the implicit cast from a fun to a delegate, even if the function signatures look compatible on the surface; second the function passed to Event.add doesn't have the source argument (but, as shown, that can be brought in as a closure).

So that's a small win from the beginning -- but that just scratches the surface. Where it actually shines is where multiple sources of events have to be handled in much the same way. For example, given a menu containing recently accessed files, with selection meaning to re-open that file, as if they had been selected from a file open action

   // Do something only if we actually pick a file
   // Use the built-in identity function as a 'T -> 'U option mapper here
   // for the case 'T being a FileInfo option and 'U being FileInfo
   let click = handler.openButton.Clicked
                 |> Event.map (fun a -> (* 
                           open a file dialog, 
                           get a FileInfo option from the selected file name; 
                           where cancel => None *))
                 |> Event.choose id

   // Selecting an item from the menu                                       
   let select =    
       handler.openMenu.AllChildren
       |> Seq.cast<MenuItem>
       |> Seq.map (fun (i:MenuItem) -> i.Activated
                                       |> Event.map (fun a -> new FileInfo((i.Child :?> Label).Text))
                                       |> Event.filter (fun (i:FileInfo) -> i.Exists))

   // The sum of all these events                                  
   let accumulation = select
                      |> Seq.fold Event.merge click  

   // do things with a file selection event                   
   accumulation |> Event.add UpdateRecentFileList
   accumulation |> Event.add LoadFile
   ...

All the inhomogeneity of the event sources (button click, menu item activation) and their associated EventArgs types is smoothed away by appropriate Event.map calls and the use of closures; different validations are handled by appropriate Event.filter or Event.choose invocations; and then the whole ensemble is gathered together by an appropriate fold with Event.merge as the accumulator.

True, there is nothing here that could not be achieved by making old-style handler functions for AddHandler from equivalent appropriately composed function chains -- the path I was starting to take for this bit of plumbing before I was indirectly reminded of this feature, and dug up this series of posts from a couple of years back (so slightly dated in the details). The big difference that using the Event module makes is that it is more expressive of intent, and separates the concerns -- general event handling plumbing vs this application's business logic -- explicitly.

A white-ish Christmas

2010-12-25T11:41:00.003Z

Last Saturday started dry but cold, though with some ice still on the road outside -- cycling into town in multiple layers, with only my nose exposed, that was very cold for the first couple of miles until I started to radiate enough heat through it. I got back home mid-afternoon, and took a nap, waking to find the world turned white.

So it was a work-from-home start of the week, while waiting for the ungritted roads in the village to be pounded clear; with a brisk yomp to Waitrose to get me out of the house. By Thursday, as the threatened midweek snow had come to nothing, I did go into the office (probably getting a lot less done than had I stayed at home).

Yesterday, I did the approximately biennial feat of cycling into work for the shortest working day -- although the fields were white, and the sky grey, the road was dry, and the weather, although still below zero on the way in, was milder than it had been (showing quite how accustomed we soon become to the cold). And in doing so that put me over the 1000 miles for this half year.

Today is brighter and sunny, but colder. Where I cleared the drive is dry, but the untouched snow is showing no signs of turning to slush like it had a few days ago. I shall have to make my way down the garden to check the greenhouse and the broccoli.

Winter Wonderland

2010-12-07T09:35:00.001Z

The car thermometer showed -6 as I started out to work at about 08:45 this morning, dropping quickly to -8, and then slowly recovering to -7. Isn't it time we had an apology for predictions like this one from ten years ago?

Sometimes, a man's gotta do what a man's gotta do...

2010-11-22T08:12:00.004Z

I encountered this outburst of testosterone fuelled experimentation a couple of weeks ago, and mentioned it to Karen -- who immediately said that this was something that had to be done. So, yesterday I actually made some of the

Manliest Bread EVARR!!!!!111elebenty!!!!!!1 ^*

Bacon. Jalapeno. Cheesy. Beer. Bread. The sheer awesomeness of the concept forced that punctuation, I swear.

And it was a great success.

The recipe (for a bread making machine)

Bread chassis:

1 tsp yeast
1.25 tsp salt
1.5 tsp sugar
350g strong white flour
150g strong wholemeal flour

And in the main mix

85g of the strongest cheddar I had in the house, grated
320ml ale -- I used Greene King IPA

In the candied fruit hopper:

4 rashers of thick-cut bacon, dry-fried (pouring off any oozing water as required) until stiff, rinds removed and chopped into ~1cm bits
As many jalapeno pepper slices as will fit.

Cook on standard raisin bread program. Cool for an hour, then enjoy!

It was wonderfully savoury warm with butter, and made a fine accompaniment to a chilli powered by the fruits of our greenhouse (to the extent of 2/3 of the loaf). Though I couldn't help feeling that there had to be some way to work some garlic into the mix without spoiling the awesome.

^* OK, some pedants will point out that this recipe is just loaded with phytoestrogens. But it's the concept that counts.

Anime — Strike Witches 2

2010-11-14T10:14:00.003Z

As hinted by the ending of the previous series, further adventures of half-dressed alternate- WWII aces; despite not being quite popular enough to stop the original studio going under, the franchise was considered worth taking up.

Unfortunately, much of the charm of the first series -- the copious period references, and character driven episodes -- was lost in favour of even more bath scenes and tacky comedy. When, finally, an attempt at plot and drama wobbled into sight in the last couple of episodes, it didn't have any foundation to rest upon.

Overall, rather disappointing for being pretty much what the premise would lead you to expect, and no more. Will I watch the inevitable third season? Probably.

Links for 8-Nov

2010-11-08T18:06:00.001Z

Asynchronous programming in .net

Async programming in F#2 and the PDC announced C#5 compared.
The F# model in detail.
CTP downloads.
Jon Skeet plays around with the internals of the CTP release here and here.
Related : continuation passing style and asynchrony i.e. what the compiler is doing for you with await.

Also from the PDC -- what next for Silverlight?

Rx Design Guidelines.

SSL/TLS isn't computationally expensive.

NuGet -- gems for .net has a new name.

The Should Assertion Library.

Sandcastle MSBuild integration.

Anime — The Book of Bantorra : Armed Librarians

2010-10-29T18:54:00.004+01:00

A belated review, by some months, belated enough that it's fallen off the Crunchryroll roster.

I tried the first episode late last year, and wasn't impressed, but went back after a lot of talk about how the series was getting awesome later on; and fortunately the awful CGI from the first episode seemed to be a one-off.

In a climate where cute girls doing cute things and high-school romance seemed to have driven almost everything else off the table, Bantorra was a welcome return to an adult cast (with well endowed women) and OTT violence. However shoehorning a series of ten light novels with more plot twists than you could shake a stick at into a 26 + 1 recap episode series didn't quite work out so well in the end.

The setting was interesting and unusual in many ways -- a sort of alternate 1940s, where magic also exists, and people have made up European-style names that look and sound more like keyboard mashing (Olivia Litlets, Parney Pealrmanta, Enrique Bis'hile...); in a world where at death, a memory snapshot fossilizes as a "book" that can be "read" by touching it.

So the big struggle starts off between the Armed Librarians under Acting Director Hamyuts Meseta of the Bantorra Library, and the Church of Drowning in God's Grace, whose noble thoughts about the value of humans is belied by their separating their flock into True Men (at the top), Mock Men (mid-range operatives) and Meats (the masses). And then things get more and more complicated, with -- except in the occasional digressive arc for a bit of back-story -- the villains of the piece changing again and again, and major characters being killed all over the show.

Perhaps had it been a 40-episode series it might not have seemed quite so erratic and all over the place; or maybe it would have still seemed like it was trying to fit in everything and the kitchen sink. Overall a decent enough piece of entertainment and one that at least did all it could to be different from the current pack.

Javascript Airport weather decoder

2010-10-27T19:06:00.004+01:00

While weatherpixie remains on extended hiatus, I have finally put together a simple in-browser replacement. It uses James Padolsey's cross-domain Ajax plug-in for jQuery (indirecting via YQL), a simple parsing script from Manuel Heras, here separated into a stand-alone script, and an iframe-based page to do the work.

The final piece of heavy lifting is to get the appropriate METAR report for the local airfield, and that's just a few lines of javascript --

  $(document).ready(function(){
    var lmurl = "http://weather.noaa.gov/mgetmetar.php?cccc=EGSC&Submit=SUBMIT"
    $.get(lmurl, function(res) { 
          $(res.responseText).find('hr + p').each(function(i, a) {
            var text = a.innerText;
            if (text == undefined) text = a.textContent;
            text = do_metar(text);
            while (text.indexOf('\n') >= 0)
              text = text.replace('\n', '< br />');
            $('#report').html(text);
            });
          });
    });

Here, EGSC is the ICAO code for Cambridge Airport. And that's a lot closer to home than the Met. Office who give out current conditions at Bedford on the Cambridge forecast!

This made me laugh

2010-10-12T18:30:00.000+01:00

Though by rights, it shouldn't -- http://www.telegraph.co.uk/news/8048844/Brits-sense-of-humour-fails-at-the-age-of-52-study-finds.html

Road rage

2010-10-09T15:26:00.005+01:00

Some weeks ago, I was working late, and so cycling home after the usually flurry of commuter traffic.

Coasting one of the downhill parts, doing in the 15-20mph range, I suddenly heard this strange whirring sound behind me -- and then, suddenly, a whole flotilla of sport cyclists.

Now, when they travel solo, going along in their spray-on advertising, nose on the tarmac, arse in conjunction with the planet Jupiter, they are just subjects of merry jest. When they travel in packs, it's different.

Six to eight pairs of them streamed past me, peddling like the clappers, leaving little to no clearance, doing only the absolute minimum to move out of their straight line and pass, nearly forcing me into the verge -- much more threatening and aggressive road use than any of the buses or farm vehicles I usually have to contend with.

Between their silly outfits and bad behaviour, the sort of thing that gives cyclists a bad name, and discourages cycling as a more relaxed form of transport.

A season in the saddle

2010-10-07T19:22:00.002+01:00

Three months ago, I finally got around to getting a cheap and cheerful trip computer for my bike. Three months later I have done 767.4 miles at an average of 11 mph, as well as measuring that the final part of my journey to work that I couldn't measure with the car odo, starting with an off-road part, comes to just over 1.6 miles each way.

This also measures up to my order of magnitude guesswork that I average closer to 2000 miles in the saddle each year than 1000.

Computing per-service SIDs without sc.exe

2010-10-05T20:54:00.000+01:00

How it's done is well known:

When configured to have a per-service SID (i.e. type of SID either "Unrestricted" or "Restricted"), the service SID is computed as S-1-5-80-{SHA-1(service name in uppercase)}

Here's a quick script to do it, that can be the basis for including the computation programmatically into e.g. installer generation.

open System.Security.Cryptography
open System.Security.Principal
open System.Text
open System

// 1st arg (index [0]) is this script
let name = fsi.CommandLineArgs.[1]
let hash = new SHA1CryptoServiceProvider()
let tail = name.ToUpper()
              |> Encoding.Unicode.GetBytes
              |> (fun x -> hash.ComputeHash(x))
              
let head = [|1uy; 6uy; 0uy; 0uy; 0uy; 0uy; 0uy; 5uy; 80uy; 0uy; 0uy; 0uy |]
let binary = Array.append head tail

let sid = new SecurityIdentifier(binary, 0)
Console.WriteLine(sid.ToString())

So running it we get:

>& 'C:\Program Files\FSharp-2.0.0.0\bin\fsi.exe' .\ssid.fsx MyService
S-1-5-80-517257762-1253276234-605902578-3995580692-1133959824
>

which compares nicely with:

>sc showsid MyService

NAME: MyService
SERVICE SID: S-1-5-80-517257762-1253276234-605902578-3995580692-1133959824
>

where each of the 5 trailing facets is just the decimal representation of 4 bytes of the SHA-1 hash taken as a little-endian unsigned integer.

Film — True Legend (Su Qi-Er)

2010-09-25T01:55:00.001+01:00

Woo-Ping Yuen's latest chop-socky film about Beggar Su, the master of Dunken Fist wu-shu had its UK première here this evening.

It was almost solid martial arts mayhem, with just a few pauses for breath; however the structure of the film was a bit of a mess -- the intrusive 'now put on your 3D glasses/OK take them off again' around the training arc and what looked like it was going to be the Boss Fight didn't help (nor did the 3D itself where the actors often seemed to be floating detached from the rather washed-out scene around them.

The story itself seems to be a fairly direct one about Su and his adopted brother Yuan who spurns the generosity of his adopted family, and cue two-way revenge tragedy -- then just when you think it's all over, a long coda culminating in the real Boss Fight out of nowhere.

Clearly it must be following the high points of a well known tale -- but it ends up feeling unresolved because of the broken narrative rhythm, whereas stopping at the tragedy of the 3/4 mark would have given closure (if you don't ask too deeply why a martial artist of Su's calibre would dig a box out of sandy ground with his bare hands, rather than using some one-inch-punch technique to break open the lid in situ).

In all, not entirely satisfying popcorn fare.

Film — Henri Quatre (Henry of Navarre)

2010-09-22T10:19:00.002+01:00

Added to the Film Festival programme after the program went to print, a competently done edited-highlights costume drama (German production, dialogue in French with some Italian and Latin) of the life and turbulent times of the eponymous French King.

Not something I would have gone out of my way to see, had I not already earmarked this week for the festival, but equally not something where I felt I'd wasted the afternoon.