Launching PowerShell 2.0 into .net4 via F# Interactive or IronPython 2.7

With .net4 being 18 months old, and the Win8/PowerShell 3.0 being both in CTP and only for Win7+, it's a drag having to do things with .net 4 code when you really could do them faster and better in PowerShell. So, why not bootstrap ourselves from out of the box PowerShell 2.0 into the .net 4 world with a little help from another scripting language which is .net 4 aware? This saves messing about with environment activation variables, centralized registry settings or application config files, and takes advantage of the fact that PowerShell, like other scripting languages, can be hosted by another .net process.

The recipe is based around Bart de Smet's Option 2 – Hosting Windows PowerShell yourself.

Via F# interactive:

Via IronPython 2.7 (augmented to take command line arguments via the -Options parameter):

Since we can't pipe into Start-Process, the inner script is written to a temporary file; and then the initial PowerShell process waits so that it can safely delete it. If you would rather, it's reasonable to save the script to a constant .fsx or .py, replace $source with the path to that file, then the -Wait and Remove-Item -Force $source can be removed.

Replacing the fixed command line arguments for the F# interactive launched PowerShell with ones input to the script as per IronPython follows the same pattern. In either case, care has to be taken with getting your quotation marks right, and the IPy example is rather rough-and-ready. Besides, doing appropriate quotation parsing and escaping would obscure the real point of the exercise.

Note: Launching PowerShell directly from the .net4 command line launches PowerShell in the plain old .net 2 configuration; it is, alas, not sticky.

Note: These scripts are intended to give you a .net4 based interactive session; an alternative approach aimed at running individual commands can be found on Jason Stangroome's GitHub.

Using LINQ/Extension methods with IronPython 2.7's clr.ImportExtensions method

Woefully underdocumented on the internet, I worked this one out by analogy to the IronRuby using_clr_extensions method:

Scripting the Win32 API - F# and IronPython FFI

Mainly a worked example for reminding me of how the syntax goes, reimplementing sn -k in F# and IronPython:

This about halfway between how you'd just call the APIs naturally in C++/CLI, and the full process of C# P/Invoke. You do need to do your own extern method declarations, whereas IronPython is something else again:

where it was easier to fudge the pointer value for the key buffer into an IntPtr than to try and dereference it via ctypes and make a manual copy.

The code here is an adapted subset of a managed API for strong-name keys; the use-case being a contingent key generation as part of a build process.

How-To : IronPython and callback delegates with 'out' or 'ref' parameters

Following up on the earlier post, having found time to engage the brain a little...

Let's do the int flavoured version, everything non-default:

and then in IronPython ask the variable about itself

Now we get

>"\Program Files\IronPython 2.6\ipy.exe" callback.py
<type 'StrongBox[int]'>
['Equals', 'GetHashCode', 'GetType', 'MemberwiseClone', 'ReferenceEquals', 'ToSt
ring', 'Value', '__class__', '__delattr__', '__doc__', '__format__', '__getattri
bute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__re
pr__', '__setattr__', '__str__', '__subclasshook__']
Array[object]((<System.IntPtr object at 0x000000000000002B [42]>, 17))

That Value member looks interesting, so...

which gives us

>"\Program Files\IronPython 2.6\ipy.exe" callback.py
Array[object]((<System.IntPtr object at 0x000000000000002B [42]>, 23))

which is what we were looking for.

Rebuilding the C# assembly with out rather than ref and re-instating the print lines shows that the argument is still passed as a StrongBox, so there is no change on the IronPython side.

Armed with this piece of knowledge about the type, I tried Googling again, and didn't find any obvious mentions of callbacks -- just the converse matter of calling .Net methods that take out or ref parameters. But that at least is enough to reassure me that this is not just some completely undocumented hack that would be subject to change in later versions.

Note also that if I had written

print x.GetType().ToString()

(which is what I did the first time around) rather than

print type(x)

I would have gotten the result System.Int32, which gives no clue that adding

print dir(x)

is at all something worth doing.

IronPython, WPF, and button-free windows

So, how do we achieve something like this with WPF, using IronPython?

Assume Vista or later (if you want to support XP, do something different if System.Environment.OSVersion.Version.Major < 6)

Start with some XAML

<Window
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    Title="Window2" Height="300" Width="400" MinWidth="400" MinHeight="100"  WindowStyle="None" AllowsTransparency="True">
    <Grid>
        <Grid Name="banner">
            <Image Name="icon" Stretch="Fill" HorizontalAlignment="Left" VerticalAlignment="Top" Width="20" Height="20" Margin="4,4,0,0" />
            <Label Height="28" Name="title" VerticalAlignment="Top" Background="WhiteSmoke" Margin="28,0,0,0">Label</Label>
        </Grid>
        <StackPanel Margin="0,28,0,0" Name="body" VerticalAlignment="Top" Height="230">
            <TextBlock Height="100" Name="message" VerticalAlignment="Top" />
            <Image Name="image" Stretch="None" Height="130" Width="400" />
        </StackPanel>
        <StackPanel Name="buttons" Height="55" VerticalAlignment="Bottom" Orientation="Horizontal">
            <!-- Button Height="23" Name="button1" Width="75" Margin="2">Button</Button -->
        </StackPanel>
    </Grid>
</Window>

We can then use ctypes to get at the glass APIs, and set up some re-usable types

which we can then hook into our window (assume we have loaded the XAML into a variable window)

Of course you now need to ensure you have some button or other control that will let you dismiss the window, now you no longer have the kiss of death available at top right.

The ref parameter in the WndProc hook is the reason behind the previous post...

IronPython and callback delegates with 'out' or 'ref' parameters -- the obvious way doesn't work

The actual how-to is given here.

The only on-line documentation I could find that mentioned this has a

TODO - Delegates with out/ref parameters

about halfway down.

So, experiment time. Build callback.dll containing (or the same with 'out'):

and call it from IronPython 2.6:

which yields

Array[object]((<System.IntPtr object at 0x000000000000002B [42]>, False))

In sum -- it fails silently; the direct return value is fine, but mutating the actually by-value parameter does nothing.

Replacing the bool with int, and using non-default values shows that the value of a ref parameter is passed in correctly, it just can't be passed out with direct assignment.

“Hello Glade#” from IronPython

I was a little pessimistic yesterday about this operation -- but with a little sidestepping to use a different method on the Glade XML object, and a little bit of Python metaprogramming, we can actually get something much more general than the F# solution.

Same app as before

The trick here is in writing our own __getattr__ method that calls the name-to-widget look-up method of the Glade model, if we haven't already made a note of the object of that name. Written like this, the Handler class could be used in any IronPython/Glade# application.

Now I wonder whether there is a better way to do it in F# that can use generic typing, and not lose the elegance under the weight of type coercion operators.

Links for 19-Nov

New from the PDC:

• Windows Identity Framework RTM.
• Silverlight 4 beta.
• Reactive Extensions release (Introduction parts 3,4, another introduction).
• IronScheme 1.0 RC1.

WPF - from C# to F#.

Automating signing of PoSh scripts.

Prefer structured lifetimes – local, nested, bounded, deterministic.

Embedding IronPython in Silverlight.

Mono tools for Visual Studio.

Moonlight futures.

Google's 'go' -- is it the new VB?

Alpha-encoding file versions

When building installers the UpgradeVersion must have a unique property value that is an installer public property (upper-case alpha). So, what better way of adding uniqueness than making it have the form "product name + product version" with the version suitably encoded...

So, a script for turning a file version (4 x 16bit ints) encoded as a System.Version into a short alpha string, assuming that Major and Minor will be small, and that common approaches are to step Build, to use a stepped Build plus date-stamped Revision, or a timestamp Build and Revision --

where the first two facets are encoded as telescoped base-13 (with a bit to say "more to come"), and the second two are encoded as pairs of bytes -- Z for a zero byte or as a 2-character base-25 representation if non-zero, with a zero Revision being dropped. This gives 10 characters in a plausible worst case, or as low as 5 in some conventions (stepped build numbers only); as opposed to the naive 64-bit as all-base-26 which would give 11 characters always.

“Hello GTK#” from the latest IronPython and F#

A little post to record a short bit of spiking with GTK# as UI toolkit, porting the simple C# examples from here to my preferred .net languages. Neither of these explorations are totally novel -- there are examples in either language to Google, but not all of them recorded all the details of the slight rough spots that needed a little working (and they were often not at all recent, either).

For IronPython 2.6 latest with GTK# 2.12.9-2, running the programs as ipy Program.py, the code looks like

where the dynamic nature of the language means we can lose a lot of the declaration clutter. We just have to explicitly reference the main GTK# assembly (which is GAC'd by the GTK# installer), and away we go.

F# was almost, but not quite, as smooth. You have to add references to additional assemblies atk-sharp and glib-sharp, and the types are a little more explicit:

With the project built as a Windows Application, the Console output doesn't show (even to the Output window in Visual Studio), so the code has been changed to update the button caption after clicking. Apart from that, GTK# follows the WinForms eventing model, so wiring up the events is just a matter of adding the appropriate handler functions in the same way as you would normally -- including the effect that the function value OnDelete needs to be coerced to a delegate type of the same signature such as via a wrapper fun as shown, and can't just be added directly.

This program is also not FxCop clean, but there's nothing GTK# specific about the tidying operations required.

As expected, these work unchanged with the 1.9.9.9 CTP

Links for 9-Oct

Exploding Software-Engineering Myths.

Dark Corners of IronPython

Gtk# for .Net developers

Tell don't ask.

Smart Stubs with Rhino Mocks.

Functional

• Matthew Podwysocki’s Generically Constraining F# Parts One, Two and Three
• M<’a> Lib (F# and C# LINQ) Monads Library
• Functional Programming for Everyday .NET Development.

Web

• Free CSS Toolbox - Free CSS Validator, CSS Formatter, CSS Compressor
• Starting with @font-face.

Links for 17-Aug

C# -- Four switch oddities.

F# -- Creating WPF dependency properties.

Compiling Python packages into Assemblies.

Better unit tests with Test.Assert(expression).

IronPython for build scripting

Following on from the earlier post with the snippet about generating GUIDs -- and covering a good chunk of what's been occupying me since...

I have ended up in charge of the build system for the current project at work. This started out with one framework that used a number of custom projects inside a solution to perform unit test, FxCop and coverage analysis, with a lot of magic happening in post-build steps, including direct calls to Wix command-line utilities. Another team had developed a better separated MSBuild-based system, which split out things like the analysis and installer building from the assembly-building solution. We can argue the merits of taking the unit tests out of every checking compile; but separating out the installer build does have a significant benefit in terms of cycle time for a recompilation.

Frankensteining the two together was an interesting task; and IronPython has been a valuable component of the mix. As I noted quite a while ago, the convenience of an XCOPY install on a machine with a current .net installation (any build or dev machine), and the access to the full APIs makes for a powerful tool during a build -- it's not just the fact that you get better string manipulation than a batch file, or can easily spawn off a call to source control to get a synch-level value to stamp an assembly with.

In the current context, there are a number of components being built with a common architecture, so there are plenty of opportunities to DRY the system out.

• There are repetitive pieces of code (declaring concrete subtypes of shared base classes, to inject component specific information) which can be run by having a couple of .py files (just containing a single map initialization with common keys and component or project specific values) to define the files affected and the component-specific substitutions to make (including in some cases stable but component specific GUIDs, that can be keyed off the component specific names)
• The shared architecture makes the MSBuild .proj file just as valid for such String.Format based substitution
• Wix source files are just XML documents -- they can be generated programmatically from XML fragments, inspecting the solution output and filling in the appropriate entities.
• So are Wix project files (or any MSBuild project for that matter) -- a project to build a 64-bit installer can be derived from a 32-bit installer project by a similar set of XML manipulations.

Taking the latter as an example

Of course, assumes that the .wxs files have Win64='$(var.X64)' attributes sprinkled appropriately.

Using IronPython with NetBeans Python IDE

With the arrival of the IronPython 2.6 beta 1, with ctypes support, the IronPython 1.x only support in IronPython Studio really takes the IDE from a bit dated to seriously obsolescent.

Now, I had tried to get IronPython 2.0 to talk with NetBeans, but hadn't immediately achieved success, so, with less motivation, had put that to one side. Perhaps for similar reasons, a recipe for this wasn't out on Google already -- but now I really needed to crack the problem, I wasn't going to let that earlier abject failure put me off.

Also, since then, I'd seen someone else doing similar stuff to get IronRuby to play with an IDE, in this case, RubyMine, by tinkering with the file actually getting called by the IDE.

Well, checking my Python 2.6.1 command line with -?, and the same for IronPython 2.6 beta 1, they overlapped in almost every essential. So, I tried the experiment of copying ipy.exe,ipyw.exe to python.exe,pythonw.exe in the same C:\Program Files\IronPython 2.6 folder, and then adding the copied python.exe as a new NetBeans Python platform.

And it worked!

So, I created a new python project for platform Python 2.6.0 (as opposed to CPython 2.6.1's Python 2.6), and entered

and ran it, which yielded

24/05/2009 14:28:30
2.6.0 (IronPython 2.6 Beta 1 (2.6.0.10) on .NET 2.0.50727.3074)

which is of course what we wanted.

Book — IronPython in Action by Foord & Muirhead

IronPython is an important language.

Back in 2000 at the Microsoft PDC, where the .net platform was unveiled after much hype, we were given a vision of the Common Language Runtime as a truly polyglot platform, with 15-20 languages having been fed into the design, tantalizing code snippets shown -- and even previews of ActiveState's real-soon-now Python.NET and Perl.NET implementations.

And then for five years, .net programming meant no more nor less than C# (or VB.Net if you swung that way), with the two languages adopted for the platform from the wider world being at best niche -- J# a known dead end, and C++ for .net never really recovering from the horrible extensions used in the first attempt at the job.

And then, in 2005, it was steam-engine time : the first inklings of an ML dialect, labelled F#, for .net from Microsoft Research, and IronPython, a project initially undertaken to show why Python on .net couldn't be done -- a meme whose genesis I strongly suspect lies in the failure of that early ActiveState effort. While, however, F# still remains a CTP release (even at a 1.9.x release number) to this day, the IronPython 1.0 production release (Python 2.4 compatibility) was the best birthday present I never realised I'd received in 2006. Yet, until very recently, whereas F# was supported by several books, all published even before the CTP announcement, even as a production release, IronPython languished in the English language press.

IronPython in Action now fills that surprising gap.

The task the authors set themselves is an heroic one -- to teach Python to .net programmers, and .net to Python programmers, and, just in case that was not enough, several of the more outré parts of .net, and good programming practices, for just about everybody as well. What makes this a great book is that, in the course of about 450 pages, with copious external citations, they actually succeed.

Part of the secret of the success is that this (like Programming in Scala) is not a beginner's book and assumes the reader has a degree of familiarity with basic programming concepts -- for example, the Python if, for and while statements are covered together in just over a page, with the link collection in Appendix C there in case a more at length treatment is required -- so freeing space for more advanced material to be covered.

The scope of the material covered came as a most pleasant surprise -- when I pre-ordered the book, it was as a gesture of support, because the language deserved a presence in print (after all, I'd been programming since whenever, using .net since it hit 1.0, and IronPython for a couple of years, so not much of it should be exactly new...); when it finally arrived and I could read it, I found there were significant things I could learn from it, new insights and just better ways to achieve some things already did.

In particular, the chapter on testing is pretty much worth the admission price all by itself, simply for the worked example of how to solve that perennial problem -- performing automated testing of a .net GUI application (as opposed to just testing the backing libraries).

You don't even need to use IronPython as your .net language of choice to benefit from much of the third section of the book, Advanced .NET. This not only covers parts of the framework too often neglected in C# texts (like the extremely useful, if unsexy, System.Management namespace), but also provides a more measured introduction to the sexier new technologies (WPF and Silverlight) than the books dedicated to those technologies that I have read. Where it is specific to IronPython, this section also serves to emphasise the importance of the language within the larger world of .net, with the ASP.Net extensions for dynamic languages, and the inclusion of the dynamic language extensions (the DLR) within Silverlight.

And if you don't yet use IronPython (or the less mature IronRuby), there's really only one thing that this book neglects to point out about the language -- but it's one that might finally change your mind. Although it's now distributed as an .msi installer, once unpacked, the required assemblies can still be simply XCOPY installed to run on any machine with the .net 2.0 or later framework already in place, giving you a .net-aware scripting language anywhere you have .net already.

---

Disclaimers: Yes, I do get a shout-out in the book for one of my earlier postings in this blog. And while I would have written a review anyway, for the same reason I bought the book sight unseen, I have since been suggested to the guys at Manning Publications as a potential reviewer by the authors. So, here is the review you were looking for.

This review is released under the WTFPL.

Links for 18-Feb

Windows 7 - native VHD support.

More usable thread-safe collections.

Inconsolata -- A free monospace font alternative inspired by Consolas, Avenir, Franklin Gothic and others.

Getting Mono and Visual Studio to play nicely together.

Aero Glass managed UIs - how to.

Spaces in C# identifier names? Interesting hack.

IronScheme beta 2.

Consuming extension methods from IronPython (parts 1,2,3) -- uses dynamic C# compilation.

IronPython 2.0 RTW + Silverlight 2.0

15-Apr-15 : Please ignore -- obsolete technologies

Following on from the β2 sample…

Changes made as follows:

• web page page, silverlight object type : type="application/x-silverlight-2"
• web page, fallback download location : "http://go.microsoft.com/fwlink/?LinkID=124807"
• AppManifest : RuntimeVersion="2.0.31005.0"
• AppManifest : add <AssemblyPart Source="Microsoft.Scripting.ExtensionAttribute.dll" />
• AppManifest : remove <AssemblyPart Name="System.Windows.Controls.Extended" Source="System.Windows.Controls.Extended.dll" />
• app.py -- remove the line clr.AddReference('System.Windows.Controls.Extended, Version=2.0.5.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35')
• Delete all earlier assemblies and use all the assemblies from C:\Program Files\IronPython 2.0\Silverlight\bin.

Or, from scratch, with the embedding paged base around C:\Program Files\IronPython 2.0\Silverlight\script\templates\python\index.html ...

app.py

app.xaml

AppManifest.xaml

The XAP file is now down to 935kb, which considerably closes the gap with Jython 2.2.1's 670k but is still noticeably larger than a minimal Jython 2.1 applet:

274467 Aug 13 21:48 applet.jar
       956662 Dec 13 17:50 app.xap
         2266 Dec 13 17:53 index.html
         2071 Aug 13 22:02 jython.html

However, now both products are at a stable release, it starts to make sense to actually use the technology.

Following up here.

IronPython + Silverlight 2β2

This finally bubbled to the top of my "must get around to, some time" stack today…

Recording what I needed to do to get the second Voidspace controls example to work with the later beta:

1. Remove all the assemblies in the app folder.
2. Copy in System.Windows.Controls.Extended.dll (only) from the Silverlight beta 2 SDK.
3. In the web page, set the type to be "application/x-silverlight-2-b2".
4. In AppManifest.xaml, update RuntimeVersion to "2.0.30523.06".
5. Add <AssemblyPart Source="Microsoft.Scripting.Core.dll" /> to the manifest.
6. In the python source, remove the System.Windows.Controls reference, and update System.Windows.Controls.Extended to version 2.0.5.0.
7. Replace WatermarkTextBox with just plain TextBox.

Only needing an <object> tag with type and data attributes, it works in Firefox 3 -- you just have to explicitly go and install the beta (the fallback link inside the <object> tag didn't show with FF3 and Silverlight 1).

So here it is -- 1k of code along with 1.2M of infrastructure assemblies (which is a megabyte more weight than Jython would need for the equivalent).

Sample removed from here 13-Dec-08 as the beta release is now obsolete.. New sample here.

For comparison

274467 Aug 13 21:47 applet.jar
     1229665 Aug 12 20:22 app.xap
        2260 Aug 12 20:22 index.html
        2071 Aug 13 21:47 jython.html

So, what is that assembly called?

A trivial bit of PoSh that I know I'll end up reusing:

>[system.reflection.assemblyname]::getassemblyname( …path to assembly… ).FullName

gives you the string you need to put into e.g. clr.Addreference().

Links for 08-08-08

The IronPython compilation sample is now belatedly available, and discussed here.

Making IList<T> implementations more debugger friendly.

PoSh owner's Manual and Toolbox.

IronNails -- Rails like syntax for WPF/Silverlight.