Computing cyclomatic complexity with F# 5.0 interactive and Mono.Cecil

Many moons ago, I posted a simple PowerShell script to use the FxCop SDK to introspect over a bunch of assemblies and compute a measure of the cyclomatic complexity of each method; in this case, by counting the number of IL branch instructions that do not branch to the next instruction, and which have a distinct target from any other branch.

That in turn was based upon the algorithm used in NDepend 1.x, which was sufficiently antique to not consider switch opcodes, a deficiency which can be worked around. And as an alternative, the algorithm used in Mono.Gendarme can be implemented in F# instead.

The main barrier to creating a simple replacement with Mono.Cecil has been the location of the assemblies with the NuGet package version in the file path. Now, however, with F# 5.0 interactive allowing reference-to-nuget statements, providing a pair of scripts that can do the job, without the awkward question "Where's Cecil?", becomes a simple task.

The classic version first

and another using the algorithm from Mono.Gendarme

Configuring Jenkins with PowerShell

No sooner do I say I have nothing technical to write up of general interest, than I spend a day stitching together pieces across the internet, because most Jenkins examples tend to be written to *nix or the JVM, and I'm on Windows where the admin tool of choice is PowerShell.

So, start on the Jenkins wiki page for Authenticating scripted clients, which tells you how to get your API key -- visit $(JENKINS_URL)/me/configure in your browser and look for the API token so you don't have to script your password (especially if you're using AD authentication on the server). Setting username/API Token as a Credentials object on a WebClient will just get you 403 errors, until you notice that the Groovy script example sets pre-emptive auth on its web client. The secret to HTTP Authorization and .NET WebRequest, WebClient Classes needs to be sought separately.

At this point you can GET from $(JENKINS_URL)/job/[job name]/config.xml, with DownloadString and cast to [xml] PowerShell style to read and modify.

Then you have to POST the modified XML back; but if you just do that with the Basic auth header, suddenly more 403 out of nowhere, until you read the small print about the Jenkins Remote access API about CSRF protection. When you get that and add it to your headers, it's now just a case of using UploadString to push the xml as xml.

Putting it all together we get

This appears to correctly preserve line endings as is, so you don't need to do anything non-default, equivalent to the --data-binary as you need for scripting with curl.

FsCheck is great for for native code too!

I've just had call to write some good old-fashioned 'C' code recently, for use in an environment where my access to anything fancy like external libraries is limited. And a one part needs a simple transcoding function to take data in and manipulate it from the form convenient to provide into a form that's actually useful.

But of course, I want to test that bit before I start wiring it into place. There are, of course a whole slew of 'C' unit test frameworks out there, none of which I've used before, but really I wanted to just soak-test the algorithm quickly and simply, with a maximum of coverage and a minimum of distractions, so, I actually wanted a QuickCheck-equivalent tool that would "just work".

And this is where F#'s easy P/Invoke syntax (almost as simple as cutting and pasting out of the 'C' header file) and FsCheck were just the tools I was looking for -- write a property (or, in fact, a series of properties) that asserted that an input mapped by my function was the same as obtained using an equivalent .net library function already to hand as reference, invoke them with FsCheck.Check.Quick, and then TDD away, starting with the rough outlines as asserted by the early properties, and then filling in details as asserted by the later ones.

Playing with (almost) the latest C++ features -- groundwork

Having had my interest in playing with native code reawoken by the new C++11 features, the first thing I went to look at was portability. One of the advantages of managed code -- JVM or CLR -- is that the VM handles portability for you and the code can be built pretty much anywhere; with native code we have to see what the compilers have in common.

I've been using VC++2010 on Windows as having many of the "big rocks" for the new standard, while being backward compatible onto OS versions before Win7 (unlike the VC++11 compiler and its runtime); and for *nix-like systems, I have cygwin and debian squeeze... Well the distro support for these is a bit behind-hand (gcc versions 4.5 and 4.4 respectively), whereas gcc 4.7 is now out with quite a broad coverage of the new standard. While 4.5 has a good chunk of the new stuff, 4.4 doesn't -- in particular, it doesn't have the new lambda syntax. So, it's build from source time to get an upgrade to a sensible version there, which means I might as well go to the latest and greatest on both platforms...

Building gcc 4.7 from source

debian

Fortunately there are some handy instructions out there which can be used as a baseline for debian. Following them, I found that I needed to tweak how I built GMP to fit PPL's requirements, by modifying the configure step to be:

CPPFLAGS=-fexceptions ../../sources/gmp-5.0.4/configure --prefix=$PROJECT_DIR/output/ --enable-cxx

before PPL would go through happily. The --enable-cxx is required for the PPL ./configure stage to run through, the CPPFLAGS=-fexceptions is optional, but it avoids a make-time warning about possible unwanted runtime behaviours if you don't.

ClooG also needed a CPPFLAGS=-I$PROJECT_DIR/output/include CFLAGS=-L$PROJECT_DIR/output/lib on the configure line to find GMP.

In the gcc build, as well as pointing at ../../sources/gcc-4.7.0/configure it's also worth taking the advice from the MacOS build instructions and only selecting languages of interest to you i.e. to play with new C/C++ there's no need for Java or Fortran, at a considerable saving in time.

Then it's a matter of just adding soft links from whichever g*-4.7 files to the unadorned versions, and

export LD_LIBRARY_PATH=~/gcc-4.7/output/lib/
export PATH=~/gcc-4.7/output/bin:$PATH

to your .bashrc or equivalent

cygwin

Cygwin is more fun -- I've not yet managed to get that to build all the way through with the loop optimization libraries. When you get to PPL, you find you also need to go back and re-configure GMP with --disable-static --enable-shared, as explained in the friendly manual, to build the shared library version. However then when building gcc, we get a mismatch with the earlier libraries in the configure stage, where it just stops:

checking for the correct version of gmp.h... yes
checking for the correct version of mpfr.h... yes
checking for the correct version of mpc.h... yes
checking for the correct version of the gmp/mpfr/mpc libraries... no

It is possible that if you start by building PPL and CLooG with shared library GMP in a first pass, then build the rest starting with reconfiguring and building a static GMP it will work, but life is too short. The MacOS build instructions didn't use the PPL/ClooG/graphite libraries either, so we can do this to configure gcc instead:

$ ../../sources/gcc-4.7.0/configure          \
>     --prefix=$PROJECT_DIR/output/    \
>     --with-gmp=$PROJECT_DIR/output/  \
>     --with-mpfr=$PROJECT_DIR/output/ \
>     --with-mpc=$PROJECT_DIR/output/  \
>     --program-suffix=-4.7            \
>     --without-ppl --without-cloog    \
>     --enable-languages=c,c++

which sits and cooks for quite some time to get you the new compiler build.

Linkbait: fixing cygwin "mkdir foo mkdir: cannot create directory `foo': Permission denied"

I got into a state where I had this error, which other people have seen, after having tried to trash the build output of one of the failed PPL/CLooG attempts from Windows Explorer, where anywhere under in my home directory and down it was rejecting mkdir with "mkdir foo mkdir: cannot create directory `foo': Permission denied". Having spotted a tangentially related mailing list message about this sort of problem happening on network shares and there being ACL related, I tried the following and it worked to clear things up:

1. Start a PowerShell console as Administrator window
2. Run Get-Acl a folder (like /tmp) which you can mkdir in in cygwin (this will be %cygwin_root%\tmp where %cygwin_root% is where you installed cygwin)
   $acl = Get-Acl C:\cygwin\tmp
3. In Windows Explorer set yourself Full Control on all the affected folders -- %cygwin_root%\home and %cygwin_root%\home\%USERNAME% at least
4. In the PowerShell, Set-Acl on each folder you've just frobbed with the saved ACL object
   Set-Acl C:\cygwin\home $acl
   Note that the Set-Acl call may take considerable time (tens of seconds) to execute when it gets to the really problematic node and has to roll permissions down.
   

SCons -- Death to makefiles

Since I last did native code seriously at home (c. year 2000), I had discovered the very handy MiniCppUnit tool as a nice light-weight unit testing framework, so of course I went and fetched a copy to be going on with. This time, curiosity prompted me to wonder "WTF is this SConstruct file anyway?" and now when I opened it, I immediately recognised that it was some form of Python script, and wondered what sort of Python based make system this might be.

It was simple enough to find where it came from -- http://www.scons.org/ -- and looking at the user guide, I felt that it is much more intuitive system than makefiles (admittedly there's not a high barrier there), and far less cluttered than declarative XML based systems like Ant or MSBuild; so I'll be using that for my *nix builds -- it works very nicely for doing things like running unit tests as part of the build e.g.

MiniCppUnit -- Building it under modern C++

Just like I had to patch it to build in C++/CLI, I needed to make some changes to MiniCppUnit to get it to build clean under VC++2010, the out-of-the-box gcc 4.5 on cygwin 1.7.x and gcc 4.7 debian squeeze with -Wall -std=gnu++0x or -Wall -std=c++11 on respectively. First MiniCppUnit.hxx:

92c92
< #if _MSC_VER < 1300
---
> #if defined(_MSC_VER) && _MSC_VER < 1300
93a94
> /* Without the "defined(_MSC_VER) &&" this code gets included when building on cygwin 1.7.x with gcc 4.5.3 at least */
204c205
<  static void assertTrue(char* strExpression, bool expression,
---
>     static void assertTrue(const char* strExpression, bool expression,
207c208
<  static void assertTrueMissatge(char* strExpression, bool expression, 
---
>     static void assertTrueMissatge(const char* strExpression, bool expression, 
304c305
<    catch ( TestFailedException& failure) //just for skiping current test case
---
>    catch ( TestFailedException& /*failure*/) //just for skiping current test case

and the corresponding signature change in the .cxx file:

108c108
< void Assert::assertTrue(char* strExpression, bool expression,
---
> void Assert::assertTrue(const char* strExpression, bool expression,
122c122
< void Assert::assertTrueMissatge(char* strExpression, bool expression, 
---
> void Assert::assertTrueMissatge(const char* strExpression, bool expression, 

Of course there may be other things lurking to be scared out when I ramp up my standard warning levels to beyond the misleadingly named -Wall (when you have -Wextra, formerly -W, provided to switch on a whole bunch more including spotting signed/unsigned comparisons, before getting onto the really specialized ones) and switch on -Werror to force everything to be really clean.

Computing cyclomatic complexity with PowerShell and FxCop

A measure of the cyclomatic complexity of a .net method can be gauged by counting the number of IL branch instructions that do nor branch to the next instruction, and which have a distinct target from any other branch -- this is essentially the algorithm used in NDepend 1.x to make the computation. The introspection mechanism of FxCop provides enough decompilation of the IL that getting the instruction type and offset, and the target offset of a branch. An actual FxCop rule, though feasible to write, however, would be less useful than one could hope.

For one thing, FxCop itself doesn't provide any convenient way of passing parameters to custom rules (to e.g. set a trigger threshold); and for another, any analysis is likely to be run over a debug build (DEBUG and CODE_ANALYSIS variables defined there to not contaminate the released code with [SuppressMessage] annotations), which is likely to have a different underlying complexity to a release build (this is especially true in F# debug builds where there are a lot of sequences that look like

which can be replaced by

which, as it turns out, is pretty much what the release build does.

A more configurable and flexible way of performing the operation is to drive the FxCop facilities from a script -- these days I'm doing a lot of my .net scripting in PowerShell, but it could equally well be done with IronPython or similar .net scripting language. The result looks like this:

and running it over a set of F# code shows that the reported complexity of the release build of a method halves (or more) what is reported for a debug build. I haven't made a comparison over an significant amount of C# as yet, to see how much complexity the compiler removes between the two configurations.

dotCover 1.0 beta -- first impressions

The beta of the coverage tool from the same people who brought you Resharper came out late last week, and I took the chance to have a quick play with what it offers.

As expected, it integrates smoothly with the R# unit test running -- you can see exactly what code your test or set of tests cover; marked in green overlay in the source view for covered by the test(s) you just ran, red elsewhere, in a manner reminiscent of NCoverExplorer.

There is also a command-line option

Usage: JetBrains.dotCover.ConsoleRunner.exe /Executable='executable' [/Arguments='arguments'] [/WorkingDir='working dir'] /Output='output' 

Not having R# at home -- because I rarely use C# even if I'm writing for .net -- it's this which interested me more. So, I gave it a try on my current F# project

>& 'C:\Program Files\JetBrains\dotCover\v1.0\Bin\JetBrains.dotCover.ConsoleRunner.exe' /executable='..\..\..\_Tools\Microsoft FxCop 1.36\FxCopCmd.exe' /Arguments='/q /s /console /f:Tinesware.Recorder.dll /f:Tinesware.Instrumentation.exe /f: Tinesware.Infrastructure.dll /f:Tinesware.Rules.dll /f:Tinesware.InfrastructureTests.dll /f:BaseTests.dll  /rule:Tinesware.Rules.dll /o:fxcop.xml  /rule:.\Rules /dictionary:..\..\..\CustomDictionary.xml' /workingdir=. /output=dotcover.xml

which yields a bald total coverage number

JetBrains dotCover Console Runner v1.0.56.11 JetBrains s.r.o.
Coverage session started [12/07/2010 20:14:16]
Coverage session finished [12/07/2010 20:14:28]
Index files: ...AppData\Local\Temp\ssc06AE2.tmp
Log files: ...AppData\Local\Temp\lgc02A14.tmp
Opening snapshot...
Done.
Generating report...
Done.
Total coverage: 78%

and an XML report file that makes it clear that that percentage is based on all assemblies dragged into the AppDomain, counting those which have no pdb information as containing no statements to cover, which means if you're running unit tests with Rhino.Mocks that it includes things like

<Assembly Name="021925b1-cbe9-42d3-ad44-a39206195c2a" CoveredStatements="0" TotalStatements="0" CoveragePercent="0">
    <Type Name="IsolatorProxyb66a653c9a984566ac790c4fa654cab5" CoveredStatements="0" TotalStatements="0" CoveragePercent="0">
      <Type Name="InvocationShouldCallOriginal_1" CoveredStatements="0" TotalStatements="0" CoveragePercent="0">
        <Member Name="HandleEvent" CoveredStatements="0" TotalStatements="0" CoveragePercent="0" />

emanating from dynamic assemblies. And it seems from simple experiment that the filters that you can apply in Visual Studio don't seem to carry over to the command-line version.

The report is formatted as shown above, with hierarchy

Root -> Assembly -> Namespace -> Type -> Nested Type or Member (repeating Type as often as necessary)

There’s also no granularity below Member, so even for code with .pdb information available the reports only look like

<Type Name="Local" CoveredStatements="166" TotalStatements="173" CoveragePercent="95">
...
        <Member Name="get_Mutex" CoveredStatements="0" TotalStatements="0" CoveragePercent="0" />
        <Member Name="ExpandFile" CoveredStatements="2" TotalStatements="4" CoveragePercent="50" />
        <Member Name="LoadFile" CoveredStatements="4" TotalStatements="7" CoveragePercent="57" />
        <Member Name="op_GreaterQmarkGreater" CoveredStatements="4" TotalStatements="4" CoveragePercent="100" />
        <Member Name="op_GreaterPlusQmarkQmark" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="op_GreaterMultiplyGreater" CoveredStatements="4" TotalStatements="5" CoveragePercent="80" />
        <Member Name="op_GreaterPlusQmark" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="op_GreaterPlus" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="GetXmlData" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="DoWithLock" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="op_Append" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="GetByName" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="SetByName" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="GetStatementRange" CoveredStatements="2" TotalStatements="2" CoveragePercent="100" />
        <Member Name="FindSignatureForMethod" CoveredStatements="8" TotalStatements="8" CoveragePercent="100" />
        <Member Name="FindXmlForMethod" CoveredStatements="7" TotalStatements="7" CoveragePercent="100" />
        <Member Name="get_DeclaredExempt" CoveredStatements="0" TotalStatements="0" CoveragePercent="0" />
        <Member Name="get_AutomaticExempt" CoveredStatements="0" TotalStatements="0" CoveragePercent="0" />
...
      </Type>

So there can be no useful equivalent of NCoverExplorer for after the fact detail – to get detailed line un-coverage information you actually have to run every one of your unit tests in Visual Studio in one big bang.

Its metric of what counts as a statement is subtly different from what NCover (free) and the other tools I've been working on recently. It reports an F# property like

let AutomaticExempt = "-2"

or an equivalent C# automatic property, for that matter, as having 0 lines, rather than N/A and 1 respectively; but that aside it seems to be counting the self-same list of sequence points.

So, it seems that this is primarily a tool for interactive use and positive coverage testing -- "Does this set of tests cover this line, and if so, which one(s)?"; in a build process environment the lack of ability to focus down to only the assemblies of interest in the report (discarding tool and system code), and of per-statement reporting it is less interesting.

However it did take the F# code which PartCover had balked at in its stride.

Meanwhile in other news

There is a branch of PartCover that supports .net 4 originally on GitHub and which has now been merged into the main-line at SourceForge, removing one of the objections to using this tool. There also seems to be rather more life in this than there was six months ago when I tried it, so I shall have to revisit it and see if the IL issue is also resolved.

coverage.exe and instrumenting F# files

I observed yesterday that coverage.exe balked in PdbReader.GetSegmentsByMethod() when trying to instrument F#-derived assemblies.

By checking to see whether a new CodeSegment entry would provoke the exception by having an identical offset to a pre-existing one, and dumping both old and new to the console, I observed that at least one, and often both, had associated line number 0x00Feefee -- i.e. were compiler generated code with no source reference.

And in the method Executor.ProcessMethod(), which is the only place to call PdbReader.GetSegmentsByMethod(), entries with this line number are discarded before beginning to instrument the IL code -- so it would seem to make perfect sense to discard these CodeSegments before putting them in the dictionary. With that change, instrumentation completes cleanly.

Of course, having built coverage.exe with C#4 for .net 4, the helper DLL it built is compiled for .net 4 as well, which has stymied my first test attempt. Oh well, enough for today.

Getting coverage.exe (trunk) to work with nUnit and .net 4

Following up from yesterday, about an instrumenting coverage tool that I'd spotted on Googlecode (Apr 2015 : rescued to GitHub), what I needed to do to get the instrumenting coverage tool working under .net 3.5sp1 with nUnit 2.5.2 driving unit tests.

First, the command line looks like

[path to]\ClassLibrary1.dll [path to]\UnitTestClassLibrary1.dll /x coverage.xml /r /exe [tool path to]\nunit-console.exe [path to]\UnitTestClassLibrary1.dll

where the /r (backup and replace) flag is essential; so you need to work on a copy directory with your assemblies and their .pdb files.

Second, you need to make the following change to the coverage main program in Runner.cs, line 92 from

to be

The hard-coded relative path could be added as yet another argument instead; and I've not played around with the shadow-copy parameter.

But that at least -- with a totally trivial test set -- provided me with a clean run and non-empty coverage data.

So the next thing to do will be to port the whole lot to .net 4 and see what gives. Also, it will be worth trying the teamcity branch (faking NCover 1.x) to see whether the different AppDomain usage there means we don't have to resort to this trick.

Later: doing a rebuild of Coverage and the sample test under .net 4, with command line

[path to]\ClassLibrary1.dll [path to]\UnitTestClassLibrary1.dll /x coverage.xml /r /exe [tool path to nunint 2.5.4]\nunit-console.exe /framework=net-4.0.30319 [path to]\UnitTestClassLibrary1.dll

where the nunit-console.exe.config has been adjusted appropriately, I get

Copyright (C) 2002-2004 James W. Newkirk, Michael C. Two, lexei A. Vorontsov.
Copyright (C) 2000-2002 Philip Craig.
All Rights Reserved.

Runtime Environment -
   OS Version: Microsoft Windows NT 6.0.6002 Service Pack 2
  CLR Version: 2.0.50727.4200 ( Net 2.0 )

ProcessModel: Default    DomainUsage: Multiple
Execution Runtime: net-4.0.30319
Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object.

Server stack trace:
   at NUnit.Util.ProcessRunner.Load(TestPackage package)
   at NUnit.Core.ProxyTestRunner.Load(TestPackage package)
   at NUnit.Util.RemoteTestAgent.AgentRunner.Load(TestPackage package)
   at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.PrivateProcessMessage(RuntimeMethodHandle md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg, Int32 methodPtr, Boolean fExecuteInContext)

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at NUnit.Core.TestRunner.Load(TestPackage package)
   at NUnit.Core.ProxyTestRunner.Load(TestPackage package)
   at NUnit.Util.ProcessRunner.Load(TestPackage package)
   at NUnit.ConsoleRunner.ConsoleUi.Execute(ConsoleOptions options)
   at NUnit.ConsoleRunner.Runner.Main(String[] args)

However, if I do this as a two-stage operation

[path to]\ClassLibrary1.dll [path to]\UnitTestClassLibrary1.dll /x coverage.xml /r

to instrument the code and create a coverage file with zero visit counts throughout -- and then run nunit separately with rest of the command line

[tool path to nunint 2.5.4]\nunit-console.exe /framework=net-4.0.30319 [path to]\UnitTestClassLibrary1.dll

this then fills in the visit counts as expected.

We have code coverage for .net 4 -- at least under some conditions! And without having to rebuild any tools apart from coverage.exe!

Later yet (refrigerator logic) -- of course when working coverage in two passes, I don't need to do the fiddle to change the settings for the AppDomain. And I don't need to have coverage built in the same .net version as I want to run nUnit in, either. I can use the code as synched from Googlecode built under .net 3.5 even for .net 4 codebases; all I have to do now is fix the problems with running it over F# code.

Book — IronPython in Action by Foord & Muirhead

IronPython is an important language.

Back in 2000 at the Microsoft PDC, where the .net platform was unveiled after much hype, we were given a vision of the Common Language Runtime as a truly polyglot platform, with 15-20 languages having been fed into the design, tantalizing code snippets shown -- and even previews of ActiveState's real-soon-now Python.NET and Perl.NET implementations.

And then for five years, .net programming meant no more nor less than C# (or VB.Net if you swung that way), with the two languages adopted for the platform from the wider world being at best niche -- J# a known dead end, and C++ for .net never really recovering from the horrible extensions used in the first attempt at the job.

And then, in 2005, it was steam-engine time : the first inklings of an ML dialect, labelled F#, for .net from Microsoft Research, and IronPython, a project initially undertaken to show why Python on .net couldn't be done -- a meme whose genesis I strongly suspect lies in the failure of that early ActiveState effort. While, however, F# still remains a CTP release (even at a 1.9.x release number) to this day, the IronPython 1.0 production release (Python 2.4 compatibility) was the best birthday present I never realised I'd received in 2006. Yet, until very recently, whereas F# was supported by several books, all published even before the CTP announcement, even as a production release, IronPython languished in the English language press.

IronPython in Action now fills that surprising gap.

The task the authors set themselves is an heroic one -- to teach Python to .net programmers, and .net to Python programmers, and, just in case that was not enough, several of the more outré parts of .net, and good programming practices, for just about everybody as well. What makes this a great book is that, in the course of about 450 pages, with copious external citations, they actually succeed.

Part of the secret of the success is that this (like Programming in Scala) is not a beginner's book and assumes the reader has a degree of familiarity with basic programming concepts -- for example, the Python if, for and while statements are covered together in just over a page, with the link collection in Appendix C there in case a more at length treatment is required -- so freeing space for more advanced material to be covered.

The scope of the material covered came as a most pleasant surprise -- when I pre-ordered the book, it was as a gesture of support, because the language deserved a presence in print (after all, I'd been programming since whenever, using .net since it hit 1.0, and IronPython for a couple of years, so not much of it should be exactly new...); when it finally arrived and I could read it, I found there were significant things I could learn from it, new insights and just better ways to achieve some things already did.

In particular, the chapter on testing is pretty much worth the admission price all by itself, simply for the worked example of how to solve that perennial problem -- performing automated testing of a .net GUI application (as opposed to just testing the backing libraries).

You don't even need to use IronPython as your .net language of choice to benefit from much of the third section of the book, Advanced .NET. This not only covers parts of the framework too often neglected in C# texts (like the extremely useful, if unsexy, System.Management namespace), but also provides a more measured introduction to the sexier new technologies (WPF and Silverlight) than the books dedicated to those technologies that I have read. Where it is specific to IronPython, this section also serves to emphasise the importance of the language within the larger world of .net, with the ASP.Net extensions for dynamic languages, and the inclusion of the dynamic language extensions (the DLR) within Silverlight.

And if you don't yet use IronPython (or the less mature IronRuby), there's really only one thing that this book neglects to point out about the language -- but it's one that might finally change your mind. Although it's now distributed as an .msi installer, once unpacked, the required assemblies can still be simply XCOPY installed to run on any machine with the .net 2.0 or later framework already in place, giving you a .net-aware scripting language anywhere you have .net already.

---

Disclaimers: Yes, I do get a shout-out in the book for one of my earlier postings in this blog. And while I would have written a review anyway, for the same reason I bought the book sight unseen, I have since been suggested to the guys at Manning Publications as a potential reviewer by the authors. So, here is the review you were looking for.

This review is released under the WTFPL.

Links for 24-Sep

When windows are not enough -- there is still the command line

Bringing the browser to the server -- server-side JavaScript + browser-focussed JavaScript toolkits = test automation + whatever else you can think of

Collected Java Practices -- a collection of wheels for not reinventing

FePy r6 + 1 line of code runs PyFit

Following up this trail again --

IPCE-r6 doesn't run PyFit out of the box, but it gets a lot closer than the mainline does:

Traceback (most recent call last):
 File C:\PyFIT-0.8a1\fit\FitServer.py, line 7, in Initialize
 File , line 0, in __import__##4
 File C:\PyFIT-0.8a1\fit\fitnesse\FitServerImplementation.py, line 49, in Initialize
 File , line 0, in __import__##4
 File C:\PyFIT-0.8a1\fit\fit\Fixture.py, line 21, in Initialize
 File , line 0, in __import__##4
 File C:\PyFIT-0.8a1\fit\fit\TypeAdapter.py, line 34, in Initialize
 File , line 0, in __import__##4
 File C:\PyFIT-0.8a1\fit\fit\taBase.py, line 36, in Initialize
 File C:\PyFIT-0.8a1\fit\fit\taBase.py, line 82, in TypeAdapter
AttributeError: 'module' object has no attribute 'ast'

This line is the first to look for compiler.ast members to initialise data members. Looking at the IPCE bundle, C:\IPCE-r6\Lib\compiler\ast.py exists and has the values required.  It's just that its __init__.py is empty. , so we can patch C:\IPCE-r6\Lib\compiler\__init__.py to be

import ast

and it all works, without having to touch PyFit code at all.

This is the "no change to client code" equivalent to decorating PyFit with a suitably guarded explicit load

import fepy
fepy.install_option('ast')

The behaviour is not a bug, but a feature as FePy by default does not load code that is rarely used, so as to speed start-up.

Note

We still don't have parsing of lists, tuples, dictionaries or complex numbers as arguments, since we still don't have the transformer.parse method to expose as compiler.parse; the unit tests in TypeAdapterTest.py report

Ran 117 tests in 2.468s

FAILED (failures=3, errors=5)

However real tests that don't have these non-scalar types do run happily on FePy-r6.

Some C and Java test and analysis tools

For static analysis of Java, the PMD tool (http://pmd.sourceforge.net/) can provide lint-like (or FxCop-like, depending on your exposure to these things) coverage of your source, with a configurable rule-set. It can generate its reports as part of an Ant build, and be integrated live into most Java IDEs to give live reports.

Like most of these tools, you will curse the first time you expose legacy code to it, and aim to clear the reports. In code that has to also play nice with .Net, you'll have to switch off two rules MethodNamingConventions and LongVariable; and some of its rules e.g. about empty default constructors cannot always be all consistently applied (so may need case-by-case suppression).

To go along with the use of JUnit test, Cobertura (http://cobertura.sourceforge.net/) is a coverage tool that works by instrumenting the Java bytecode of the generated classes. It provides Ant tasks (and thus can be manually inserted into any Ant based build system, such as NetBeans projects) -- instrument the normal output in a post-compile step, and run JUnit against the instrumented code.

This tool gives you line and branch coverage reports like this sample -- all the way down from package-level summaries to line-by line indications.

While most of this post is about Java, I might as well add to the pot a 'C'-based tool, Splint, which, as the name suggests, is an uprated lint-style tool, which includes in its analysis some basic probing for code that might be vulnerable to buffer overrun -- at least in cases when a buffer is passed into a routine without any length, and then gets written to.

I've not managed to track down any good free tools for C++ -- Microsoft's Prefast (at least as of 2 years ago) balked when fed code containing STL headers; other tools are 'C' only, or pay-ware. Given the syntactic complexity of C++, this is not so surprising.

The hidden delights of Unit testing

One of the things that has left this blog light of content of late is having been provided with internal blogging, wherein I try to enlighten my colleagues. Often the entries on that blog are collations of links to and through other blogs I read. Some are from experience. Like this one.

One part of the current development project has brought home to me quite how much how serious unit testing results in cleaner code -- and that the closer you strive for 100% coverage in the testing, the more incentive there is to write that clean code.

The closer to 100% you strive to get, especially with a coverage tool (such as gcov) that does branch coverage rather than just line coverage, the more it squeezes your code. At the most brutal, the more code you have, that means the more tests there are to write to cover it all -- the incentive is there to make the code tighter, just to reduce the amount of work to do for completion.

Much of the 'C' code being written contains routines that are explicitly each a little state machine. As such the structure of a routine is along the lines of

• check preconditions
• determine "state"
• "switch" on the current state
• tidy
• return outcome

Some of the precondition checks are assert() but others cannot be (so will include an exit-on-failure); and the switch may not be a simple flat one -- some cases may have sub-cases; and some might overlap, in a structure like

where a and b are independent.

However, even if you're not explicitly thinking of the code as a state machine as such, the routine structure is still quite generic.

Coverage testing

A set of unit tests can make sure that expected inputs map to expected outputs, both positive and negative; coverage helps tell you if you have "enough" tests. It answers the questions "Has all that code been exercised, yet?" (if not the related "if it doesn't get used, why did you write it in the first place?").

The first thing that a set of obvious positive tests will show are the bits that are difficult to reach. The obvious one is handling exception states -- and here automation and good mocks in the test framework, or your own wrapper to it, are essential. After all, exceptions are meant to be, well, exceptional, but here need to be generated on demand.

With those out of the way, the real difficult-to-reach corner-cases of the logic stand out -- and with only those to concentrate on, you're either faced with writing a lot of tests to reach them; or figuring a way to simplify the code so you don't have to.

It is often tempting to write code like this:

but, damn it, if arranging the case a and c is hard work, you don't want to go through the slog with b as well. Factoring out the special case goes from being something you could do, if you had the enthusiasm, to something you want to do, because it's less work than writing the extra tests. "Don't Repeat Yourself" becomes positively encouraged.

Coverage types

The code metric you use is important in how much benefit you can derive. For a first pass, NCover isn't too bad. But it only counts line visits, being as it is an instance of the profiling API for .Net. In particular if you have code like--

NCover will never show you that you're missing the case of zero and negative values of a. One of the up-sides of working in 'C' on a *nix platform is that that has meant that gcov is available. And that will take code like--

and distinguish between whether a or b triggered the do something -- 100% in NCover usually isn't more than 90-odd% in gcov

Squeezing out the logic

Here's a real example of code improvement in making the last step to 100% branch coverage

I had 100% in NCover; but gcov reminds me that I don't cover all the bases -- because have_token and need_token aren't independent variables : if you don't need the token, you should never have one. So, what to do when aiming for the 100% mark?

The routine here started in a state where I just enumerated all possible cases (there are more than just these), handling them individually in some sort of logical order. Now, the unit tests I already have provide me a framework to check that the code is still doing what I mean it to do when I refactor; so I can look at the code and see that what I have is actually of the form

or, more simply

refactor, re-run the tests and see that the simpler code is still right.

Similarly code guarded by an if clause, where the else is never executed under any input you can generate, perhaps because the assert() defined contract of the method or its callers enforces the constraint, can be simplified to and assert() of the condition and an unconditional block. And you get that better code because you've made yourself go the last little bit.

In the case above, user input could have, but not need, the cookie; we can't assert -- but we don't need to write (though we can) another test case to prove that is harmless, because that is just another flavour of the "else".

Painfully learned lessons

or, for this, I wrote my car off.

As I noted a while back, for the last few months I've been busy on a project that's actually intended to be released. It was getting home from another Saturday at work (the 13^th consecutive working day for all those trisdekaphobes out there) when I skidded in the sudden snow and slush and bent the car gently against a telegraph pole. The obvious damage was confined to the headlight, wing and wheel, but - given that it was over 10 years old, and a rear-ending back in '96 - just after the 3 year old mark - came close to being a write-off - I wasn't surprised when the garage told me it was a total loss.

What I'd been working on was a report of a memory leak detected in log-on stress tests. And there were a couple of obvious ones - stateless objects in .asp files that weren't being nulled, and would be better as globals anyway.

Next round, running the most significant COM object directly, I learned that CComBSTR is the invention of the very Devil. If you expect it to behave like std::string or even CString, you'll almost certainly leak memory. Better to write your own wrapper and count all the SysAllocStrings out and all the SysFreeStrings in again, with SysReallocStrings and any widening of data to append done under your own control. Run the direct test harness again - after starting transients, from 2000 uses to 25000 uses, memory and handles steady as a rock.

So I ran the program I'd coded using WinInet to drive a dumb web client, coded to match the test harness script (that used some test software I didn't have) to drive scripts plus COM objects. Steady as she goes. Hand over to test.

Leaking like a sieve, they say. At this point I went home, tired, perplexed and annoyed, to have the unexpected detour into a ditch.

Next lesson, Monday morning. My test harness was looping forever doing create an HINTERNET, create a session, use a number of HTTP requests, tear it all down, and repeat; theirs was using scripting to do the same set of GETs and POSTs. WTF!? I thought. Then I looked at a network trace.

If you use Connection: Keep-Alive in your requests, even if you destroy every HINTERNET and create then all anew, starting with a whole new Internet Session, the same TCP connection is held underfoot. (I'd probably have had to unload and re-load wininet.dll to force a connection close). So it was all within one ASP session, even though I was running what I thought were many. So remove the for(;;) and loop in a script. Suddenly the leak shows.

The guy who wrote the scripts looks over my shoulder at this point and asks "why are you calling this page with that argument?" - so I show him the test script. "Are you sure?" he says. So I do the obvious - capture a network trace with a real browser exercising the real server. And find that the script does indeed depart from what a browser does - it misses out the final crucially important GET-after-redirect to the page that winds up the session.

Adapt my test harness to match reality. Run it again. Steady as a rock. At this point it is probably just as well that the test team are 70 miles away, as I swear if one of them had been in the room, I would have bludgeoned them to death with my chair. But as the Ash Wednesday Chopping Block says, there are, alas, things that one can't validly give up for Lent.