Building PartCover 4 on Vista

Updated again: PartCover Revision 35 includes a patch I provided that builds the same functionality in C#, transiently registering the COM component in HKCU during build, and also providing a --register command line option which is equivalent to the old nCover //r option.

Updated with new complete registry frobbing script, based on actually diffing the before and after states of HKCR when using regsvr32 on the PartCover dll.

Following on from earlier in the week, I pulled the current trunk from SourceForge, and set about building it then putting it to work on some F# code which had caused an earlier version (the most recent available at the end of last year) to balk.

Unzipping the tools (Boost, ATL server) into the appropriate libraries, guided by the include paths (and moving the Debug relative paths to match the Release ones) was trivial, and then it's just a case of firing up MSBuild. Or so I thought.

First hiccup -- it attempts to register the PartCover assembly during the build, and I don't run as administrator by default. So I replaced the

regsvr32 /s /c "$(TargetPath)"

by a manual registry frobbing trick (akin to what NCover 1.5.8 used for its //r option)

"$(ProgramFiles)\FSharp-2.0.0.0\bin\fsi.exe" "$(ProjectDir)\partcover.fsx" "$(TargetPath)"

where partcover.fsx is

This updated per-user registration permits both building and running the PartCover utility without needing full administrator privilege.

So emitting

...\PartCover.exe --target "..\..\..\_Tools\Microsoft Fxcop 10.0\FxCopCmd.exe"  --target-work-dir . --target-args "/q /s /console /f:Tinesware.Recorder.dll /f:Tinesware.Instrumentation.exe /f:Tinesware.Infrastructure.dll /f:Tinesware.Rules.dll /f:Tinesware.InfrastructureTests.dll /f:BaseTests.dll /rule:Tinesware.Rules .dll /o:fxcop.xml /dictionary:..\..\..\CustomDictionary.xml" --include [Tinesware.*]* --include [CSharpTests]* --output PartCoverage.xml

to monitor code coverage when using an FxCop rule written in F#, this reported

open driver pipe
modify target environment variables
create target process
wait for driver connection
[00000] [05724] Options dump:
[00000] [05724]   VerboseLevel: -842150451
[00016] [05724]   Log file: ...\Infrastructure\_Binaries\Tinesware.InfrastructureTe
sts4\Debug+AnyCPU\partcover.driver.log
[00016] [05724]   Log pipe: yes
[00016] [05724]   Count Coverage - ON
[00016] [05724]   Count Call Tree - OFF
[00016] [05724]   Exclude [mscorlib]*
[00016] [05724]   Exclude [System*]*
[00016] [05724]   Include [Tinesware.*]*
[00016] [05724]   Include [CSharpTests]*
Project : warning : CA0060 : The indirectly-referenced assembly 'Microsoft.VisualStudio.CodeAnalysis, Version=10.0.0.0,
Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' could not be found. This assembly is not required for analysis, howeve
r, analysis results could be incomplete. This assembly was referenced by: ...\Infra
structure\_Binaries\Tinesware.InfrastructureTests4\Debug+AnyCPU\FxCopSdk.dll.
[04774] [05724] CorProfiler is turned off
Target PageFaultCount: 33536
Target PagefileUsage: 84246528
Target PeakPagefileUsage: 87359488
Target PeakWorkingSetSize: 92827648
Target QuotaNonPagedPoolUsage: 10920
Target QuotaPagedPoolUsage: 328336
Target QuotaPeakNonPagedPoolUsage: 43160
Target QuotaPeakPagedPoolUsage: 374344
Target WorkingSetSize: 89481216
Total 0 bytes ...\Infrastructure\_Binaries\Tinesware.InfrastructureTests4\Debug+AnyCPU

which completed cleanly, unlike before, and gave a sane looking output file. There is now obvious code in the system for doing the short branch instruction fix-up, which seems to have resolved the issue I had last time I tried this tool.

So, it really does look good to go for .net 4 code -- and I just have to add hooks to consume that format as well as the old NCover style.

Links for 13-Jul

Hacking PoSh 2.0 to run under .net 4 -- a highly unofficial fix-up.

PowerGUI 1.0 RTM.

Setting up MSFT's WebMatrix beta.

Learning resources for jQuery.

Mozilla's Rust programming language.

The Active Object pattern for concurrency.

dotCover 1.0 beta -- first impressions

The beta of the coverage tool from the same people who brought you Resharper came out late last week, and I took the chance to have a quick play with what it offers.

As expected, it integrates smoothly with the R# unit test running -- you can see exactly what code your test or set of tests cover; marked in green overlay in the source view for covered by the test(s) you just ran, red elsewhere, in a manner reminiscent of NCoverExplorer.

There is also a command-line option

Usage: JetBrains.dotCover.ConsoleRunner.exe /Executable='executable' [/Arguments='arguments'] [/WorkingDir='working dir'] /Output='output' 

Not having R# at home -- because I rarely use C# even if I'm writing for .net -- it's this which interested me more. So, I gave it a try on my current F# project

>& 'C:\Program Files\JetBrains\dotCover\v1.0\Bin\JetBrains.dotCover.ConsoleRunner.exe' /executable='..\..\..\_Tools\Microsoft FxCop 1.36\FxCopCmd.exe' /Arguments='/q /s /console /f:Tinesware.Recorder.dll /f:Tinesware.Instrumentation.exe /f: Tinesware.Infrastructure.dll /f:Tinesware.Rules.dll /f:Tinesware.InfrastructureTests.dll /f:BaseTests.dll  /rule:Tinesware.Rules.dll /o:fxcop.xml  /rule:.\Rules /dictionary:..\..\..\CustomDictionary.xml' /workingdir=. /output=dotcover.xml

which yields a bald total coverage number

JetBrains dotCover Console Runner v1.0.56.11 JetBrains s.r.o.
Coverage session started [12/07/2010 20:14:16]
Coverage session finished [12/07/2010 20:14:28]
Index files: ...AppData\Local\Temp\ssc06AE2.tmp
Log files: ...AppData\Local\Temp\lgc02A14.tmp
Opening snapshot...
Done.
Generating report...
Done.
Total coverage: 78%

and an XML report file that makes it clear that that percentage is based on all assemblies dragged into the AppDomain, counting those which have no pdb information as containing no statements to cover, which means if you're running unit tests with Rhino.Mocks that it includes things like

<Assembly Name="021925b1-cbe9-42d3-ad44-a39206195c2a" CoveredStatements="0" TotalStatements="0" CoveragePercent="0">
    <Type Name="IsolatorProxyb66a653c9a984566ac790c4fa654cab5" CoveredStatements="0" TotalStatements="0" CoveragePercent="0">
      <Type Name="InvocationShouldCallOriginal_1" CoveredStatements="0" TotalStatements="0" CoveragePercent="0">
        <Member Name="HandleEvent" CoveredStatements="0" TotalStatements="0" CoveragePercent="0" />

emanating from dynamic assemblies. And it seems from simple experiment that the filters that you can apply in Visual Studio don't seem to carry over to the command-line version.

The report is formatted as shown above, with hierarchy

Root -> Assembly -> Namespace -> Type -> Nested Type or Member (repeating Type as often as necessary)

There’s also no granularity below Member, so even for code with .pdb information available the reports only look like

<Type Name="Local" CoveredStatements="166" TotalStatements="173" CoveragePercent="95">
...
        <Member Name="get_Mutex" CoveredStatements="0" TotalStatements="0" CoveragePercent="0" />
        <Member Name="ExpandFile" CoveredStatements="2" TotalStatements="4" CoveragePercent="50" />
        <Member Name="LoadFile" CoveredStatements="4" TotalStatements="7" CoveragePercent="57" />
        <Member Name="op_GreaterQmarkGreater" CoveredStatements="4" TotalStatements="4" CoveragePercent="100" />
        <Member Name="op_GreaterPlusQmarkQmark" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="op_GreaterMultiplyGreater" CoveredStatements="4" TotalStatements="5" CoveragePercent="80" />
        <Member Name="op_GreaterPlusQmark" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="op_GreaterPlus" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="GetXmlData" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="DoWithLock" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="op_Append" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="GetByName" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="SetByName" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="GetStatementRange" CoveredStatements="2" TotalStatements="2" CoveragePercent="100" />
        <Member Name="FindSignatureForMethod" CoveredStatements="8" TotalStatements="8" CoveragePercent="100" />
        <Member Name="FindXmlForMethod" CoveredStatements="7" TotalStatements="7" CoveragePercent="100" />
        <Member Name="get_DeclaredExempt" CoveredStatements="0" TotalStatements="0" CoveragePercent="0" />
        <Member Name="get_AutomaticExempt" CoveredStatements="0" TotalStatements="0" CoveragePercent="0" />
...
      </Type>

So there can be no useful equivalent of NCoverExplorer for after the fact detail – to get detailed line un-coverage information you actually have to run every one of your unit tests in Visual Studio in one big bang.

Its metric of what counts as a statement is subtly different from what NCover (free) and the other tools I've been working on recently. It reports an F# property like

let AutomaticExempt = "-2"

or an equivalent C# automatic property, for that matter, as having 0 lines, rather than N/A and 1 respectively; but that aside it seems to be counting the self-same list of sequence points.

So, it seems that this is primarily a tool for interactive use and positive coverage testing -- "Does this set of tests cover this line, and if so, which one(s)?"; in a build process environment the lack of ability to focus down to only the assemblies of interest in the report (discarding tool and system code), and of per-statement reporting it is less interesting.

However it did take the F# code which PartCover had balked at in its stride.

Meanwhile in other news

There is a branch of PartCover that supports .net 4 originally on GitHub and which has now been merged into the main-line at SourceForge, removing one of the objections to using this tool. There also seems to be rather more life in this than there was six months ago when I tried it, so I shall have to revisit it and see if the IL issue is also resolved.