dotCover 1.0 beta -- first impressions

The beta of the coverage tool from the same people who brought you Resharper came out late last week, and I took the chance to have a quick play with what it offers.

As expected, it integrates smoothly with the R# unit test running -- you can see exactly what code your test or set of tests cover; marked in green overlay in the source view for covered by the test(s) you just ran, red elsewhere, in a manner reminiscent of NCoverExplorer.

There is also a command-line option

Usage: JetBrains.dotCover.ConsoleRunner.exe /Executable='executable' [/Arguments='arguments'] [/WorkingDir='working dir'] /Output='output' 

Not having R# at home -- because I rarely use C# even if I'm writing for .net -- it's this which interested me more. So, I gave it a try on my current F# project

>& 'C:\Program Files\JetBrains\dotCover\v1.0\Bin\JetBrains.dotCover.ConsoleRunner.exe' /executable='..\..\..\_Tools\Microsoft FxCop 1.36\FxCopCmd.exe' /Arguments='/q /s /console /f:Tinesware.Recorder.dll /f:Tinesware.Instrumentation.exe /f: Tinesware.Infrastructure.dll /f:Tinesware.Rules.dll /f:Tinesware.InfrastructureTests.dll /f:BaseTests.dll  /rule:Tinesware.Rules.dll /o:fxcop.xml  /rule:.\Rules /dictionary:..\..\..\CustomDictionary.xml' /workingdir=. /output=dotcover.xml

which yields a bald total coverage number

JetBrains dotCover Console Runner v1.0.56.11 JetBrains s.r.o.
Coverage session started [12/07/2010 20:14:16]
Coverage session finished [12/07/2010 20:14:28]
Index files: ...AppData\Local\Temp\ssc06AE2.tmp
Log files: ...AppData\Local\Temp\lgc02A14.tmp
Opening snapshot...
Done.
Generating report...
Done.
Total coverage: 78%

and an XML report file that makes it clear that that percentage is based on all assemblies dragged into the AppDomain, counting those which have no pdb information as containing no statements to cover, which means if you're running unit tests with Rhino.Mocks that it includes things like

<Assembly Name="021925b1-cbe9-42d3-ad44-a39206195c2a" CoveredStatements="0" TotalStatements="0" CoveragePercent="0">
    <Type Name="IsolatorProxyb66a653c9a984566ac790c4fa654cab5" CoveredStatements="0" TotalStatements="0" CoveragePercent="0">
      <Type Name="InvocationShouldCallOriginal_1" CoveredStatements="0" TotalStatements="0" CoveragePercent="0">
        <Member Name="HandleEvent" CoveredStatements="0" TotalStatements="0" CoveragePercent="0" />

emanating from dynamic assemblies. And it seems from simple experiment that the filters that you can apply in Visual Studio don't seem to carry over to the command-line version.

The report is formatted as shown above, with hierarchy

Root -> Assembly -> Namespace -> Type -> Nested Type or Member (repeating Type as often as necessary)

There’s also no granularity below Member, so even for code with .pdb information available the reports only look like

<Type Name="Local" CoveredStatements="166" TotalStatements="173" CoveragePercent="95">
...
        <Member Name="get_Mutex" CoveredStatements="0" TotalStatements="0" CoveragePercent="0" />
        <Member Name="ExpandFile" CoveredStatements="2" TotalStatements="4" CoveragePercent="50" />
        <Member Name="LoadFile" CoveredStatements="4" TotalStatements="7" CoveragePercent="57" />
        <Member Name="op_GreaterQmarkGreater" CoveredStatements="4" TotalStatements="4" CoveragePercent="100" />
        <Member Name="op_GreaterPlusQmarkQmark" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="op_GreaterMultiplyGreater" CoveredStatements="4" TotalStatements="5" CoveragePercent="80" />
        <Member Name="op_GreaterPlusQmark" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="op_GreaterPlus" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="GetXmlData" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="DoWithLock" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="op_Append" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="GetByName" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="SetByName" CoveredStatements="1" TotalStatements="1" CoveragePercent="100" />
        <Member Name="GetStatementRange" CoveredStatements="2" TotalStatements="2" CoveragePercent="100" />
        <Member Name="FindSignatureForMethod" CoveredStatements="8" TotalStatements="8" CoveragePercent="100" />
        <Member Name="FindXmlForMethod" CoveredStatements="7" TotalStatements="7" CoveragePercent="100" />
        <Member Name="get_DeclaredExempt" CoveredStatements="0" TotalStatements="0" CoveragePercent="0" />
        <Member Name="get_AutomaticExempt" CoveredStatements="0" TotalStatements="0" CoveragePercent="0" />
...
      </Type>

So there can be no useful equivalent of NCoverExplorer for after the fact detail – to get detailed line un-coverage information you actually have to run every one of your unit tests in Visual Studio in one big bang.

Its metric of what counts as a statement is subtly different from what NCover (free) and the other tools I've been working on recently. It reports an F# property like

let AutomaticExempt = "-2"

or an equivalent C# automatic property, for that matter, as having 0 lines, rather than N/A and 1 respectively; but that aside it seems to be counting the self-same list of sequence points.

So, it seems that this is primarily a tool for interactive use and positive coverage testing -- "Does this set of tests cover this line, and if so, which one(s)?"; in a build process environment the lack of ability to focus down to only the assemblies of interest in the report (discarding tool and system code), and of per-statement reporting it is less interesting.

However it did take the F# code which PartCover had balked at in its stride.

Meanwhile in other news

There is a branch of PartCover that supports .net 4 originally on GitHub and which has now been merged into the main-line at SourceForge, removing one of the objections to using this tool. There also seems to be rather more life in this than there was six months ago when I tried it, so I shall have to revisit it and see if the IL issue is also resolved.