Hacking Fake.Build for net7.0

It's too early yet to do the AltCover after-action for net7.0 like I did for net5.0; but it is proving a slog like net5.0 rather than being too transparent to mention like net6.0 was.

While the dotnet 7.0.100 build has its own problems, the Fake build system currently (5.23.x, 6.0 alpha) doesn't handle it at all - the former may work for limited scenarios, but more complicated ones fall over with library version mismatches; the latter politely askes for a v6 runtime and stops.

So, you have a build process based on a Fake script, but want to run with v7.0.100 as SDK, through your project global.json. What do?

Well, the Fake documentation gives some alternatives - turn the script into a net7.0 project that links against the Fake libraries; but that imposes some restrictions on how you set up targets, because you can only define such things (as opposed to the functions that define the target's intended action) after setting up a build context. Or you can run as a fsi script, but the documentation then leads into how to hack paket into the mix.

However, since net5.0, scripts are able to reference nuget packages directly, so rather than having a paket.dependencies, with a #r "paket: groupref [whatever] //", and a call out to paket on the command line, your scripts can be self-contained, with a block of #r "nuget: [library][, optional version]" lines at the top, and a command line that is just dotnet fsi [my fake script.fsx]

And doing it this way, you can build on the net7.0 runtime.

Yet another MSBuild-on-Linux back-slash gotcha

Another variant of a known family of such issues, encountered as AltCover issue #49, the interesting tale of what you get when your MSBuild sets a task array parameter with

AssemblyExcludeFilter="$(AltCoverAssemblyExcludeFilter.Split('|'))"

and the value of AltCoverAssemblyExcludeFilter is the-|xunit\.; surprisingly, the answer is

the-
xunit/.

There isn't even an explicit ItemList in sight, and yet the '\' still gets interpreted as a path separator and "helpfully" *nix-ified. Doubling up the '\' via .Replace('\','\\') ahead of the split doesn't escape the character -- you just get a '//' in the output, nor does escaping on the command line as the-|xunit%5C..

For the moment, with no obvious MSBuild level fix, I'm working around this by doing .Replace('\', %00) and then converting the NUL back inside the custom MSBuild task.

`dotnet build`ing Eto.Forms projects

A couple of months back, Scott Hanselman blogged about the Eto.Forms cross-platform UI toolkit for .net. I filed that away for reference, and this weekend got around to giving it a whirl.

So I took the default new F# app and made a personalized little skeleton project (standard boiler-plate for the "About..." dialog, that sort of thing), and as part of making it re-usable as such, added a trivial FAKE build script. Eto.Forms generates new-style projects, so of course that would be using the DotNet.build and DotNet.publish tasks...

Not so fast.

Do that and it barfs with

error : MSB4801: The task factory "CodeTaskFactory" is not supported on the .NET Core version of MSBuild.
error MSB4175: The task factory "CodeTaskFactory" could not be loaded from the assembly "C:\Program Files\dotnet\sdk\2.1.200\Microsoft.Build.Tasks.Core.dll". The task factory must return a value for the "TaskType" property.

which turns out to be a known issue (or two). Rather than sit tight and wait, I did a bit of looking around. Turns out there's a similar, closed, issue against msbuild itself, to which the resolution is to use the RoslynCodeTaskFactory as a replacement.

So, after some experimentation, here's a work-around while a fix works its way through the system; it functions by overriding the _BuildAppBundle target, so can be fitted into existing build processes without hacking any of your downloaded packages.

1. Add the RoslynCodeTaskFactory NuGet package to each affected project (probably just the .Desktop one
2. Get the override target file MacTemplateOverride.targets from this gist
   
3. Adjust the path to your eto.platform.mac64 version (assumed 2.4.1 in default location) as needed


4. Add <Import Project="[path to]\MacTemplateOverride.targets" /> at the end of each affected project to load the file from where you've saved it in your solution


5. dotnet build

Configuring Jenkins with PowerShell

No sooner do I say I have nothing technical to write up of general interest, than I spend a day stitching together pieces across the internet, because most Jenkins examples tend to be written to *nix or the JVM, and I'm on Windows where the admin tool of choice is PowerShell.

So, start on the Jenkins wiki page for Authenticating scripted clients, which tells you how to get your API key -- visit $(JENKINS_URL)/me/configure in your browser and look for the API token so you don't have to script your password (especially if you're using AD authentication on the server). Setting username/API Token as a Credentials object on a WebClient will just get you 403 errors, until you notice that the Groovy script example sets pre-emptive auth on its web client. The secret to HTTP Authorization and .NET WebRequest, WebClient Classes needs to be sought separately.

At this point you can GET from $(JENKINS_URL)/job/[job name]/config.xml, with DownloadString and cast to [xml] PowerShell style to read and modify.

Then you have to POST the modified XML back; but if you just do that with the Basic auth header, suddenly more 403 out of nowhere, until you read the small print about the Jenkins Remote access API about CSRF protection. When you get that and add it to your headers, it's now just a case of using UploadString to push the xml as xml.

Putting it all together we get

This appears to correctly preserve line endings as is, so you don't need to do anything non-default, equivalent to the --data-binary as you need for scripting with curl.

PowerShell 3 Start-Process MSBuild.exe -Wait hang

Having recently upgraded a Win7 machine that I had previously been using to check PoSh 2 compatibility (and had forgotten was not upgraded) to PoSh3, a build script launched from a PoSh prompt that did Start-Process MSBuild.exe -Wait -PassThru - ArgumentList ... started hanging on the wait more often than not, with Process Explorer showing the MSBuildTaskHost executable still lingering outside of the PowerShell process tree.

Killing that MSBuildTaskHost process usually unwedged the process, so it could go on to do the post build analyses in the rest of the script. Running with DISABLEOUTOFPROCTASKHOST=1 to avoid the separate process was worse -- there was no process around to kill and maybe release the PowerShell session when it hung.

Replacing the cmlet with a direct invocation & MSBuild.exe ..., and using $LastExitCode rather than the process object ExitCode property value seems to avoid the hang so far.

I don't know whether it's just MSBuild that's affected by this hang on wait, but I've not seen this behaviour elsewhere yet.

Playing with (almost) the latest C++ features -- groundwork

Having had my interest in playing with native code reawoken by the new C++11 features, the first thing I went to look at was portability. One of the advantages of managed code -- JVM or CLR -- is that the VM handles portability for you and the code can be built pretty much anywhere; with native code we have to see what the compilers have in common.

I've been using VC++2010 on Windows as having many of the "big rocks" for the new standard, while being backward compatible onto OS versions before Win7 (unlike the VC++11 compiler and its runtime); and for *nix-like systems, I have cygwin and debian squeeze... Well the distro support for these is a bit behind-hand (gcc versions 4.5 and 4.4 respectively), whereas gcc 4.7 is now out with quite a broad coverage of the new standard. While 4.5 has a good chunk of the new stuff, 4.4 doesn't -- in particular, it doesn't have the new lambda syntax. So, it's build from source time to get an upgrade to a sensible version there, which means I might as well go to the latest and greatest on both platforms...

Building gcc 4.7 from source

debian

Fortunately there are some handy instructions out there which can be used as a baseline for debian. Following them, I found that I needed to tweak how I built GMP to fit PPL's requirements, by modifying the configure step to be:

CPPFLAGS=-fexceptions ../../sources/gmp-5.0.4/configure --prefix=$PROJECT_DIR/output/ --enable-cxx

before PPL would go through happily. The --enable-cxx is required for the PPL ./configure stage to run through, the CPPFLAGS=-fexceptions is optional, but it avoids a make-time warning about possible unwanted runtime behaviours if you don't.

ClooG also needed a CPPFLAGS=-I$PROJECT_DIR/output/include CFLAGS=-L$PROJECT_DIR/output/lib on the configure line to find GMP.

In the gcc build, as well as pointing at ../../sources/gcc-4.7.0/configure it's also worth taking the advice from the MacOS build instructions and only selecting languages of interest to you i.e. to play with new C/C++ there's no need for Java or Fortran, at a considerable saving in time.

Then it's a matter of just adding soft links from whichever g*-4.7 files to the unadorned versions, and

export LD_LIBRARY_PATH=~/gcc-4.7/output/lib/
export PATH=~/gcc-4.7/output/bin:$PATH

to your .bashrc or equivalent

cygwin

Cygwin is more fun -- I've not yet managed to get that to build all the way through with the loop optimization libraries. When you get to PPL, you find you also need to go back and re-configure GMP with --disable-static --enable-shared, as explained in the friendly manual, to build the shared library version. However then when building gcc, we get a mismatch with the earlier libraries in the configure stage, where it just stops:

checking for the correct version of gmp.h... yes
checking for the correct version of mpfr.h... yes
checking for the correct version of mpc.h... yes
checking for the correct version of the gmp/mpfr/mpc libraries... no

It is possible that if you start by building PPL and CLooG with shared library GMP in a first pass, then build the rest starting with reconfiguring and building a static GMP it will work, but life is too short. The MacOS build instructions didn't use the PPL/ClooG/graphite libraries either, so we can do this to configure gcc instead:

$ ../../sources/gcc-4.7.0/configure          \
>     --prefix=$PROJECT_DIR/output/    \
>     --with-gmp=$PROJECT_DIR/output/  \
>     --with-mpfr=$PROJECT_DIR/output/ \
>     --with-mpc=$PROJECT_DIR/output/  \
>     --program-suffix=-4.7            \
>     --without-ppl --without-cloog    \
>     --enable-languages=c,c++

which sits and cooks for quite some time to get you the new compiler build.

Linkbait: fixing cygwin "mkdir foo mkdir: cannot create directory `foo': Permission denied"

I got into a state where I had this error, which other people have seen, after having tried to trash the build output of one of the failed PPL/CLooG attempts from Windows Explorer, where anywhere under in my home directory and down it was rejecting mkdir with "mkdir foo mkdir: cannot create directory `foo': Permission denied". Having spotted a tangentially related mailing list message about this sort of problem happening on network shares and there being ACL related, I tried the following and it worked to clear things up:

1. Start a PowerShell console as Administrator window
2. Run Get-Acl a folder (like /tmp) which you can mkdir in in cygwin (this will be %cygwin_root%\tmp where %cygwin_root% is where you installed cygwin)
   $acl = Get-Acl C:\cygwin\tmp
3. In Windows Explorer set yourself Full Control on all the affected folders -- %cygwin_root%\home and %cygwin_root%\home\%USERNAME% at least
4. In the PowerShell, Set-Acl on each folder you've just frobbed with the saved ACL object
   Set-Acl C:\cygwin\home $acl
   Note that the Set-Acl call may take considerable time (tens of seconds) to execute when it gets to the really problematic node and has to roll permissions down.
   

SCons -- Death to makefiles

Since I last did native code seriously at home (c. year 2000), I had discovered the very handy MiniCppUnit tool as a nice light-weight unit testing framework, so of course I went and fetched a copy to be going on with. This time, curiosity prompted me to wonder "WTF is this SConstruct file anyway?" and now when I opened it, I immediately recognised that it was some form of Python script, and wondered what sort of Python based make system this might be.

It was simple enough to find where it came from -- http://www.scons.org/ -- and looking at the user guide, I felt that it is much more intuitive system than makefiles (admittedly there's not a high barrier there), and far less cluttered than declarative XML based systems like Ant or MSBuild; so I'll be using that for my *nix builds -- it works very nicely for doing things like running unit tests as part of the build e.g.

MiniCppUnit -- Building it under modern C++

Just like I had to patch it to build in C++/CLI, I needed to make some changes to MiniCppUnit to get it to build clean under VC++2010, the out-of-the-box gcc 4.5 on cygwin 1.7.x and gcc 4.7 debian squeeze with -Wall -std=gnu++0x or -Wall -std=c++11 on respectively. First MiniCppUnit.hxx:

92c92
< #if _MSC_VER < 1300
---
> #if defined(_MSC_VER) && _MSC_VER < 1300
93a94
> /* Without the "defined(_MSC_VER) &&" this code gets included when building on cygwin 1.7.x with gcc 4.5.3 at least */
204c205
<  static void assertTrue(char* strExpression, bool expression,
---
>     static void assertTrue(const char* strExpression, bool expression,
207c208
<  static void assertTrueMissatge(char* strExpression, bool expression, 
---
>     static void assertTrueMissatge(const char* strExpression, bool expression, 
304c305
<    catch ( TestFailedException& failure) //just for skiping current test case
---
>    catch ( TestFailedException& /*failure*/) //just for skiping current test case

and the corresponding signature change in the .cxx file:

108c108
< void Assert::assertTrue(char* strExpression, bool expression,
---
> void Assert::assertTrue(const char* strExpression, bool expression,
122c122
< void Assert::assertTrueMissatge(char* strExpression, bool expression, 
---
> void Assert::assertTrueMissatge(const char* strExpression, bool expression, 

Of course there may be other things lurking to be scared out when I ramp up my standard warning levels to beyond the misleadingly named -Wall (when you have -Wextra, formerly -W, provided to switch on a whole bunch more including spotting signed/unsigned comparisons, before getting onto the really specialized ones) and switch on -Werror to force everything to be really clean.

Writing StyleCop rules in F#

Alas not "Writing StyleCop rules for F#" (or should that be StyleFop?), which would be nice, but a true job of work...

I don't see much need for more StyleCop rules for C# (except one to harness the FxCop spellcheck facilities to scan comments, which would have to be done via much dodgy reflection to winkle out internal types, and would anyway give issues for being tied to 32-bit native executables), so haven't tried this before. But, like mountains, it's there...

---

So, you have StyleCop 4.5 or 4.6 installed, and start by creating an empty class library targeted at the .net 3.5 environment, with added references to the StyleCop and StyleCop.CSharp assemblies. The bare rule class looks like

and to go with it, add an XML file as an Embedded Resource with name My.Namespace.MyRules.xml -- first gotcha : the long name needs to be given in full because the default namespace doesn't get applied in the build like it would for C#. That can then be filled in as per the instructions in the StyleCop SDK documentation. Then just override the AnalyzeDocument method and go wild:

where I'm using the concept from the StyleCop+ rule SP2000 as an example of a new rule not yet present in the core tool.

Unlike FxCop, there is not a "one class = one rule" model here; the classes exist to group related rules, and you control how each scans the current source file of interest, and, indeed, how distinct they are in the code -- a closely related set of rules could be checked off a single scan, and in some ways StyleCop rules are more akin to the distinct named resolutions that can be defined within a single FxCop rule.

Another consequent difference from FxCop is how the object graph is visited. In FxCop, there are VisitXxx methods to override that will be called for every object of type Xxx, in an essentially stateless manner (your rule class is responsible for maintaining state between calls); here the equivalent methods are passed to a WalkXxx graph walker method as a callback, provide a mutable state object argument, and can signal through their return value whether the traversal should continue.

This approach of using a mutable object, needing down-casting, to carry inputs into each call, and contain the resultant outputs, feels a bit odd in a functional environment; fortunately it is possible to bypass this in many cases by representing the tree-walk as a seq and operating on that. For the trailing whitespace rule, where the offending lines are most easily found by scanning the raw source line by line, all we need the traversal for is to find an ICodeElement context object matching that line number. A simple walk of the code element tree looks like

which we apply to the document RootElement and from the result find the last (so most deeply nested, and hence most constrained) item whose Location has a span that includes the line number of interest (skip while the end is before the line of interest, take while the start includes the line, reverse, get head), the line number being a value that the operation can close over.

For deployment onto machines that don't have F# installed, static linkage of the runtime via the --standalone build flag can be used rather than having to explicitly drop the F# runtime assembly into the StyleCop folder (second gotcha).

Scripting the Win32 API - F# and IronPython FFI

Mainly a worked example for reminding me of how the syntax goes, reimplementing sn -k in F# and IronPython:

This about halfway between how you'd just call the APIs naturally in C++/CLI, and the full process of C# P/Invoke. You do need to do your own extern method declarations, whereas IronPython is something else again:

where it was easier to fudge the pointer value for the key buffer into an IntPtr than to try and dereference it via ctypes and make a manual copy.

The code here is an adapted subset of a managed API for strong-name keys; the use-case being a contingent key generation as part of a build process.

MSBuild -- getting the assemblies you depend on from other projects

So in one project, you'd like to do something where you automatically pick up the other assemblies you've built in projects you depend on -- an obvious case in point being running some automated test based on the dependent projects. Just going for the list of references will pick you up a whole raft of system assemblies that you can't get rid of in .net4; so you need to turn the project references into the output names.

For that we have the MSBuild task and the GetTargetPath target, thus:

which is a trivial example where we just echo the values to the console, like

BeforeBuild:
  My reference list is AnotherAssembly.dll; NewAssembly.dll

Links for 25-Dec Bumper catch-up edition

2010 having spanned the run from first integration point through to RTM and GA for XenDesktop 5, I was being increasingly preoccupied as the year went on; and blogging of any form was one of the main casualties. Which means I have a not-so-little list of backlog of things I want to keep track of from the last couple of months:

F#

• F# Goes Open Source
• MonoDevelop plugin for F# now available
• The MVVM Pattern Using F#

.net General

• Creating Zip files with System.IO.Packaging namespace
• Kill That Util Class! with C# extension methods
• Debugging with Async -- C#5 CTP
• Dynamic Language Integration in a C# World
• Algorithmia Source Code released on CodePlex
• Mono: What we are Cooking (9-Dec-10)
• How to: Determine which Microsoft .Net Framework version and service pack installed?
• Decoding clr20r3 .NET exception – using mono cecil
• C# Closure Constructors -- from the original Javascript

Build Process

• Fixing Incorrect solution build ordering when using MSBuild.exe
• StyleCop Compliant Visual Studio 2010 Code Snippets November Release
• Taming Sandcastle: A .NET Programmer's Guide to Documenting Your Code
• ILMerge – How to Merge Assemblies After A Build
• MSI Explorer
• Creating a NuGet Package

Misc.

• An HTML5 Remote Desktop - Part II
• Are we documenting the wrong things?
• Javascript: Defining Classes with Closures

Links for 23-Feb

Martin Fowler on Version Control Tools.

PowerShell:

• Standard members property.
• MSBuild Task factory.

F#:

• Survival guide e-book.
• A simpler mailbox processor and agents.
• Structural typing constraints.

JavaScript Reactive Framework and Observables.

50 CSS+JavaScript techniques.

Ruby + Haskell.

Rake + Ant.

F# October '09 & February '10 CTP and VS 2008 - output window glitch

I first noticed this when adding a simple post-build step through the project properties for running unit tests as part of the build inside Visual Studio -- the May '09 CTP echoed the data going to stdout just like with any other language; in October everything went black; and it remains that way at the February '10 CTP.

Subsequently I find that this also applies to other MSbuild targets. For example, take a completely new F# library project, and edit the .fsproj file to have the following AfterBuild target:

<Target Name="AfterBuild">
    <Error Text="Hello world" />
  </Target>

Reloading and rebuilding gives

------ Build started: Project: Library4, Configuration: Debug Any CPU ------
        C:\Program Files\FSharp-1.9.7.8\\bin\fsc.exe -o:obj\Debug\Library4.dll -g --debug:full --noframework --define:DEBUG --define:TRACE --optimize- --tailcalls- -r:"C:\Program Files\FSharp-1.9.7.8\\bin\FSharp.Core.dll" -r:C:\Windows\Microsoft.NET\Framework\v2.0.50727\mscorlib.dll -r:"C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5\System.Core.dll" -r:C:\Windows\Microsoft.NET\Framework\v2.0.50727\System.dll --target:library --warn:3 --warnaserror:76 --vserrors --utf8output --fullpaths --flaterrors Module1.fs
========== Build: 0 succeeded or up-to-date, 1 failed, 0 skipped ==========

Using msbuild at a console, the output is shown

Project "C:\Users\Steve\Documents\Visual Studio 2008\Projects\Library4\Library4.fsproj" on node 0 (default targets).
  Processing 0 EDMX files.
  Finished processing 0 EDMX files.
CopyFilesToOutputDirectory:
  Copying file from "obj\Debug\Library4.dll" to "bin\Debug\Library4.dll".
  Library4 -> C:\Users\Steve\Documents\Visual Studio 2008\Projects\Library4\bin\Debug\Library4.dll
  Copying file from "obj\Debug\Library4.pdb" to "bin\Debug\Library4.pdb".
C:\Users\Steve\Documents\Visual Studio 2008\Projects\Library4\Library4.fsproj(51,5): error : Hello world
Done Building Project "C:\Users\Steve\Documents\Visual Studio 2008\Projects\Library4\Library4.fsproj" (default targets)
 -- FAILED.


Build FAILED.

On the up side -- messages emitted by the Error or Warning tasks (but not the Message task) do appear in the Error List tab; so you can get some feedback inside Visual Studio as to why your build failed, even if not all the realtime tracing you might be used to with other languages.

I suspect this may be as simple as a linker flag change to the VS plugin (all the plausibly relevant F# assemblies I've looked at in 1.9.7.8 are marked as GUI sub-system, and IIRC, VS plug-ins need to be Console -- i.e. to have a useful stdout).

Links for 7-Sep

F#

• Composing functions.
• Events and asynchronous workflows (series).
• Custom monads.
• Delayed Maybe monad.
  

Silverlight/Web

• Silverlight <canvas> emulator (follow-up).
• SVG for IE via Flash.
• Silverlight L10n.

PowerShell

• Exploring WMI.
• Formatted dumps of byte arrays.

Misc.

• Ultimate tool list for Windows.
• Exploit analysis plug-in for WinDbg post-mortem work.
• Winforms & WPF data UI without code or XAML.
• Visual Studio ignores some MSBuild conditional imports.
• Unit tests -- definition and best practice.

MSBuild -- build from solutions without DevStudio

This is a tool that had passed me by until I spotted a passing reference to it just the other day. Buried in your C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727 folder is an executable called MSBuild, which will take .*proj and .sln files and build them.

It consumes XML files in a format of which the .*proj files are a special case; and handles solutions (.sln files)  as a special case.  While I've not had chance to explore the ramifications of the XML build format, this appears to be Microsoft's own preferred build system and one that will bear investigation and adoption, especially for those projects (like Callisto) which use DevStudio for developer builds and thus have had it  installed on their build machines, when they could have gotten by with just the .Net 2.0 Framework.

MSBuild with solutions -- how to capture the temporary script file generated from a solution; and how to actually build just one project from a solution

MSBuild with ASP.Net -- Visual Studio 2005 no longer uses project files for ASP.NET; this is how to roll your own from the solution.

Solution files and MSBuild -- Automated hoisting of project build from the solution file.

These latter look like something that could be ported into an IronPython script with ease.

…and caught up. Phew!