Late May holiday cycling

We took the Bank Holiday week to go down to Netley Waterside again, after a couple of year gap, and for the first time, taking the folding bike I got last spring for holiday purposes, which meant that in just an initial quick ramble I could go further in either direction than the long walks I did back then.

The weather was fine, except for a wet Friday, and generally pleasant to be out in, even if at times through an uneasy balance of sultry heat and strong breezes, as for the expedition to the Sir Harold Hillier Gardens on the Bank Holiday Monday. In fact, the weather usually exceeded expectations set by the forecasts, such that I ended up acquiring more of a tan than I'd expected.

Having forgotten to pack a map, I ended up following the National Cycle Network 2 signs a lot, filling in the gaps by dead reckoning, especially in the places where the paved surface suddenly stopped, but being hemmed in between coast and motorway, that didn't leave too much scope for getting lost -- just scope for somewhat more ups-and-downs than I needed, grinding in bottom gear, as in the sub-optimal routes through the back doubles in Bursledon, and too much main A-road for crossing the Hamble. Next time, having discovered the foot ferry on the last way back, I might be able to cut out much of the main road and hills.

Hovercraft Museum, Lee-on-Solent

Also, lacking a map, I didn't realise quite how close I came to running out of road entirely, not realising that Portsmouth Harbour was that close.

D-Day plaque at the Rising Sun, Warsash

Many of the pubs I passed on my wanderings were rather sea-sidey, but the Rising Sun at Warsash was a more traditional rural pub, that overcame the handicap of being a Greene King house, and served most excellent sandwiches. Their take on chicken bacon mayo was not the vaguely unpleasant mush by the same name that were offered as packed lunches by the Revitalise team, but a coarsely sliced chicken breast, a couple of rashers, and a pot of mayo to add to taste on doorstops from a fresh soft bloomer loaf -- washed down with a pint of Rsing Sun bitter, an excellent lunch!

On the move I

Usual cycling round-up, with the end of the month numbers 11950.5 and 1370.4, with the belated arrival of the rains eating into my cycling days, for a disappointing 50.4 + 168.2 + 78 off-meter or 296.7 miles total for the month, and 1585 year to date, making 2000 for the half year a stretch.

Also, on Saturday 16th, I put my car past the 11,000 mark, so 1000 miles in 52 weeks, and under 2^10 in the year.

F# under the covers XV -- functions, types, and what you see isn't what you get

Consider this innocent example

	open System
	open System.IO
	open System.Runtime.Serialization.Formatters.Soap
	open System.Text
	let mhead(m : list<Object>) : list<Object> = [m.Head]
	let addN n =
	(fun x -> x + n)
	let add2 = addN 2
	let emitString x =
	let sf = new SoapFormatter()
	let sink = new MemoryStream()
	sf.Serialize(sink, x)
	printfn "%s" <| x.GetType().FullName
	printfn "%s" <| Encoding.Default.GetString(sink.ToArray())
	[ mhead :> obj; addN :> obj; add2 :> obj; emitString :> obj]
	|> Seq.iter emitString
	printfn "-------------------"
	[ mhead :> obj; addN :> obj; add2 :> obj; emitString :> obj]
	|> Seq.iter emitString;
	;;

view raw gistfile1.fs hosted with ❤ by GitHub

When run in a new interactive session, it yields (after trimming the SOAP baggage, and noting that _x002B_ would be an encoding of + and _x0040_ of @)

FSI_0002+clo@20
<SOAP-ENV:..>
<SOAP-ENV:Body>
<a1:FSI_0002_x002B_clo_x0040_20 id="ref-1" xmlns:a1=...>
</a1:FSI_0002_x002B_clo_x0040_20>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

FSI_0002+clo@20-1
<SOAP-ENV:...>
<SOAP-ENV:Body>
<a1:FSI_0002_x002B_clo_x0040_20-1 id="ref-1" xmlns:a1=...>
</a1:FSI_0002_x002B_clo_x0040_20-1>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

FSI_0002+add2@11
<SOAP-ENV:...>
<SOAP-ENV:Body>
<a1:FSI_0002_x002B_add2_x0040_11 id="ref-1" xmlns:a1=...>
</a1:FSI_0002_x002B_add2_x0040_11>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

FSI_0002+clo@20-2
<SOAP-ENV:..>
<SOAP-ENV:Body>
<a1:FSI_0002_x002B_clo_x0040_20-2 id="ref-1" xmlns:a1=...>
</a1:FSI_0002_x002B_clo_x0040_20-2>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

-------------------
FSI_0002+it@25
<SOAP-ENV:...>
<SOAP-ENV:Body>
<a1:FSI_0002_x002B_it_x0040_25 id="ref-1" xmlns:a1=...>
</a1:FSI_0002_x002B_it_x0040_25>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

FSI_0002+it@25-1
<SOAP-ENV:...>
<SOAP-ENV:Body>
<a1:FSI_0002_x002B_it_x0040_25-1 id="ref-1" xmlns:a1=...>
</a1:FSI_0002_x002B_it_x0040_25-1>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

FSI_0002+add2@11
<SOAP-ENV:...>
<SOAP-ENV:Body>
<a1:FSI_0002_x002B_add2_x0040_11 id="ref-1" xmlns:a1=..>
</a1:FSI_0002_x002B_add2_x0040_11>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

FSI_0002+it@25-2
<SOAP-ENV:...>
<SOAP-ENV:Body>
<a1:FSI_0002_x002B_it_x0040_25-2 id="ref-1" xmlns:a1=...>
</a1:FSI_0002_x002B_it_x0040_25-2>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>


val mhead : m:System.Object list -> System.Object list
val addN : n:int -> x:int -> int
val add2 : (int -> int)
val emitString : x:'a -> unit
val it : unit = ()

Where we observe that of the user input names, only add2 survives as part of the serialized name; and that the second instance of what looked like the same function is actually an instance of a different type.

Running the same code as a compiled .exe, we get

Program+clo@20
<SOAP-ENV:...>
<SOAP-ENV:Body>
<a1:Program_x002B_clo_x0040_20 id="ref-1" xmlns:a1=...>
</a1:Program_x002B_clo_x0040_20>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

Program+clo@20-1
<SOAP-ENV:...>
<SOAP-ENV:Body>
<a1:Program_x002B_clo_x0040_20-1 id="ref-1" xmlns:a1=...>
</a1:Program_x002B_clo_x0040_20-1>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

Program+add2@11
<SOAP-ENV:...>
<SOAP-ENV:Body>
<a1:Program_x002B_add2_x0040_11 id="ref-1" xmlns:a1=...>
<n>2</n>
</a1:Program_x002B_add2_x0040_11>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

Program+clo@20-2
<SOAP-ENV:...>
<SOAP-ENV:Body>
<a1:Program_x002B_clo_x0040_20-2 id="ref-1" xmlns:a1=...>
</a1:Program_x002B_clo_x0040_20-2>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

-------------------
Program+clo@25-4
<SOAP-ENV:...>
<SOAP-ENV:Body>
<a1:Program_x002B_clo_x0040_25-4 id="ref-1" xmlns:a1=...>
</a1:Program_x002B_clo_x0040_25-4>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

Program+clo@25-5
<SOAP-ENV:...>
<SOAP-ENV:Body>
<a1:Program_x002B_clo_x0040_25-5 id="ref-1" xmlns:a1=...>
</a1:Program_x002B_clo_x0040_25-5>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

Program+add2@11
<SOAP-ENV:...>
<SOAP-ENV:Body>
<a1:Program_x002B_add2_x0040_11 id="ref-1" xmlns:a1=...>
<n>2</n>
</a1:Program_x002B_add2_x0040_11>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

Program+clo@25-6
<SOAP-ENV:...>
<SOAP-ENV:Body>
<a1:Program_x002B_clo_x0040_25-6 id="ref-1" xmlns:a1=...>
</a1:Program_x002B_clo_x0040_25-6>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

where the names are similar -- and now the closure add2 actually contains that 2 as part of its body.

So, what's going on? Well, let's look at what we get when we decompile.

	internal static class $Program
	{
	[DebuggerBrowsable(DebuggerBrowsableState.Never)]
	internal static FSharpFunc<int, int> add2@11;
	[DebuggerBrowsable(DebuggerBrowsableState.Never), DebuggerNonUserCode, CompilerGenerated]
	internal static int init@;
	public static void main@()
	{
	int n = 2;
	$Program.add2@11 = new Program.add2@11(n);
	FSharpList<object> fSharpList = FSharpList<object>.Cons(new Program.clo@20(), FSharpList<object>.Cons(new Program.clo@20-1(), FSharpList<object>.Cons(Program.add2, FSharpList<object>.Cons(new Program.clo@20-2(), FSharpList<object>.Empty))));
	FSharpFunc<object, Unit> action = new Program.clo@21-3();
	FSharpList<object> fSharpList2 = fSharpList;
	SeqModule.Iterate<object>(action, (IEnumerable<object>)fSharpList2);
	ExtraTopLevelOperators.PrintFormatLine<Unit>(new PrintfFormat<Unit, TextWriter, Unit, Unit, Unit>("-------------------"));
	FSharpList<object> fSharpList3 = FSharpList<object>.Cons(new Program.clo@25-4(), FSharpList<object>.Cons(new Program.clo@25-5(), FSharpList<object>.Cons(Program.add2, FSharpList<object>.Cons(new Program.clo@25-6(), FSharpList<object>.Empty))));
	FSharpFunc<object, Unit> action2 = new Program.clo@26-7();
	FSharpList<object> fSharpList4 = fSharpList3;
	SeqModule.Iterate<object>(action2, (IEnumerable<object>)fSharpList4);
	}
	}
	[CompilationMapping(SourceConstructFlags.Module)]
	public static class Program
	{
	[Serializable]
	internal class add2@11 : FSharpFunc<int, int>
	{
	[DebuggerBrowsable(DebuggerBrowsableState.Never), DebuggerNonUserCode, CompilerGenerated]
	public int n;
	internal add2@11(int n)
	{
	this.n = n;
	}
	public override int Invoke(int x)
	{
	return Program.addN(this.n, x);
	}
	}
	[Serializable]
	internal class clo@20 : FSharpFunc<FSharpList<object>, FSharpList<object>>
	{
	internal clo@20()
	{
	}
	public override FSharpList<object> Invoke(FSharpList<object> m)
	{
	return Program.mhead(m);
	}
	}
	[Serializable]
	internal class clo@20-1 : OptimizedClosures.FSharpFunc<int, int, int>
	{
	internal clo@20-1()
	{
	}
	public override int Invoke(int n, int x)
	{
	return Program.addN(n, x);
	}
	}
	[Serializable]
	internal class clo@20-2 : FSharpFunc<object, Unit>
	{
	internal clo@20-2()
	{
	}
	public override Unit Invoke(object x)
	{
	Program.emitString<object>(x);
	return null;
	}
	}
	[Serializable]
	internal class clo@21-3 : FSharpFunc<object, Unit>
	{
	internal clo@21-3()
	{
	}
	public override Unit Invoke(object x)
	{
	Program.emitString<object>(x);
	return null;
	}
	}
	[Serializable]
	internal class clo@25-4 : FSharpFunc<FSharpList<object>, FSharpList<object>>
	{
	internal clo@25-4()
	{
	}
	public override FSharpList<object> Invoke(FSharpList<object> m)
	{
	return Program.mhead(m);
	}
	}
	[Serializable]
	internal class clo@25-5 : OptimizedClosures.FSharpFunc<int, int, int>
	{
	internal clo@25-5()
	{
	}
	public override int Invoke(int n, int x)
	{
	return Program.addN(n, x);
	}
	}
	[Serializable]
	internal class clo@25-6 : FSharpFunc<object, Unit>
	{
	internal clo@25-6()
	{
	}
	public override Unit Invoke(object x)
	{
	Program.emitString<object>(x);
	return null;
	}
	}
	[Serializable]
	internal class clo@26-7 : FSharpFunc<object, Unit>
	{
	internal clo@26-7()
	{
	}
	public override Unit Invoke(object x)
	{
	Program.emitString<object>(x);
	return null;
	}
	}
	[CompilationMapping(SourceConstructFlags.Value)]
	public static FSharpFunc<int, int> add2
	{
	get
	{
	return $Program.add2@11;
	}
	}
	public static FSharpList<object> mhead(FSharpList<object> m)
	{
	return FSharpList<object>.Cons(m.Head, FSharpList<object>.Empty);
	}
	[CompilationArgumentCounts(new int[] {1,1})]
	public static int addN(int n, int x)
	{
	return x + n;
	}
	public static void emitString<a>(a x)
	{
	SoapFormatter sf = new SoapFormatter();
	MemoryStream sink = new MemoryStream();
	sf.Serialize(sink, x);
	FSharpFunc<string, Unit> fSharpFunc = ExtraTopLevelOperators.PrintFormatLine<FSharpFunc<string, Unit>>(new PrintfFormat<FSharpFunc<string, Unit>, TextWriter, Unit, Unit, string>("%s"));
	a a = x;
	string fullName = a.GetType().FullName;
	fSharpFunc.Invoke(fullName);
	FSharpFunc<string, Unit> fSharpFunc2 = ExtraTopLevelOperators.PrintFormatLine<FSharpFunc<string, Unit>>(new PrintfFormat<FSharpFunc<string, Unit>, TextWriter, Unit, Unit, string>("%s"));
	string @string = Encoding.Default.GetString(sink.ToArray());
	fSharpFunc2.Invoke(@string);
	}
	}

view raw gistfile1.cs hosted with ❤ by GitHub

This is the debug version, the release version just moves the main program locals to be static members of the Program class, with no other relevant changes.

This shows us that our named functions are compiled as normal functions as we expect in C#, but the first-class function objects we pass around are actually indirections to those functions; and different ones at each site, even if they close over nothing, and so must be identical, even in the release build.

So, what does this mean?

Well, for one thing, type-needy serialization methods like DataContract aren't going to work well with functions (including lambdas inside of methods of innocent looking types), because the types are instance-unique. And deserialization is only going to be possible in the same binary as the serialization came from (or at least have the same functions on the same types, methods and line numbers), unless you do a fair bit of decompilation to provide a translation binding.

The latter is a potential pitfall -- if you implement a Windows Workflow in F#, and have objects persisted, things could work just fine within the first working version of your program; but processes that are hibernating mid-task while you're rolling out a new version can break when they wake because these secret types are no longer there by the same names.

Getting the types in an F# union type II -- the perils of relying on undocumented implementation details

Recalling this example --

	open System
	open Microsoft.FSharp.Reflection
	let GetUnionTypes (t:Type) =
	seq { yield t
	if FSharpType.IsUnion t
	then yield! FSharpType.GetUnionCases t
	// implementation detail leaks here -- nested type name
	|> Seq.map (fun x -> (string t) + "+" + x.Name)
	|> Seq.map Type.GetType }
	type expr = Num of int
	| Operation of (expr * expr)
	let baseType = typeof<expr>
	GetUnionTypes baseType
	|> Seq.iter (printfn "%A")
	;;

view raw gistfile1.fs hosted with ❤ by GitHub

It was adequate for the use case I had of it at the time; but then I tried a different union type, at which point a bug showed up

	type Card = King | Queen| Jack | Spot of int;;
	GetUnionTypes typeof<Card> |> Seq.iter (fun x -> printfn "%s" x.FullName);;

view raw gistfile1.fs hosted with ❤ by GitHub

yields

FSI_0003+Card
System.NullReferenceException: Object reference not set to an instance of an object.
   at FSI_0005.it@24-1.Invoke(Type x)
   at Microsoft.FSharp.Collections.SeqModule.Iterate[T](FSharpFunc`2 action, IEnumerable`1 source)
   at .$FSI_0005.main@()
Stopped due to error

And, of course, that is exactly what I deserved for relying on the undocumented internals.

A quick decompilation reveals this to be because union cases that carry no data are actually concrete base type instances, bearing the tag number as data, like --

internal static readonly Program.Card _unique_King = new Program.Card(0);

view raw gistfile1.cs hosted with ❤ by GitHub

rather than being redundant empty subtypes.

So we need to tweak our function to be

	let GetUnionTypes (t:Type) =
	seq { yield t
	if FSharpType.IsUnion t
	then yield! FSharpType.GetUnionCases t
	// implementation detail leaks here -- nested type name
	|> Seq.map (fun x -> Type.GetType ((string t) + "+" + x.Name))
	// except when the case has no data, so there's no such class
	|> Seq.filter (fun x -> x <> null) }

view raw gistfile1.fs hosted with ❤ by GitHub

and all is well. For the moment. Until the representation changes.

Getting the types in an F# union type

This arose in the context of performing data contract serialization of a union type, and needing to feed the concrete types in as known types. As the FSharp.Reflection facilities only let you at the names of the union cases, it's necessary to rely on the secret knowledge that the union cases are implemented as nested subtypes of the abstract union type

	open System
	open Microsoft.FSharp.Reflection
	let GetUnionTypes (t:Type) =
	seq { yield t
	if FSharpType.IsUnion t
	then yield! FSharpType.GetUnionCases t
	// implementation detail leaks here -- nested type name
	|> Seq.map (fun x -> (string t) + "+" + x.Name)
	|> Seq.map Type.GetType }
	type expr = Num of int
	| Operation of (expr * expr)
	let baseType = typeof<expr>
	GetUnionTypes baseType
	|> Seq.iter (printfn "%A")
	;;

view raw gistfile1.fs hosted with ❤ by GitHub

Then we can serialize objects of the union type like

	open System.IO
	open System.Runtime.Serialization.Json
	open System.Text
	let exprTypes = GetUnionTypes baseType
	let ToJson<'t> (myObj:'t) =
	use ms = new MemoryStream()
	DataContractJsonSerializer(typeof<'t>, exprTypes).WriteObject(ms, myObj)
	Encoding.Default.GetString(ms.ToArray())
	let FromJson<'t> (jsonString:string) : 't =
	use ms = new MemoryStream(Encoding.Default.GetBytes(jsonString))
	let obj = DataContractJsonSerializer(typeof<'t>, exprTypes).ReadObject(ms)
	obj :?> 't
	let value = Num 3
	let json = ToJson value
	let recovered : expr = FromJson json
	;;
	val exprTypes : seq<Type>
	val ToJson : myObj:'t -> string
	val FromJson : jsonString:string -> 't
	val value : expr = Num 3
	val json : string = "{"__type":"FSI_0002.expr.Num:#","item":3}"
	val recovered : expr = Num 3

view raw gistfile1.fs hosted with ❤ by GitHub

where we feed the list of types to be known to the constructor for the serializer.