Playing with (almost) the latest C++ features -- groundwork

Having had my interest in playing with native code reawoken by the new C++11 features, the first thing I went to look at was portability. One of the advantages of managed code -- JVM or CLR -- is that the VM handles portability for you and the code can be built pretty much anywhere; with native code we have to see what the compilers have in common.

I've been using VC++2010 on Windows as having many of the "big rocks" for the new standard, while being backward compatible onto OS versions before Win7 (unlike the VC++11 compiler and its runtime); and for *nix-like systems, I have cygwin and debian squeeze... Well the distro support for these is a bit behind-hand (gcc versions 4.5 and 4.4 respectively), whereas gcc 4.7 is now out with quite a broad coverage of the new standard. While 4.5 has a good chunk of the new stuff, 4.4 doesn't -- in particular, it doesn't have the new lambda syntax. So, it's build from source time to get an upgrade to a sensible version there, which means I might as well go to the latest and greatest on both platforms...

Building gcc 4.7 from source

debian

Fortunately there are some handy instructions out there which can be used as a baseline for debian. Following them, I found that I needed to tweak how I built GMP to fit PPL's requirements, by modifying the configure step to be:

CPPFLAGS=-fexceptions ../../sources/gmp-5.0.4/configure --prefix=$PROJECT_DIR/output/ --enable-cxx

before PPL would go through happily. The --enable-cxx is required for the PPL ./configure stage to run through, the CPPFLAGS=-fexceptions is optional, but it avoids a make-time warning about possible unwanted runtime behaviours if you don't.

ClooG also needed a CPPFLAGS=-I$PROJECT_DIR/output/include CFLAGS=-L$PROJECT_DIR/output/lib on the configure line to find GMP.

In the gcc build, as well as pointing at ../../sources/gcc-4.7.0/configure it's also worth taking the advice from the MacOS build instructions and only selecting languages of interest to you i.e. to play with new C/C++ there's no need for Java or Fortran, at a considerable saving in time.

Then it's a matter of just adding soft links from whichever g*-4.7 files to the unadorned versions, and

export LD_LIBRARY_PATH=~/gcc-4.7/output/lib/
export PATH=~/gcc-4.7/output/bin:$PATH

to your .bashrc or equivalent

cygwin

Cygwin is more fun -- I've not yet managed to get that to build all the way through with the loop optimization libraries. When you get to PPL, you find you also need to go back and re-configure GMP with --disable-static --enable-shared, as explained in the friendly manual, to build the shared library version. However then when building gcc, we get a mismatch with the earlier libraries in the configure stage, where it just stops:

checking for the correct version of gmp.h... yes
checking for the correct version of mpfr.h... yes
checking for the correct version of mpc.h... yes
checking for the correct version of the gmp/mpfr/mpc libraries... no

It is possible that if you start by building PPL and CLooG with shared library GMP in a first pass, then build the rest starting with reconfiguring and building a static GMP it will work, but life is too short. The MacOS build instructions didn't use the PPL/ClooG/graphite libraries either, so we can do this to configure gcc instead:

$ ../../sources/gcc-4.7.0/configure          \
>     --prefix=$PROJECT_DIR/output/    \
>     --with-gmp=$PROJECT_DIR/output/  \
>     --with-mpfr=$PROJECT_DIR/output/ \
>     --with-mpc=$PROJECT_DIR/output/  \
>     --program-suffix=-4.7            \
>     --without-ppl --without-cloog    \
>     --enable-languages=c,c++

which sits and cooks for quite some time to get you the new compiler build.

Linkbait: fixing cygwin "mkdir foo mkdir: cannot create directory `foo': Permission denied"

I got into a state where I had this error, which other people have seen, after having tried to trash the build output of one of the failed PPL/CLooG attempts from Windows Explorer, where anywhere under in my home directory and down it was rejecting mkdir with "mkdir foo mkdir: cannot create directory `foo': Permission denied". Having spotted a tangentially related mailing list message about this sort of problem happening on network shares and there being ACL related, I tried the following and it worked to clear things up:

1. Start a PowerShell console as Administrator window
2. Run Get-Acl a folder (like /tmp) which you can mkdir in in cygwin (this will be %cygwin_root%\tmp where %cygwin_root% is where you installed cygwin)
   $acl = Get-Acl C:\cygwin\tmp
3. In Windows Explorer set yourself Full Control on all the affected folders -- %cygwin_root%\home and %cygwin_root%\home\%USERNAME% at least
4. In the PowerShell, Set-Acl on each folder you've just frobbed with the saved ACL object
   Set-Acl C:\cygwin\home $acl
   Note that the Set-Acl call may take considerable time (tens of seconds) to execute when it gets to the really problematic node and has to roll permissions down.
   

SCons -- Death to makefiles

Since I last did native code seriously at home (c. year 2000), I had discovered the very handy MiniCppUnit tool as a nice light-weight unit testing framework, so of course I went and fetched a copy to be going on with. This time, curiosity prompted me to wonder "WTF is this SConstruct file anyway?" and now when I opened it, I immediately recognised that it was some form of Python script, and wondered what sort of Python based make system this might be.

It was simple enough to find where it came from -- http://www.scons.org/ -- and looking at the user guide, I felt that it is much more intuitive system than makefiles (admittedly there's not a high barrier there), and far less cluttered than declarative XML based systems like Ant or MSBuild; so I'll be using that for my *nix builds -- it works very nicely for doing things like running unit tests as part of the build e.g.

MiniCppUnit -- Building it under modern C++

Just like I had to patch it to build in C++/CLI, I needed to make some changes to MiniCppUnit to get it to build clean under VC++2010, the out-of-the-box gcc 4.5 on cygwin 1.7.x and gcc 4.7 debian squeeze with -Wall -std=gnu++0x or -Wall -std=c++11 on respectively. First MiniCppUnit.hxx:

92c92
< #if _MSC_VER < 1300
---
> #if defined(_MSC_VER) && _MSC_VER < 1300
93a94
> /* Without the "defined(_MSC_VER) &&" this code gets included when building on cygwin 1.7.x with gcc 4.5.3 at least */
204c205
<  static void assertTrue(char* strExpression, bool expression,
---
>     static void assertTrue(const char* strExpression, bool expression,
207c208
<  static void assertTrueMissatge(char* strExpression, bool expression, 
---
>     static void assertTrueMissatge(const char* strExpression, bool expression, 
304c305
<    catch ( TestFailedException& failure) //just for skiping current test case
---
>    catch ( TestFailedException& /*failure*/) //just for skiping current test case

and the corresponding signature change in the .cxx file:

108c108
< void Assert::assertTrue(char* strExpression, bool expression,
---
> void Assert::assertTrue(const char* strExpression, bool expression,
122c122
< void Assert::assertTrueMissatge(char* strExpression, bool expression, 
---
> void Assert::assertTrueMissatge(const char* strExpression, bool expression, 

Of course there may be other things lurking to be scared out when I ramp up my standard warning levels to beyond the misleadingly named -Wall (when you have -Wextra, formerly -W, provided to switch on a whole bunch more including spotting signed/unsigned comparisons, before getting onto the really specialized ones) and switch on -Werror to force everything to be really clean.

Deleting files, keeping a few -- in Python

And the same again, only as a script, just because:

If there wasn't that comparison function to write, it'd be terser...

Book — Hello World - Computer programming for kids and other beginners by Warren and Carter Sande

There are a lot of books out there about learning new languages or software related technologies. The problem is that almost all of them assume that you know something about the field already; or are at least hard-core enough that you can survive being thrown in at the deep end -- the equivalent of the kids of 20-odd years ago who started out by self-teaching assembler on a BBC micro or Z80 -- and can take being started out on ODBC related code (I'm looking at you, Dive into Python.)

Hello World! is one of the rare exceptions, a gentle, and humorous introduction to the idea of programming, using Python, a language well suited to filling the niche that BASIC dialects did a generation ago. Even to a crusty and cynical old-timer, like myself, it makes entertaining reading -- not only is the writing style light and engaging, but you can nod sagely and think "Been there, done that" at all the "In the good old days" asides.

That said, it's not a book aimed at teaching people in my position, so I showed it to some members of the target demographic.

My wife, who has programmed a mean Excel spreadsheet in her time, but never anything in the way of more procedural languages, has enjoyed reading it and trying the examples (even if composing Python code using dictation software is a truly painful exercise); which is tribute to its style. While the examples may be rather trivial, in the wider sense, the immediate sense of achievement they bring help motivate and reinforce the reader.

I also showed the book to a teenager who had wanted to learn programming over the school holidays, and to whom I'd earlier loaned a copy of the O'Reilly Learning Python. She liked Hello World! very much for its very approachable style, and tells me that (to me, unsurprisingly) it gave her a better understanding of the real basics than the more advanced text has done.

However, as she also perceptively pointed out to me, "while it is very good at giving a budding programmer the groundings you'd definitely have to move on to a different book to get the more complicated bits."

It is strictly an introduction or 'taster' sort of book, a teaching aid rather than a reference. If the coding bug bites, then you will want to move onto something else to get a broader and deeper understanding, beyond the introductory simplifications (such as Hello World!'s simplifying use of old-style objects)

Looking at it as an old-timer, it is clear that the book is intended to be a whistle-stop tour of the idea of coding, and getting the computer to respond to what you have done. There isn't a sense of structured incremental development in the way material is presented, more one of getting the user to the "Ooh! Shiny!" quickly. That makes sense in terms of seeing if the coding bug bites before possibly dampening enthusiasm with comparatively boring, if worthy, things like operations on strings (split, join, search), which are left until late on. Unfortunately, one of the topics deemed too boring to mention was testing -- even though Python comes with a unit test library in the "batteries included", the idea of "how do you know if what you've written works?" wasn't raised.

The book comes with the expected accompanying download bundle : Python 2.5, and all the source and tools mentioned in the book. My wife had a little trouble with getting some of the tools to work -- the Stani's Python Editor program as supplied didn't appear when invoked from the start menu, and, when started from a command line, complained at not being able to find wxPython; although when I installed the latest wxPython for Python 2.5, the install process detected a prior version being present -- so it would help to have someone around who is confident with managing the computer being used.

Overall: definitely a good book for the absolute beginner, and I'd rate it a very solid four stars out of five. The reason I mark it down is that while it's great for teaching as it's laid out, it's less good as a reference afterwards.

Links for 4-Dec

CryptoConfig -- run-time Crypto provider switching in .Net.

WinForms, Synchronization and lambdas.

.Net 3.5 SP1 update hotfixes -- now in GDR, due for Windows Update in Q1.

Permission vs role based authorization.

IE rounded corners -- via JavaScript and VML.

Python 3.0 released! -- what next, Duke Nukem Forever?

1000th Post — The correct solution to a pesky interview riddle

Rather than navel gaze for my 1000^th post, the answer one should really give to the tired old interview chestnut about the guys who take 1,2,5 and 10 minutes to cross a bridge, no more than two at a time, needing a torch to light their way.

The answer is -- “It's a simple shortest path traversal for a graph, where the nodes are the 30 possible states (5 entities, 4 people and a torch, with the two torch-by-itself states not allowed). So use the A* algorithm to traverse the graph. You can even deduce the links in the graph algorithmically.”

Here's the Python code:

which outputs one of the two symmetrically equivalent shortest paths

>pythonw -u "torch.py"
    2+1 -->
<-- 1
    10+5 -->
<-- 2
    2+1 -->
Time taken = 17 minutes
>Exit code: 0

Fortunately, it seems that MSFT have long given up this interview habit -- just that the rest of the world may not yet have caught up.

Doing it programmatically also allows you to investigate how stable the solution is vis-à-vis the naïve strategy of using the 1-minute guy as escort, which is only 1 minute slower than the optimum.

Later

A little refactoring to put the times into one place (the new crossing_time function) and deduce everything from there. The use of the bits (which bits) is still a little dispersed, being there in display as well. Also changed the shortest possible distance heuristic h_score to be strictly monotonic (never overestimating the distance to the goal). Also play the path in the correct order.

Links for 4-Aug

Custom configuration in three easy steps.

A neural net library.

Finalizers and the race condition from hell.

The Parser Monad -- in C# and in F#.

Recursion and memoization (F# and C#).

A layer of hurt.

Practical concurrency -- cyclic lock-free buffer.

Concurrency on a single thread.

Getting rid of the Server OS shutdown dialog.

Python dependency injection and mocking.

WCF implementation security Guide.

Asynchronous operations in C# using iterators -- neat idea to turn the flow of control inside out again.

Locking with computational expressions in F#.

FsUnit 0.5.0 -- what it sounds like.

Links for 24-Jul

linking everywhere -- PoC for HTML5 style href="<url>" attributes on everything.

Python for Executives. 

FsTest : A Unit Testing DSL for F#. 

Object Oriented JavaScript.  

A gentle introduction to (Iron)Ruby from C#.

Links for 7-May

Cross platform standalone Silverlight applications.

Error reporting in Silverlight for Ruby and Python, and why they are as they are in Silverlight 2.

Sneaking Ruby 1.9 into the Google App Engine -- _why is a cunning fellow.

And IronPython 2 is now at beta 2.

Links for 3-Apr

Terse form for PowerShell ErrorAction SilentlyContinue.

A Cmdlet syntax for Python. 

The Highly Extensible CSS Interface. 

Content overlay with CSS. 

C++0x will have lambdas and closures natively, not via template meta-programming as in Boost.

The Nice Programming Language -- an interesting language for the JVM.  I wonder how easy it would be to port to the CLR.

Links for 11-Feb

PowerTab -- PowerShell tab completion

C# extension methods -- why they are conceptually a good thing 

The Insurgency of Quality -- Interaction Design keynote speech

Ruby Cheat-sheet

Truly transparent text -- the mechanism is more generally applicable

Links for 14-Nov

Loading WCF configuration from different files on the client side.

Unexpected Security Exception in ASP.Net -- scope your impersonations carefully.

Android and Google's non-JVM Java.

Re-throwing exceptions in Python.

XP style widgets with wxRuby on Windows.

Links for 10-Sep

New releases -- IronPython2.0α4 and Jython2.2

WCF and WS-* is hard -- and this from an insider

Does Java need an overhaul? -- or are Other JVM based languages the way forwards?

Links for 4-Sep

Python 3.0α1 released

WCF and LINQ -- getting them to play together is a messy business

Understanding and extending semantics in HTML -- Part 1 Part 2 Part 3 and see also Classifying the semantics of HTML.

IronPython α3

It's out, and it fixes the bug at α2 that broke DLR Hosting.

No question now about which platform to use going forwards.

IronPython and PyFit -- revisited

In my earlier post, I omitted one crucial detail, which I hadn't actually connected with getting the PyFit system to work, and that is I had already added the existing Python 2.5 standard libraries to my IronPython 1.0.1 site.py file thus:--

This should not be needed if you use FePy instead; and may not be needed for 1.1 or 2.0

Links for 1-Jun

I HAS 1337 CODE, LOL -- LOLCODE, a scripting language infused with the Zeitgeist (YARLY, KTHXBYE)

AD FS SDK documentation due in July 

In Defence of Checked Exceptions (Not Really) 

Zen of the DLR and DLR Experiences -- Two presentations on DLR design philosophy and architecture

Simple ASP.NET 2.0 Tips and Tricks --  that you may not have heard about

Secure Python Interpreter -- Sandboxed to restrict socket and file access.

IronPython Cookbook Wiki -- Using .Net from Python/DLR

Decisions, decisions

When I'm doing stuff for my own interest and enjoyment, it's almost certainly client-side applications. And unless it's something that is very definitely targeted at a particular set of OS specific functionality, I like it to be cross-platform and easy for a naive end-user to run. This was a particular pull of Java — though it was soon shown to be very definitely “debug everywhere” when the two main browser JVMs (Microsoft and Netscape) interpreted the thumb-width of a scrollbar differently (inclusive or exclusive of the range — or “lol, box-model”). And the API, even for AWT, was far nicer than any of raw Win32/Win16, MFC or OWL, back in the day.

So then the progression went AWT to GWT (a light-weight widget set that had all the missing widgets from AWT) to Swing… and there it stayed. But that did mean programming in Java, which, over the years, I've come to find lives at the wrong level of abstraction : it doesn't afford the ruthless power that C++ does (templates, multiple inheritance), but without giving very much in return (generics are latecomers, and comparatively weaksauce; there's nothing like mix-ins; and everywhere a lot of the same scaffolding that C++ would require).

Growing dissatisfaction pushed me to look at native (as opposed to bytecode) toolkits, where I could get to use C++, eventually settling on FOX, over wxWidgets. The downside of FOX is that it was not built for localisation (too many strings baked in), and the event handling model seemed to fight the language compared with Java's X-style callback registration (wxWidgets lost on having what felt like a more opaque layout model). But than meant compiling on Windows and Linux, and having to tweak platform dependent system header files (and hoping that other *nix-like platforms would work).

The moving target that is C# provided a bit of a distraction — I'll learn it for work, but are all new languages going to be just warmed-over Java? It didn't matter that Mono was providing portable CLR GUI by stages, when the language (C#) had little appeal in and of itself. So it wasn't until stumbling into Ruby and Python in the last few months that I discovered some more shiny in terms of language.

But what to do for client-side applications? Python with wxWidgets? Ruby with FOX? and what about naive users? Jython 2.2 in stand-alone mode? Pity about JRuby's sprawling nature. But what about AllInOneRuby and its friends? I had just about come down on standalone Jython, when the whole silverlight business blew up.

So now I think I know my GUI toolkit. And while we only have IronPython out (and ported to Mono) at the moment, that both Python and Ruby should soon co-exist in the DLR means that it should be possible to do what works best for the application at hand. When an object is an object is an object, much fun should be available.

IronPython and PyFit

Problem — PyFit uses the parser module.

Workround —

• remove redundant import compiler from fit\TypeAdapter.py
• remove import compiler from fit\taBase.py
• remove seq_types, map_types, oper_types from fit\taBase.py
• remove _safeAssemble from fit\taBase.py
• remove _safeEval from fit\taBase.py and lose the ability to input non-scalar types (string being scalar); or replace its body with return eval(s) and lose the protection against wild input (sorta-OK if your FitNesse wiki is on an internal network).

Either approach breaks various of the PyFit unit tests, but does permit you to run IronPython 1.x directly, so long as you keep your cell types under control.

Special case - IronPython 2.0alpha1

This currently has problems with the type() function used in CellHandlers.isValid(). Unroll the in test as follows

which passes the unit tests.

I believe that the type() problem is known in IronPython 2.0α1, but for the record, a minimal test case, in case it persists into beta

yields

C:\Documents and Settings\Steve\My Documents\code\python>\IronPython-2.0A1\ipy.exe ipy2.py
Traceback (most recent call last):
  File ipy2.py, line 5, in Initialize
  File , line 0, in _stub_##14
  File ipy2.py, line 3, in inspect
  File , line 0, in ContainsValueWrapper##18
  File , line 0, in _stub_##19
SystemError: Object reference not set to an instance of an object.

More IronPython

Since my earlier post on the topic, I find that I've been picked up by the nifty aggregator site IronPython URL's.

At that point, I'd just started using the implementation as part of the build process for the project I'm working on at the moment — while the bulk of the choreography is handled by running devenv against the main solution, the interstitial work is done through the pre- and post-build events. In most cases, .BAT-file behaviour is enough to just invoke an executable or two with a simple command line, but anything with string handling or iteration now gets pushed into IronPython.

Why IronPython in the build? Well, given that we're building with DevStudio 2005, .Net 2.0 is already installed, so IronPython's xcopy-style install means that we can put into a project's tools folder without having to perform any change-of-enviroment on the build machines (a big no-no).

What sort of tasks are we doing?

Quite a variety —

• Doing everything that needs version stamping from creating AssemblyInfo.cs files from templates, through web.config references to the generated assemblies to the installers and merge modules.
• Doing pretty much everything in managing installer creation from driving Wix to create the initial single language installers, version stamping them, and then extracting the language transforms and embedding them into the master installer

From that base, I've now moved into using IronPython as a tool for helping our system test team put scripts together. Unfortunately even 2.0Alpha1 doesn't have an implementation of parser, so it can't be directly used with FitNesse and PyFit; but even so, it still makes a great utility for CPython scripts to invoke to get at awkward or tedious bits of the Win32 APIs like

• ACLs on files and registry keys
• the current thread identity
• Assembly full-names

or even less esoteric things like DLL versions (even though pulling those out of PE format directly isn't that difficult) -- and without needing to check that all the necessary CPython Win32 add-on libraries are present.

It has also proved useful for quickly putting together scripts to drive web applications, by giving simple script-level access to all the System.Net and System.Xml facilities.

 

tl;dr — IronPython for the win for its xcopy-style install and out-of-the-box access to Windows APIs.

Undocumented Jython — Java callable Jython class idiom

You have to do

and not

in order to make jythonc generate the function calls you want in the intermediate Java code.

And as I'm in a “Jython calls Java calls Jython” state, doing some transcoding/refactoring, I have still not managed to get Jython at the front to call nicely into the Jython at the back.

causes

method myPackage.Backplane.performSetup of myPackage.Backplane instance 1
Traceback (innermost last):
  ...
AttributeError: abstract method "performSetup" not implemented

despite it being there quite plain as day:-

and despite Java code calling it quite happily.

*sigh*