X’s all the way down

I’m the author of a family of projects that do Invisible XML processing (there’s a core parser, an Invisible XML layer on top of that, an XSLT function, and a command line application). The command line application is called CoffeePot. CoffeePot takes an Invisible XML grammar and an input and returns an XML tree that represents the input parsed with that grammar (assume for simplicity that the parse is successful).

The grammar S=A. A="a"., for example, will parse the input a and return: <S><A>a</A></S>. Often, whitespace in the XML is insignificant and it’s more convenient if the output is pretty printed. There’s an option for that, creatively named --pretty-print. There’s also a pretty printing option you can put in a configuration file so that you don’t have to put it on the command line every time.

A user observed that there’s no way to turn off the pretty printing on the command line. That is, if you have pretty printing enabled in the configuration file, you’re stuck with it. Wouldn’t it be nice if you could turn it off on the command line?

And, yeah, well, sure, obviously. There are several conventions for this sort of thing. It’s quite possible that I could have made --no-pretty-print do the job. But that’s not what I wanted. For consistency with the way other options⊕Using a colon separator is a bit idiosyncratic. A space would be the more common choice. Honestly, I do it that way because Saxon has always done it that way and I spend enough of my time in Saxon or Saxon-adjacent spaces that it’s just easier to be consistent. work, what I wanted was for --pretty-print:false to work. There was no convenient way to achieve that with the third party library I was using to parse arguments. I could imagine a couple of ugly hacks, or I could try to find a different library, or I could parse them myself. And parsing the arguments myself would mean I could remove a third party dependency, which is always nice.

I mentioned this conundrum in passing to Bethan who thought for a moment and said “couldn’t you use iXML to parse the arguments?”

At this point, you have to imagine me, staring a bit slack-jawed into the middle distance, with one of those cartoon x-ray bubbles over my skull revealing a hamster running furiously on its little wheel in my head.

When the power of speech returned, I wiped a little drool off my lips and said, “well, yes, of course I could.” Now, I am not saying that you should add an Invisible XML processor to your application for this purpose, but my application absolutely, already has one, so…

That task filled two or three evenings and most of my weekend with absolutely joyous hacking.

The details aren’t terribly important, but there’s one interesting wrinkle worth describing. The heart of the grammar is:

unknown = "-", "-"?, nc+, (":", ~[" "]+)? .

-command = accept
         | axe
         | pretty-print
         | unknown
         | dashdash
         | input
         .

accept = -"--accept", value? .
...

Except for a lot more options. Bear in mind that you have to parse the whole command line. If you’re going to recognize that --accept and --axe and --pretty-print etc. are valid options, you also have to be able to recognize that --spoon is not (at the time of writing) a valid option. You can’t just fail the parse because you have to be able to reflect that error back to the user in some meaningful way.

That’s what unknown is for, and that’s a tripping hazard. Invisible XML allows ambiguous parses. If pretty-print recognizes --pretty-print and unknown recognizes anything preceded by two hyphens, then any parse that contains --pretty-print is ambiguous because both rules match it. And the more (valid) arguments a user provides, the more ambiguous it gets!

In practice, it would probably have been entirely sufficient to use a priority pragma to make sure that the unknown option was always the lowest priority choice. But I mused about this fact to Bethan and she said, “but a list of terminal strings is regular and the complement of a regular language is always regular, so why don’t you just use the complement?” (That’s the sort of thing she says.)

And, come on, I just had to. I wrote a little Python to generate the complement. and got something like this:

unknown = "-" | complement, (":", ~[" "]+)? .

-command = accept
         | axe
         | pretty-print
         | unknown
         | dashdash
         | input
         .
accept = -"--accept", value? .
...
-complement = "-", ~[" p-"], nc+
            | "-p", ~[" p"], nc*
            | "-pp", nc+
            | "--", ~[" ap"], nc*
            | "--a", ~[" cx"], nc*
            | "--ac", ~[" c"], nc*
            | "--acc", ~[" e"], nc*
            | "--acce", ~[" p"], nc*
            | "--accep", ~[" t"], nc*
            | "--accept", nc+
            | "--ax", ~[" e"], nc*
            | "--axe", nc+
            | "--p", ~[" r"], nc*
            | "--pr", ~[" e"], nc*
            | "--pre", ~[" t"], nc*
            | "--pret", ~[" t"], nc*
            | "--prett", ~[" y"], nc*
            | "--pretty", ~[" -"], nc*
            | "--pretty-", ~[" p"], nc*
            | "--pretty-p", ~[" r"], nc*
...

Except it goes on a lot longer. And it’s just glorious.⊕Performance-wise, it’s a bit of a pig, but I implemented saving compiled grammars to get around that. (Longer term, the solution is to improve the parser so that it uses faster techniques for non-ambiguous grammars. I’ll get there, but there are a lot of things on my todo list!) The story could end there and it would be fine. But there are a few other threads lying about that we can tug on.

First, I need to be able to generate the command-line summary when a user enters --help. I could just put that in a bunch of print statements or a text file, but then I’d have to remember to keep it in sync with the arguments as they evolve.

No, what I need is⊕Oh! You could design a little text format for this and parse it with iXML! Yes, yes, you could, but XML is going to be better here. Just hang on. a summary for each option. In addition to the option name and the summary, some options have both short and long forms. And before long, I had a little configuration file to generate the command line grammar that could also generate the help:

<config xmlns="https://nineml.org/ns/coffeepot/config">
   <arg name="accept"
        summary="Accept a result with at most the specified number of parses"
        dashdash="accept">
   </arg>
   <arg name="axe"
        summary="Specify an axe"
        dashdash="axe">
   </arg>
   <arg name="pretty-print"
        summary="Pretty-print (indent) the output"
        dashdash="pretty-print"
        dash="pp">
   </arg>
   ...
</config>

And then something else became obvious: I could add information about types and allowed values that I could use to automate checking the argument values. A quick rewrite of the complement function in XSLT and I could generate the iXML grammar from the XML using XSLT. Nice!

Another thread lying around is the fact that the summaries aren’t always useful on their own. For example, “Specify an axe”, is not entirely self explanatory. If you know what an axe is, maybe that’s fine. But if you don’t know, for example, that you can get a random parse (from a set of ambiguous results) by⊕Oh, note-to-self: there’s another post I want to write about random parses. selecting the random axe, that description probably doesn’t help.

With a bit more information in the configuration file, I could support things like --help:axe to provide more information about what the --axe option does!

And…wait a minute!…that’s very similar to the information that needs to be in the documentation. (I can see some of you nodding along already). At the end of the day, I created a set of reference pages, one for each option and each configuration property. They look like this:

<refentry xmlns="http://docbook.org/ns/docbook"
          xmlns:xi="http://www.w3.org/2001/XInclude"
          xmlns:xlink="http://www.w3.org/1999/xlink"
          xml:id="ref.axe">
   <refmeta>
      <refentrytitle>axe</refentrytitle>
      <refmiscinfo role="long">--axe</refmiscinfo>
      <refmiscinfo role="value">xpath</refmiscinfo>
      <refmiscinfo role="value">random</refmiscinfo>
      <refmiscinfo role="value">priority</refmiscinfo>
      <refmiscinfo role="value">sequential</refmiscinfo>
   </refmeta>
   <refnamediv>
      <refname>--axe</refname>
      <refpurpose>Specify an axe</refpurpose>
   </refnamediv>
   <refsynopsisdiv>
      <cmdsynopsis role="long">
         <arg>--axe<replaceable>:value</replaceable></arg>
      </cmdsynopsis>
   </refsynopsisdiv>
   <refsection>
      <title>Description</title>
<para>Choose an “axe” to use when selecting trees from the
forest. The default axe, <literal>sequential</literal>,
gives sequential results for ambiguous parses. The
<literal>random</literal> axe gives random results for
ambiguous parses. The <literal>priority</literal> axe uses
priority pragmas. The <literal>xpath</literal> axe uses
XPath expressions.
</para>
   </refsection>
</refentry>

Squeezing extra metadata into the refmiscinfo elements isn’t the prettiest thing ever, but it’ll do.

Each entry contains enough metadata to generate a config file entry:

   <arg name="axe"
        summary="Specify an axe"
        dashdash="axe"
        values="xpath random priority sequential">
Choose an "axe" to use when selecting trees from the forest.
The default axe, `sequential`, gives sequential results for
ambiguous parses. The `random` axe gives random results
for ambiguous parses. The `priority` axe uses priority
pragmas. The `xpath` axe uses XPath expressions.
</arg>

The configuration file is used to generate the grammar to parse the command line and it’s used at runtime to provide the short and long help summaries. And the same reference pages are used to construct the summaries of the arguments and properties in the documentation.

One source of truth: a parser, a help system, and the documentation.

Bonus: JUnit uses the configuration file to dynamically generate unit tests so that every option and configuration property is tested against all of the possible legal values and an illegal value to assure that the type checking code is correct.

How sweet it is.