Side quests

In my day job, we run regression tests before each release. Regression tests generate a bunch of XML and the easiest way to make that data available to everyone is to toss it in an S3 bucket. We have some Python scripts that run against that data to build nice reports.

A colleague asked, “why is that in Python and not XProc?” A fair question. Couple of reasons: one, the regression reports predate my XProc implementation being capable enough to do the job and two, because it was a chance to play with Saxon running natively in Python which is really cool.

But that got me thinking. Today, it’d be perfectly reasonable to use XML Calabash to write tooling for that data. There’s only the minor inconvenience that most of it is in S3⊕To be fair, the Python scripts run off data on the local filesystem. Copying data off S3 to run the scripts isn’t a difficult problem to solve. But… and there’s no way to get to that data from XML Calabash.

There’s a Java API for S3, so on a Sunday afternoon a couple of weeks ago, I had an “S3” step that could read and write⊕Side quest on a side quest: get a mock version of S3 stood up so that I didn’t accidentally destroy something. data from S3. Cool. I’ve no particular interest in trying to make an XProc version of the entire AWS suite of APIs; reading and writing S3 is all I need at the moment.

Cool, but also kind of stupid. Having cx:s3-directory-list, cx:s3-file-copy, etc. would work, but it ain’t elegant. This would be better, right?

<p:directory-list path="s3://bucket-name/path/"/>

There’s a messy way to do it, where you just add conditional code to all of the file steps, but That Would Be Wrong™. For one thing, using the AWS tooling adds another 40+ jars to your application. Support for S3 really needs to be optional.

If we reimagine the file steps as operating on a collection of URIs, instead of actual files, then it’s possible⊕Possible, but not exactly “easy.” to separate the functionality of the steps from the “fileness” of the inputs. Then you can imagine a “file URI collector” and an “S3 URI collector”.

Except now we’ve done that magical thing where we’ve changed our perspective on a problem. Suddenly, we’re open to completely new possibilities. Like:

<p:directory-list path="https://xmlcalabash.com/"/>

which (at the moment, mostly) returns

<?xml version="1.0" encoding="UTF-8"?>
<c:directory xmlns:c="http://www.w3.org/ns/xproc-step"
             xmlns:cx="http://xmlcalabash.com/ns/extensions"
             xml:base="https://www.xmlcalabash.com/"
             name="www.xmlcalabash.com">
   <c:file xml:base="img/icon.png" name="icon.png"/>
   <c:file xml:base="css/fonts.css" name="fonts.css"/>
   <c:file xml:base="css/style.css" name="style.css"/>
   <c:file xml:base="img/homelogo.jpg" name="homelogo.jpg"/>
   <c:directory xml:base="test-report/current/" name="current"/>
   <c:directory xml:base="test-suites/" name="test-suites"/>
   <c:directory xml:base="archive-1.x/" name="archive-1.x"/>
</c:directory>

That’s a URI collector that operates on HTTP URIs! (It’s a bit of a stretch, semantically. The way it works is that it follows links⊕“Links” in HTML documents being HTML elements that link to external resources (or, at least, the obvious ones, I’ve quite possibly missed a few), and in XML documents, elements that use XLink simple links. that are “below” the starting point, up to the specified depth. It only returns any given resource once, so it’s not a reflection of the web structure in the general case. And it’s not a general web mirroring tool because it won’t follow links outside the tree it’s working in.)

Lots of services integrate with the web by inventing their own URI scheme. So, at this point, it became clear that this ought to be a pluggable “service provider interface” (or “SPI”), in Java parlance, so that additional URI collectors can be added by extensions without requiring changes in XML Calabash.

To test that, I implemented a basex://⊕Shout out to Christian Grün for answering a bunch of stupid questions about the BaseX API and making polite suggestions about my very rusty XQuery coding skills. collector as an SPI in the XML Calabash BaseX extension. What this all means is that you can copy things from the web or on the filesystem. If you have the S3 extension, you can copy them to and from S3 with the standard file steps. If you have the BaseX extension, you can copy them to and from BaseX. (Obviously, a URI collector for eXist-db also makes sense, but I haven’t got around to that one yet.)

It feels slightly weird to run

<p:file-copy href="basex://XProcUnitTestDB/"
             target="s3://xmlcalabash-mock2/frombasex/"/>

but it works!

The last obvious (to me, so far) thing to do was wire the URI collectors into the pipeline document resolver. That allows you to do this:

<p:xslt>
  <p:with-input port="source"
     href="s3://xmlcalabash-mock1/doc.xml"/>
  ...
</p:xslt>

You should also be able to do this in a stylesheet:

<xsl:sequence
   select="doc('s3://xmlcalabash-mock1/doc.xml')"/>

But that doesn’t quite work yet. So there’s that to do still. And probably about 100 tests to write.

Side quests end eventually, right? Right!?