URI collectors

Earlier today, I published XML Calabash version 3.1.0. It’s not intended to have backwards compatibilities, but bumping the middle digit and making the last digit zero is a hint that you might want to approach this release with a little bit of caution. Pay close attention, especially, to any pipelines that use the file steps. Preserving, exactly, I hope, the semantics of file: URIs in all the edge cases was a lot of work. I wrote a lot of tests, but I might have missed something. (And, of course, there may be bugs in the new features which have been more lightly tested.)

Most of the changes are in aid of supporting new URI schemes in an extensible way. I’ve added an extension for AWS-compatible S3⊕S3 support is built on top of Amazon’s SDK. You can login with credentials or using single sign on, or any of the other Amazon mechanisms. If the aws toolkit works, the S3 URI collector should work. storage and implemented support for basex:// URIs in the BaseX extension.

What this means is that you can do this:

<p:file-copy href="file:/path/to/book/"
             target="basex://Book/"/>

to copy all of the files from the book directory to the Book database in BaseX. Or this:

<p:file-move href="basex://Book/"
             target="s3://booksbucket/mybook/"/>

to move all of the files from the Book database into a folder in an S3 bucket.

The code that wrangles the URIs is called a “URI collector” and it’s responsible for managing the “directory and file” metaphor starting at a particular location. In addition to the file, S3, and BaseX collectors, that all work in the way you most likely expect, there’s also an “HTTP collector” that stretches the metaphor a bit. It lets you do things like this:

<p:file-copy href="https://spec.xproc.org/main/"
             target="basex://XProcSpecsDB/xproc/"
             cx:https-parameters="map{'style':'files'}"/>

to copy all of the current editorial drafts of the XProc specifications from the website into a BaseX database. (See the http: URI collector for more information and details about that cx:https-parameters attribute.)

Here’s a longer example that pulls a few of these pieces together into something that (while still a toy) illustrates the ideas in practice:

<p:declare-step version="3.1"
     xmlns:p="http://www.w3.org/ns/xproc"
     xmlns:c="http://www.w3.org/ns/xproc-step"
     xmlns:cx="http://xmlcalabash.com/ns/extensions"
     xmlns:xs="http://www.w3.org/2001/XMLSchema"
     expand-text="true"
     exclude-inline-prefixes="#all">
  <!-- ➊ -->
  <p:import
      href="https://xmlcalabash.com/ext/library/basex.xpl"/>
  <p:output port="result" sequence="true"
            serialization="map{'method':'xml','indent':true()}"/>

  <!-- ➋ -->
  <p:option name="database" select="'XProcSpecsDB'"/>
  <p:option name="text" as="xs:string" required="true"/>

  <!-- ➌ -->
  <p:directory-list name="list" message="List BaseX databases"
                    path="basex://#databases"/>

  <!-- ➍ -->
  <p:if name="checkdb" depends="list"
        test="empty(//c:directory[@name=$database])">
    <!-- ➎ -->
    <cx:basex name="create" p:message="Create {$database}">
      <p:with-input port="source"><p:empty/></p:with-input>
      <p:with-input port="query">
        <p:inline content-type="text/plain">
          db:create('{$database}', (), (),
             map{{'ftindex': true(), 'autooptimize': true(),
                 'textindex': true(), 'stemming': true()}})
        </p:inline>
      </p:with-input>
    </cx:basex>

    <!-- ➏ -->
    <p:file-copy name="copy" depends="create"
                 message="Copy spec.xproc.org to BaseX"
                 href="https://spec.xproc.org/main/"
                 target="basex://{$database}/specs/"
                 cx:https-parameters="map{'style':'files'}"/>
  </p:if>

  <!-- ➐ -->
  <cx:basex name="query" p:depends="checkdb"
            p:message="Query {$database}">
    <p:with-input port="source"><p:empty/></p:with-input>
    <p:with-input port="query">
      <p:inline content-type="text/plain">
        declare namespace h = "http://www.w3.org/1999/xhtml";
        let $matches :=
          for $match in ft:search('{$database}', '{$text}')/..
          let $uri := base-uri($match/root())
          where contains($uri, 'index.html')
          return
            $match/root()
        let $ids := distinct-values($matches/generate-id(.))
        let $specs :=
          for $id in $ids
          return ($matches[generate-id(.) = $id])[1]
        for $spec in $specs
        return
          &lt;spec href="basex:/{{base-uri($spec)}}">
            {{$spec/h:head/h:title/node()}}
          &lt;/spec>
      </p:inline>
    </p:with-input>
  </cx:basex>

  <!-- ➑ -->
  <p:wrap-sequence wrapper="specifications"/>

  <p:insert position="first-child">
    <p:with-input port="insertion">
      <p:inline>
        <title>Specifications that contain '{$text}'</title>
      </p:inline>
    </p:with-input>
  </p:insert>

</p:declare-step>

1. We’re going to use the cx:basex step, so I’m importing it. The XQuery steps could be done with the p:xquery step configured to use BaseX, but since the example explicitly relies on BaseX, this seems a bit more explicit.

2. We’re going to take two options, the name of a database and some search text.

3. We get a list of all the BaseX databases.

4. If the database we’ve asked for doesn’t exist, then we’ll create it. Making this part conditional avoids copying everything every single time. (In real life, you’d want some sort of way to update the database when things have changed.)

5. This bit of XQuery creates the specified database with the options necessary to enable full text indexing. (This is for BaseX 12.5, I gather it’s slightly simpler in BaseX 13.)

6. Then we copy all of the specs from https://spec.xproc.org/main/ into the BaseX database.

   This takes a while and there’s no feedback. Running the pipeline with --debug will give you some feedback as it works its way through the collector.

7. Next we do a full text query to find the matching documents. This is all a bit clumsy because my XQuery is rusty and I want it to fit in the weblog listing which is a bit narrow.

8. Finally, we put a wrapper around the results and add a title.

Here’s an example of the output:

$ xmlcalabash app/demo.xpl text="Schema"
List BaseX databases
Query XProcSpecsDB
=== result :: 1 :: null ================================================
<?xml version="1.0" encoding="UTF-8"?>
<specifications>
   <title>Specifications that contain 'Schema'</title>
   <spec href="basex://XProcSpecsDB/specs/xproc/index.html"/>
   <spec href="basex://XProcSpecsDB/specs/steps/index.html"/>
   <spec href="basex://XProcSpecsDB/specs/run/index.html"/>
   <spec href="basex://XProcSpecsDB/specs/validation/index.html"/>
</specifications>
========================================================================

URI collectors are integrated into the general URI resolver mechanism. You could, for example, feed that XML document into an XSLT step and use the doc() function with one of those basex:// URIs to load the relevant specification!

There are a few other bug fixes and upgrades in this release. In the course of working out how to write the BaseX URI collector, I found and fixed a number of issues in the BaseX extension.

Feedback solicited, especially from early adopters!