My worst typo

I run MarkLogic in docker containers with docker-compose. For example:

  nightly-1:
    image: f23-runner:latest
    container_name: nightly-1
    hostname: nightly-1
    networks:
      app_net:
        ipv4_address: 10.10.1.11
    volumes:
      - /space/docker/data/share:/share
      - /space/docker/data/nightly-1:/var/opt/MarkLogic
    stdin_open: true
    tty: true

Using an external volume for /var/opt/MarkLogic means both that MarkLogic configurations survive container restarts and that I can look at logs and configurations and such from outside the container. Very handy.

In my build environment, MarkLogic runs as “my” process so I can clear out the configuration data when I want to with a simple rm. But when I run a cluster of “nightly” containers, the MarkLogic process runs as root, so I can’t simply delete the data.

I’m working on a particularly convuluted bit of configuration at the moment, so I wanted to setup a couple of simple “save” and “restore” scripts. Easy peasy: stop the containers, archive or restore the data/nightly-* directories as appropriate, restart the containers.

Let’s look a little more closely at the restore script. Given that $NAME contains the name of our saved configuration and $LIST contains the list of container names to restore, we run:

echo "Restoring archive: SAVE.$NAME.tar.gz"
for container in $LIST; do
    sudo rm -rf $f/*
done

sudo tar zxf SAVE.$NAME.tar.gz

I pounded on ^C as soon as I saw messages about not being able to delete files in /dev/…. And I had done a full backup on Tuesday. And I wanted to die.

The actual lossage was quite small, a folder of docs in my home directory (none of which are likely to have changed since Tuesday) and, of course, most of the OS. Remarkably, the MarkLogic sources where I had spent many hours (since Tuesday!) tuning aforesaid convuluted configuration survived intact. Entirely by chance.

I really should investigate some sort of Time Machine-esque backup tool.

I have two sets of hard drives for my laptop; so part of the recovery was straightforward. Pull out the now borked drive, put in the other one, install a new OS on that, mount the old drive, and get on with recovery.

A couple of parts were quite a bit tricker than I expected.

• My bootable USB key didn’t. Getting a bootable USB key for a ThinkPad from a Mac took longer than it should have, isohybrid what?

• If you’re trying to mount an encrypted drive that used to be a boot drive, it’ll be in the same LVM volume group as your new encrypted boot drive. That means you can’t mount it. Booting off the USB, attaching the old drive, and renaming the volume group that it’s in worked for me.

• I had to rebuild my node/npm/[expletive deleted -ed] environment. (nave seems to be working at the moment).

• It’s going to be ages before I’ve got everything reinstalled and I’ve recovered all the configuration files. At least in Unix, they’re usually just dot files in my home directory.

It took all day but I believe I’m back in operation.

A moment on the fingertips, hours and hours [and hours -ed] in the recovery.