Publishing photographs

Volume 2, Issue 5; 31 Jan 2018

A story of duct tape, bailing wire, chewing gum, string, rubber bands, superglue, twist ties, blu tack, and assorted other odds and ends.

I’m fussy about my creative output. If I go to the trouble of making something, I want to be assured that it will last as long, and as consistently, as I want it to. I poured a lot of creative energy into Flickr once upon a time. Then Yahoo bobbled it and there was a ripple of concern that the service might be discontinued.

This taught me a valuable lesson: it is not in Random Service Provider’s best interest to preserve my data. It’s in their interest to do…whatever it is they think they’re in business to do. Curating my content is about making money off of it, or off of me, it’s not about doing what’s right or principled with my data. And even if their intentions are as pure as the driven snow, just wait’ll they start having trouble making payroll. Or paying their AWS bill.

So I’ve rolled my own: my own photo site, my own weblog software, my own “remember everything” system. When I can’t roll my own, I absolutely insist that I can get all my data back out. I archive my tweets, I archive my very occasional Google+ posts, I had a complete archive (in a useful format, not a bunch of SQLite tables) of my Evernote data. I had a complete backup of my Flickr collection. For everything else, there’s Emacs.

I do it because I enjoy writing software. Writing all those things taught me as much or more about Perl, Python, RDF, SQL, validation, Java, XQuery, XSLT, MarkLogic, XProc, Scala, Ruby, REST, web services, XML, Markdown, Apache, AWS, HTML, CSS, a dozen dozen other tools and languages, and software engineering generally than anything else I did. If you can arrange to make terrible design blunders, awful choices of implementation strategy, and completely fail to understand important concepts in software engineering on your own projects, I recommend it.

It doesn’t scale, of course. Even if everyone could write their own everything, they don’t all want to. Which is completely reasonable. I’m very interested in what we can do to rebuild a functioning, more-decentralized, more democratic web. There’s definitely something social about the web and I appreciate that the things I post on my sites are much less likely to be seen by anyone else than they would have been if I’d posted them on Facebook, Instagram, Twitter, LinkedIn, or whatever the cool kids do today.

[ Get to the point —ed ]

Right. So. I write posts on my weblog. These get posted on my site, but they also get syndicated out to RSS feeds, to Twitter, and perhaps other places.

If I’m sitting at my laptop and I want to link to a photograph, I upload it to my photos site and stick the link in my post. Nothing difficult about that.

But suppose I’m not sitting at my laptop. I’ve put in a little bit of effort so that it’s practical for me to tweet (or even write full weblog posts) from a tablet or my phone. Suppose I want to include a photograph? Let’s say, for example, I have this back-lit, over-saturated photograph of a tulip that I’ve just taken. I can crop and tweak it on the phone, but what then?

Once upon a time, I had a way to do this with email. But it wasn’t terribly reliable and once you push send, how long do you wait before you decide that an error has occurred? And what do you do then, anyway? Besides which, running a mailer on AWS has its own special problems.

I asked on Twitter, as one does. I got back a bunch of interesting replies. I spent half an hour poking at the APIs behind Google Photos (as best as I could, and to the extent that I could find and understand them). Then I just decided Dropbox would be easiest. [“Easiest” —ed]

Are you ready?

  1. Take and edit the photo on the phone. Upload it to Dropbox.
  2. Login to a web site so that the steps that follow aren’t exposed to the public internet.
  3. Use a web service to run a shell script to collect the images from the Dropbox folder(s), extract metadata, create thumbnails, and stash them in a staging area.
  4. Go to a page that runs some XQuery to load them from the staging area into MarkLogic so that it can display a form that will let me review them, adding titles and tags to the ones I want to publish.
  5. Post the selections to another web service to run a Perl script to copy the selected photographs, adding EXIF metadata for the titles and tags.
  6. Finally, call the already existing “upload photo” Perl script that scales each image several ways, uploads them all to S3, extracts the metadata as XML, and uploads it to where it’s indexed and made available.



It’s a good thing I enjoy writing software.

Please provide your name and email address. Your email address will not be displayed and I won’t spam you, I promise. Your name and a link to your web address, if you provide one, will be displayed.

Your name:

Your email:


Do you comprehend the words on this page? (Please demonstrate that you aren't a mindless, screen-scraping robot.)

What is seven plus five?  (e.g. six plus two is 8)

Enter your comment in the box below. You may style your comment with the CommonMark flavor of Markdown.

All comments are moderated. I don’t promise to preserve all of your formatting and I reserve the right to remove comments for any reason.