so

Automation

Volume 3, Issue 27; 21 Oct 2019

Making it quicker and easier to post photos.

Over time,If malignant capitalism in the twenty-first century has taught us anything, it has taught us that trusting a third party to curate your valuable data is fraught with peril. It’ll all be toast the first moment a psychopathic investor figures out how to score a third yacht by burning it all down. (Verizon is burning down Yahoo Groups as I write this, and with it a big chunk of internet history.) I’ve moved more and more of my data off of corporate properties and into my own web spaces (it helps that I have an insanely powerful platform to build them with). This has always been true of my weblog. I used Flickr for a long time, but abandoned it at the first inkling it might disappear under the waves. (I think it has a brighter future now.) About the only external site I post to regularly is Twitter, and I’ve figured out how to automate some of that. I try to hold onto the hope that services like micro.blog, Mastadon, and webmentions will someday allow us to build social connections without walled gardens.

But I’ve said all this before.

Having my own web spaces does have a practical disadvantage: they aren’t as frictionless as the big names. There’s no instagram app to post pictures to my photo site. There’s no ecosystem of tools smoothing out the rough edges. In principle, I could write apps, but I’m busy. I also think that responsive web applications would be more fun to build. But I’m busy!

So. Let’s look at a concrete example. I have a habit of photographing and publishing pictures of pints. (No, I don’t know why any more, it’s just a thing I do because it’s a thing I’ve done.)

TheEverything in this posting is about quick snaps with my mobile phone. For more serious work, I still run through a more traditional workflow on my laptop. workflow yesterday was:

  1. Order pint. Delay gratification of drinking pint long enough to take picture. Enjoy first sip of pint.
  2. Edit the picture. (I have a half dozen or more apps to do this, but my go to is Snapseed.)
  3. “Share” the picture to Google Drive. It was Dropbox, but then Dropbox stopped supporting the version of Linux that’s running on my EC2 instance. The only important thing here is getting the photo off of iOS and into something I can use outside the walled garden.
  4. Ssh into to my EC2 instance.
  5. Use rclone to copy the image off Google Drive onto local storage.
  6. Login to photos.nwalsh.com.
  7. Follow a few links, add metadata to the image (a title, at least, and often a few keywords).
  8. Publish the image.
  9. Copy the URI of the image.
  10. Navigate back to the staging site for this weblog.
  11. Create a new “micro” post.
  12. Type the post, paste in the image URI, and publish it. Micro.blog will publish it to Twitter for me automatically.

Laugh if you want, but it’s not that difficult and I sleep securely at night, safe in the knowledge that it’s all mine. Unfortunately, it takes maybe five minutes or so and requires a little bit of concentrated fiddling in the middle. That’s not a big deal, but if you’re at the pub with friends, it’s kind of rude. And if you’re traveling on a train or dealing with spotty connectivity, it might not work.

The weakest link here is all that fiddling in the middle. In particular the path from phone to website involves two different manual steps, and copying the image URI is another manual step.

My (extended) team at work is building a production service on AWS. That’s brought me into contact with more of the cloud infrastructure and it wasn’t a giant leap to see that an AWS lambda would fit the bill nicely. (Added bonus, I get to learn something new and clearly useful, always a big +1 in my book.)

There are (at least) two needles to thread here: I still need to edit the metadata for my images and I still need to get the images off the device.

After a bit of exploration,I am completely open to suggestions for different or better apps. I decided that Exif viewer would solve the first problem. Archivist solves the second problem. It will copy images off your phone into the cloud; in particular, into an S3 bucket. Serendipitously, the exif editor stores edited files in a particular folder and the archiver can be configured to automatically backup all of the files in a folder, so that part becomes automatic.

An AWS Lambda, if you’re not familiar with it, is a chunk of code that you can configure to run automaticallyThe serverless aspect of lambdas (which doesn’t literally mean what it says, of course) gets a lot of (well deserved) attention, but for this application, it’s the event based dispatch that’s interesting. I have an EC2 instance running all the time so serverless shmerverless. in response to certain events (an HTTP request, for example, or a new file being added to an S3 bucket). All of the back end service provisioning is handled for you: the event happens, your code runs, that’s it, you’re done.

I don’t think I have any particular words of wisdom to impart about lambdas. I’m still a total n00b. Experience at work made it pretty clear that the serverless framework was the way to go (there’s about 800 things going on; some help coordinating them seems like a good idea). I found a nice, straightforward posting about where to start. I found docker lambda which allows you to build and test locally in a container that’s very nearly identical to the environment your lambda will actually run in.

I reverse engineered my years-old perl script for uploading images and rewrote the core of it in Python. (Lambdas support Python natively.) I wasted several hours trying to work out how to recreate the XML output of exiftool (which my photo site expects) in Python before realizing that I could just install exiftool in my lambda and call it as a subprocess, but never mind.

The handler function for my lambda clocks in at about 300 lines of Python. Nothing terribly clever or interesting. It fires when an image appears in a particular bucket. It tweaks the filename, scales the image into four sizes, copies the original into an archival bucket, copies the scaled images into the S3 bucket that the photo site uses, extracts the metadata, and posts it to photos.nwalsh.com where it appears.

As an entirely tangential exercise, last night when I was too tired to try to figure lambdas out anymore, I realized that I could easily arrange for the “micro” post feature on my weblog to provide the most recently published photographs with a radio button to select one. So I did that too.

The workflow today is:

  1. Take photograph.
  2. Edit photograph.
  3. Add title and keyword metadata to photograph.
  4. Magic happens. Editing the metadata causes the image to be written to a particular folder on my phone. That causes it to be copied to S3. That causes my lambda to run. Photo is published.
  5. Create a new “micro” post.
  6. Type the post, pick the image I want to go with it, and publish it.

I am irrationally pleased by the improvements. That photograph was the first “production” output from the new workflow!

Please provide your name and email address. Your email address will not be displayed and I won’t spam you, I promise. Your name and a link to your web address, if you provide one, will be displayed.

Your name:

Your email:

Homepage:

Do you comprehend the words on this page? (Please demonstrate that you aren't a mindless, screen-scraping robot.)

What is six times two?  (e.g. six plus two is 8)

Enter your comment in the box below. You may style your comment with the CommonMark flavor of Markdown.

All comments are moderated. I don’t promise to preserve all of your formatting and I reserve the right to remove comments for any reason.