Category Archives: EXIF/Metadata

Setting SmugMug Print Size Keywords with Jupyter and Python

Leave a reply

For the last few weeks, my normal hobbies have been preempted by my obsessive refactoring of an old SmugMug keyword setting program. Many years ago I wrote a small application that helped me maintain meaningful SmugMug keywords. I used this application for years until inevitable software changes broke it. Replacing it with something better has been on my to-do list for ages.

I kept putting it off because I knew the tools I used in the past, mostly J and VB, were not well suited to REST API programming. Yes, I know just about any programming tool can be forced to serve, Turning completeness guarantees it, but I have grown weary of programming to prove tiresome points. If a tool makes something more difficult or tedious than it should be then change tools!

The downside of changing tools is that it usually results in learning yet another programming language. I always try to avoid this! I briefly considered languages with decent REST support that I already sort-of-know1, C#, JavaScript, VB.Net and decided that proper 64 bit versions were either too expensive (C#, VB.Net) or that I didn’t want to stare at ugly bracket infested syntax for weeks. I’m talking about you JavaScript. Even your fanboys admit that you’re an ugly duckling.

I eventually decided to go with Python. I’ve always found Python code more readable than many other programming languages. This is a widespread opinion. Python is free, open source, and comes with a large diverse set of tools. One of the best tools to come out of the Python world is Jupyter. Jupyter is the first public domain open source literate programming tool that has gained a large following.

I’ve been a fan of literate programming ever since I read Knuth’s seminal book. He used the technique to create some of the best program documentation ever created. I always wondered why literate programming never caught on. I suspected the basic problem was simply that many programmers are not particularly literate. Well, Jupyter is proving me wrong.

Jupyter is certainly helping me write and program with such clarity that all my ideas seem trivial.

Compare this notebook, (use the first link for the best layout), to my earlier blog post about setting SmugMug print size keywords.

  1. Nbviewer: Setting SmugMug Print Size and Geotag Keywords with Jupyter and Python
  2. GitHub: Setting SmugMug Print Size and Geotag Keywords with Jupyter and Python
  1. Be highly suspicious of people who claim to fully understand any programming language. Only delusional nitwits would make such a claim for any natural language. Does anybody, even luminaries like Shakespeare, truly understand English? We all sort-of-know our mother tongues and if we’re honest, we’re continually surprised by how others make use of it. Literature, it’s a thing. The same holds for programming languages. Every day I’m surprised, by unusual, stupid, silly, clever and freaking brilliant code fragments in programming languages that I have used for decades.↩

Downloading SmugMug Captions with Python and Jupyter

1 Reply

This blog post started out as an experimental rendering of a Jupyter notebook. I wanted to see how difficult it would be to convert a notebook to a WordPress.com blog post. Even though Jupyter exports notebooks in HTML and Markdown they do not display well “out of the box.” No doubt one could craft CSS that would help but the entire point of Jupyter exports is to cut down on pointless format thrashing.

This post is a teaser. If you want to get to the source notebook follow this link to my GitHub repositories.

If you only want to read the notebook use this nbviewer link.

Why am I doing this?

My photo captions have evolved into a form of milliblogging. Milliposts (milliblog posts) are terse and tiny; many are single sentences or paragraphs. Taken one-at-a-time milliposts seldom impress but when gathered in hundreds or thousands accidental epics emerge. So, to prevent “epic loss” I want a simple way of downloading and archiving my captions off-line.

If you don’t control it you cannot trust it!

When I started blogging I knew that you could not depend on blogging websites to archive and preserve your documents. We had already seen cases of websites mangling content, shutting down without warning, and even worse, censoring bloggers. It was a classic case of, “If you don’t control it you cannot trust it.” I resolved to keep complete off-line version controlled copies of my blog posts.

Maintaining off-line copies was made easier by WordPress.com‘s excellent blog export utility. A simple button push downloads a large XML file that has all your blog posts with embedded references to images and other inclusions. XML is not my preferred archive format. I am a huge fan of LaTeX and Markdown: two text formats that are directly supported in Jupyter Notebooks. I wrote a little system that parses the WordPress XML file and generates LaTeX and Markdown files. Yet, despite milliblogging long before blogging, I don’t have a similar system for downloading and archiving SmugMug metadata. This Jupyter notebook addresses this omission and shows how you can use Python and the SmugMug API to extract gallery and image metadata and store it in version controlled local directories as CSV files.

The Instagram Metadata Massacre

Instagram Fingers

Instagram Fingers

I’ve been playing around with the iPhone app Instagram.  I decided to give this freebie a try after a gaggle of soulless self promoting Apple marketing zombies declared it “imaging app of the year.”  Instagram, like Hipstamatic,  (which incidentally stopped working after the last update), crops iPhone images to squares, applies hokey filters and then offers you the opportunity to “share”  or print your masterpieces on related websites.  Apparently Instagram is already one of the words largest online photo libraries. Considering Instagram images are filtered iPhone shots it’s almost certainly the world’s second largest online collection of crappy throw-away shots.  Facebook is the largest if you’re wondering!

You probably sense that I am not drinking the Instagram cool-aid. Instagram is a fine fun freebie but it strips EXIF data from the images it drops in the iPhone camera roll. This is the second iPhone EXIF exterminator I have encountered.  I have this thing about EXIF data; it annoys me when software massacres it.

Here’s a little poem:

Exterminate EXIF – NOT

Let it be

Set your fricking EXIF free

Alas, with Instagram, it’s not to be!

More on that SmugShot Metadata Mess

A few days after posting a J script that fixes EXIF timestamps in SmugShot images SmugMug changed the filename assigned to downloaded SmugShot images.

Originally the layout was:

SmugMugID_smugshot_SmugShotID.jpg

This layout was inconsistent with the way SmugMug normally handles files. Default SmugMug behavior is to leave filenames unchanged.  This contrasts with other photo sites, (you know who you are), that rename your sacred files to suit their indexing schemes. A few days ago SmugMug corrected this discrepancy and downloaded SmugShot images reverted to their original format:

smugshot_SmugShotID.jpg

This fix broke smugshotexif: see: my previous post. I have modified SmugShotFix.ijs to handle the current format and the older one.  I will keep tweaking this hack until SmugShot’s sloppy EXIF handling is corrected.

SmugShot Metadata Mess

SmugShot iPhone App Logo

SmugShot iPhone App Logo

While languishing in a series of hospitals and rehab centers recovering from a bad fall I have been amusing myself by taking iPhone pictures and posting them to SmugMug with SmugShot.

SmugShot is a nifty little iPhone app that uploads iPhone pictures and videos directly to your SmugMug galleries. SmugShot is a freebie so I cannot, in good faith, whine about missing features but SmugShot has one serious deficiency: it strips EXIF timestamps. Native iPhone jpegs contain EXIF timestamps but after passing through SmugShot only GPS information survives. I am not sure if this is by design or oversight but I don’t like it!

The fix is straight forward:

  1. Download SmugMug metadata.
  2. Download all SmugShot pictures.
  3. Match pictures with metadata and use upload timestamps as date taken EXIF timestamps.
  4. Finally replace the uploaded SmugShot images with the EXIF repaired versions.

Simple but tedious. The following J word manipulates Phil Harvey’s superb command line utility exiftool to cleanup the SmugShot metadata mess. The words used by smugshotexif are in the J script SmugShotFix.ijs. You can also find this script in the J files sidebar of this blog. Look in the J Scripts directory. Let’s hope the next version of SmugShot fixes the date time issue and hacks like this will no longer be necessary.

smugshotexif=:4 : 0

NB.*smugshotexif v--  inserts missing EXIF  metadata  in SmugShot
NB. iPhone uploads.
NB.
NB. The  iPhone  SmugShot  app  removes  EXIF  information during
NB. upload. This verb restores items I care about.
NB.
NB. dyad:  btclSmugMetaData smugshotexif clPathFile
NB.
NB.   NB. update TAB delimited SmugMug metadata tables
NB.   SmugTablesFrXml2 'c:\pd\docs\smugmug\data\smugheavy.xml'
NB.
NB.   NB. load SmugShot specific metadata
NB.   SMUGSHOTMD=: readsmugshots 0
NB.
NB.   SMUGSHOTMD smugshotexif 'c:\pictures\2011\Missouri\wip\1204618219_smugshot_9811805.jpg'
NB.
NB.   NB. fix all jpg files in a directory
NB.   (<SMUGSHOTMD) smugshotexif&> 1 dir 'c:\pictures\2011\Missouri\wip\*.jpg'

NB. extract SmugMug id from SmugShot file name
spid=. <extractsmugid y

NB. all smugshot pids
pids=. x {"1~ (0{x) i. <'PID'

if. spid e. pids do.

  NB. metadata exists for image insert items
  pos=. pids i. spid

  exif=. setartistcopyright y

  NB. use upload date for the missing original datetime
  date=. ;pos { x {"1~ (0{x) i. <'UPLOADDATE'
  date=. '-:' charsub date
  exif=. date setdate y

  caption=. ;pos { x {"1~ (0{x) i. <'CAPTION'
  exif=. caption setdescription y

  lb=. pos { x {"1~ (0{x) i. ;:'LATITUDE LONGITUDE'
  if. *./ 0 < #&> lb do.
    lb=. _999&". &> lb
    exif=. lb setlatlng y
  end.

  NB. rename as iphone file with original smugshot number
  NB. format path chars for windows rename
  newname=. '.jpg'&beforelaststr y
  newname=. 'iphone ',('_'&afterlaststr) @:(' [s'&beforelaststr) newname
  shell '/\' charsub 'rename ' , ; ' ' ,&.> dblquote y;newname,'.jpg'

  newname ; exif
else.
  'No SmugShot metadata for';spid
end.
)

Google Earth Image Touring

In a scientific poll of one, (I sampled myself),  Google Earth was voted the greatest free program on Earth. The brilliant developers at Google have managed to turn the most unlikely subject, geography,  into a video game.  And what a game it is!  Google Earth doesn’t troll around in an adolescent make believe world.  Google Earth serves up the real thing.   This would be amazing enough but with Google Earth you can explore the Moon, Mars and the night sky as well.  If you have any geographic  or astronomical needs get Google Earth.

I got into Google Earth a few years ago when I started geotagging old pictures.  Locating old, pre-GPS pictures, is one of the better Google Earth games.  My mother snapped this picture of me in 1964.  I am standing in front of the Uintah mountains in north eastern Utah.

Me near Redwash Utah

1964 was long before GPS was even a dream.  To find exactly where I was standing I turned on Google Earth’s terrain feature and slowly followed the adjacent road until all the landscape features matched up.

For famous sites it’s not this hard.  Here I am at Stonehenge.  Stonehenge is a Google Earth nobrainer.

Me Stonehenge

Over the last few years I have geotagged hundreds of images.  It’s something I will continue to do until I drop dead. Online picture sites like SmugMug provide automatic mapping of geotagged images but Google Earth does a much better job if you do a little KML programming.  KML is an XML namespace used by Google Earth and Google Maps.  It can be used to tour geotagged images.  When KML files are compressed,  (any ZIP utility will do),  they are given a KMZ file extension.  KMZ files can be loaded by Google Earth.

The attached KMZ file geotagged.kmz was generated by a J script I put together to query my geotagged SmugMug pictures.  When loaded into Google Earth it spins around the world pausing to display all the pictures I have geotagged.  I realize few will be interested in my pictures but it’s not hard to extract the KML embedded in geotagged.kmz and adapt it to your pictures.  Go forth and hack my friends.