Flickr to WordPress Part 2: replacing old image references

Almost exactly a year ago, I shared my tool and process for moving a Flickr-hosted photo collection to a WordPress-powered photo website. That tool has been used by a few folks now and I'm so glad it was helpful.

The thing that I hadn't done yet, but now have and detail in this follow-up post, is to create an automated way to clean up references to Flickr-hosted photos that exist in an existing WordPress website or blog. Without this critical final step, one could have a lot of historical content that still references Flickr-hosted photos instead of the version being hosted on your shiny new WordPress-powered photo site.

So here's how I did it. As with my previous post, this information is geared toward a technical audience that is comfortable with the command line and possibly modifying PHP code to suit their own purposes.

Method

I thought about a couple of different ways I could handle this "find and replace" operation.

With 13 years of blog posts, many of which contain references to Flickr photos in some form, making the changes manually was not an option.

I thought about continuing to iterate on my existing PHP command line tool that takes the exported Flickr dataset and generates a WordPress import, which would mean creating some kind of lookup data file where a find-replace command would use the Flickr photo URL to find the new WordPress-hosted photo URL. But when I saw all of the different ways I had embedded Flickr photos in my blog post content:

  • on a line by itself for oembed rendering
  • as simple links to a Flickr photo page
  • as <a> plus <img> tag groups that displayed the images full size inline with a link
  • as <img> tag groups that displayed the images at various smaller sizes, aligned left or right

I realized that I would need to be able to lookup the proper image URL for each display scenario. And given that my WordPress-powered photo site generated different image sizes (and that some of these had changed since the original data migration), that was not going to be simple. No one-size-fits-all substitution would work.

The good news is that WordPress easily supports building a custom REST API endpoint that would support a dynamic lookup of the information I needed on the photo site, for use on any site where I was finding-replacing content. Once I realized I could decouple those operations, it was clear how to proceed.

Creating a "Find by Flickr URL" API Endpoint

The first step, then, was to create a REST API endpoint on my WordPress-powered photo site that would allow me to specify the original Flickr photo URL and find the related WordPress post that had been generated during the migration process.

If you look at the code of the original migration tool, you'll note that for each WordPress post it creates, it adds a post meta field _flickr_photopage where it stores the URL of the Flickr-hosted photo. That usually looks something like https://www.flickr.com/photos/myflickrusername/123456789/. We can use that post meta field to do a simple lookup of the equivalent WordPress post object.

Since I want to be able to retrieve an image URL at a specific size so that I'm not embedding full size, large image files in posts that only need, say, the 300 pixel wide version, I also needed to accept width and height parameters, and then do a lookup of the related attachment file URL in WordPress.

Here's the class that I created to do all of this. If you use it, you'll need to customize a few things, including the Flickr username.

With that class loaded into my WordPress photo site's theme or in a plugin, now I have access to this kind of API call:

https://my-wp-photo-website.com/wp-json/myphotos/v1/find-by-flickr-url/?flickr-url=https://www.flickr.com/photos/myflickrusername/123456789/

And the returned JSON response of that kind of API call would look something like this:

{
     result: "found",
     post_id: 2425,
     permalink: "https://photos.chrishardie.com/2014/08/updesk-setup/",
     thumbnail_id: 5424,
     thumbnail_url: "https://photos.chrishardie.com/wp-content/uploads/2014/08/15130177147_75d885dc2a_o.jpg",
     thumbnail_width: 2448,
     thumbnail_height: 3264
}

Nice! Now I have the permalink of the photo post that replaces the original Flickr photo page, the URL of the image media file I can use in an <img> tag, and some other meta info if I need it.

The Find-and-Replace WP CLI Plugin

Now, it's time to use that API endpoint in a big find-replace operation.

The clear choice was to make a WP CLI command that could be used to run this on the command line, where I could log and review warnings and errors and have better memory management.

In creating this command, I used this general approach:

  1. Get all the posts in the WordPress database that mentioned a Flickr URL with my username in it
  2. For each post found, look for specific kinds of Flickr link and image references that need to be replaced
  3. Extract the original Flickr photo page URL from those references
  4. Use the API endpoint created above to look up the corresponding information on the photo site
  5. Update the post content with the information retrieved from the API

It sounds fairly simple, but I ran into several challenges and opportunities for optimization:

  • Not only were there references to the Flickr URL structure above, there were variations to consider such as the internal version of my Flickr user account ID that was in wider use years ago, or the flic.kr shortened version of their domain.
  • The API lookups could generate a lot of activity on my photo site, so I added some caching since those responses should rarely be changing.
  • I found some photos that I had apparently set to "private" or "contacts only" on Flickr but had left referenced in my blog posts, so I had to manually address those.
  • My Flickr-to-WordPress migration tool didn't handle Flickr "sets" (although it preserved and stored the data needed to handle the), so I had to redirect those references.
  • I had to make sure not to replace Flickr references to other people's photos.
  • Flickr varied its use of http versus https in different embed code it generated over the years.

In the end, I had a working plugin that could do a dry-run operation to see what it was going to change, and then do a "for real" run to actually update the posts as stored in the database.

$ wp flickr-fixer fix-refs --dry_run=false
Getting all posts containing Flickr references...
Found 203 posts to process.
Success: 621 replacement(s) made across all posts

With the API lookup cache primed, on my site it only took a minute or two to run. YAY!

You can see the final plugin code here.

(If you use it, you'll need to find/replace my Flickr username and a few other hardcoded references accordingly.)

Lessons Learned

When I think about the time I put in to first creating the original Flickr-to-WordPress migration tool, and then the time put into this content cleanup tool, it turns out it was a non-trivial project. But it always felt like the right thing to do, since once I was moved fully into WordPress I would have absolute control over my photo collection without depending on the changing services or business model of Flickr or anyone else.

It also highlights a few important lessons for migrations and owning your data online:

  • Try to be consistent in the ways you reference third-party tools and services in your content or workflows. If you have a bunch of variations and inconsistencies in place, any future move to another tool is going to be that much more painful.
  • Hold on to as much metadata as you can. You never know when it's going to come in handy.
  • When tackling big migrations, break hard problems up into smaller, slightly easier problems.
  • Document your thinking and your work along the way. It's too easy to get stuck going in circles on longer projects if you forget where you've already been.
  • APIs are magical and user-facing services that don't have them should be avoided at all costs.

I think this concludes my 14.5 years of being a Flickr user. I've canceled my Pro subscription and plan to delete my account in the weeks ahead.

If you find any of these tools useful, or if you have a different approach, I'd love to hear about it.

Moving photos from Flickr to WordPress

If you're ready to move your own Flickr photo collection to WordPress and feel comfortable on the command line, you can go straight to the Flickr to WordPress tool I built and get started.

Update on Feb 18, 2020: You can now also learn about my tool for finding/replacing old Flickr image references in your WordPress post content.

Here's some backstory:

I used to love Flickr as a place to store photos, and as a community for sharing and discussing photography. But as its ownership changed hands and its future became at times uncertain, I grew reluctant to trust that it could continue to be a permanent home for my own photos. My discomfort increased as I have become more engaged with the need to have full ownership over the things I create online.

So, I set out to migrate my 3.6GB collection of 2,481 Flickr photos, along with their tags, comments and other metadata, into a new home while I still could.

Continue reading "Moving photos from Flickr to WordPress"

Links for the Week - February 24, 2009

Things feel kind of in limbo right now, don't they?  What are you in between?  While you think about that, here are some links you might enjoy:

Using Stock Photos to Show You Care

Creepy scary stock photoOne of the funniest parts of browsing the Internets is when I come across the funny stock photos of professional people in various professional settings, used by site owners to put a "human face" on their web presence in the most generic way possible. It began with using the headshot of the attentive and waiting customer service representative to show you that "operators are standing by now," and it's just gone crazy from there.

With the photo here, I don't even know what the hell is going on. It's like the creepy older guy is trying to arm wrestle with the maniacally screaming younger dude over who gets to use the laptop, while the two women totally ignore them and instead grin broadly at the hamster dancing on their screen. But I'm like "creepy older dude, BACK OFF!" Why does he need to lunge into younger dude's space like that, using his fingertips as a push-off to further invade? And why won't either of the women help younger dude? This is some messed up stock photography. What was the photographer yelling at them? "Pretend you went to the office holiday party and took Ecstasy!"
Continue reading "Using Stock Photos to Show You Care"

Flickr Photos Featured

I love Flickr, which shows off the world we live in through the eyes and camera lenses of everyday folks. In the last week I've had a few of my photos posted there (most of which are released under the Creative Commons license) appear in some interesting spots:

  • Schmap Nashville Travel Guide: a couple of photos I took at the Tennessee State Museuem are apparently now a part of the fourth edition of this online tourist guide. There may be a print version too, but I'm not sure where to find it.
  • All Around Ohio for November 11: a photo I took of Antioch Hall at Antioch College in Yellow Springs, Ohio was featured in an article about the college staying open.

Derrick Jensen's Thought to Exist in the Wild

5F2FD5EE1A8911DA.jpgI have a lot of memories of visiting zoos as a younger person. There was the time my Boy Scout troop had an overnight stay at the Cincinnati Zoo, where we had behind-the-scenes tours of the habitats and infrastructure that made up the place; I was amazed at the intricate facades created for zoo visitors. Another summer at the same zoo and I'm about to start drinking my red cream soda during a field trip lunch break, when a bird poops directly into it from a tree overhead. I remember feeling frustration and resentment that this creature had invaded my personal space so - now I laugh at the irony of that resentment, felt so strongly against one who was just answering the call of this artificial shrine to come observe animal life, poop and all. And most recently, standing with my nose and right hand pressed up against the glass at the Lincoln Park Zoo in Chicago, apologizing to the once-grand and beautiful Gorillas on display there for the noisy people, the cheesy layout of the captivity, the life stolen from them. "I'm so sorry," I mouthed. "Please forgive us."

I've had plenty of conversations about why most zoos maybe aren't such a good thing, about what they symbolize, what they mean about who we are as a people and a culture. But until I read Thought to Exist in the Wild: Awakening from the Nightmare of Zoos by Derrick Jensen and with photography by Karen Tweedy-Holmes, I hadn't really explored that symbolism and sense of concern in any depth. And to that end, the book is a thought-provoking and eye-opening treatment of the subject.
Continue reading "Derrick Jensen's Thought to Exist in the Wild"

Moving Photos to Flickr

I've been slacking off for a few years on getting my photos online for folks to see. Partly that's because I've been slacking off in actually taking photos (has the world become less beautiful and interesting as I get older?...hmm, probably not...lazy ass) but also because my ancient and clumsy methods for formatting and posting them became a significant mental barrier to even bothering. So, driven by a desire to share and a need to have better tools for doing so (and not to mention saving some space on the Summersault webserver where my site is hosted), I've engaged in what is hopefully a mutually beneficial and long-lasting relationship with Flickr, an online photo sharing service. You can already check out my photos there, but know that it's all in flux as I transition my existing online photos, add and remove some, and rearrange how they're sorted. Let me know how you like it, and I'll post something when I'm done.