Flickr to WordPress Part 2: replacing old image references

Almost exactly a year ago, I shared my tool and process for moving a Flickr-hosted photo collection to a WordPress-powered photo website. That tool has been used by a few folks now and I'm so glad it was helpful.

The thing that I hadn't done yet, but now have and detail in this follow-up post, is to create an automated way to clean up references to Flickr-hosted photos that exist in an existing WordPress website or blog. Without this critical final step, one could have a lot of historical content that still references Flickr-hosted photos instead of the version being hosted on your shiny new WordPress-powered photo site.

So here's how I did it. As with my previous post, this information is geared toward a technical audience that is comfortable with the command line and possibly modifying PHP code to suit their own purposes.

Method

I thought about a couple of different ways I could handle this "find and replace" operation.

With 13 years of blog posts, many of which contain references to Flickr photos in some form, making the changes manually was not an option.

I thought about continuing to iterate on my existing PHP command line tool that takes the exported Flickr dataset and generates a WordPress import, which would mean creating some kind of lookup data file where a find-replace command would use the Flickr photo URL to find the new WordPress-hosted photo URL. But when I saw all of the different ways I had embedded Flickr photos in my blog post content:

  • on a line by itself for oembed rendering
  • as simple links to a Flickr photo page
  • as <a> plus <img> tag groups that displayed the images full size inline with a link
  • as <img> tag groups that displayed the images at various smaller sizes, aligned left or right

I realized that I would need to be able to lookup the proper image URL for each display scenario. And given that my WordPress-powered photo site generated different image sizes (and that some of these had changed since the original data migration), that was not going to be simple. No one-size-fits-all substitution would work.

The good news is that WordPress easily supports building a custom REST API endpoint that would support a dynamic lookup of the information I needed on the photo site, for use on any site where I was finding-replacing content. Once I realized I could decouple those operations, it was clear how to proceed.

Creating a "Find by Flickr URL" API Endpoint

The first step, then, was to create a REST API endpoint on my WordPress-powered photo site that would allow me to specify the original Flickr photo URL and find the related WordPress post that had been generated during the migration process.

If you look at the code of the original migration tool, you'll note that for each WordPress post it creates, it adds a post meta field _flickr_photopage where it stores the URL of the Flickr-hosted photo. That usually looks something like https://www.flickr.com/photos/myflickrusername/123456789/. We can use that post meta field to do a simple lookup of the equivalent WordPress post object.

Since I want to be able to retrieve an image URL at a specific size so that I'm not embedding full size, large image files in posts that only need, say, the 300 pixel wide version, I also needed to accept width and height parameters, and then do a lookup of the related attachment file URL in WordPress.

Here's the class that I created to do all of this. If you use it, you'll need to customize a few things, including the Flickr username.

With that class loaded into my WordPress photo site's theme or in a plugin, now I have access to this kind of API call:

https://my-wp-photo-website.com/wp-json/myphotos/v1/find-by-flickr-url/?flickr-url=https://www.flickr.com/photos/myflickrusername/123456789/

And the returned JSON response of that kind of API call would look something like this:

{
     result: "found",
     post_id: 2425,
     permalink: "https://photos.chrishardie.com/2014/08/updesk-setup/",
     thumbnail_id: 5424,
     thumbnail_url: "https://photos.chrishardie.com/wp-content/uploads/2014/08/15130177147_75d885dc2a_o.jpg",
     thumbnail_width: 2448,
     thumbnail_height: 3264
}

Nice! Now I have the permalink of the photo post that replaces the original Flickr photo page, the URL of the image media file I can use in an <img> tag, and some other meta info if I need it.

The Find-and-Replace WP CLI Plugin

Now, it's time to use that API endpoint in a big find-replace operation.

The clear choice was to make a WP CLI command that could be used to run this on the command line, where I could log and review warnings and errors and have better memory management.

In creating this command, I used this general approach:

  1. Get all the posts in the WordPress database that mentioned a Flickr URL with my username in it
  2. For each post found, look for specific kinds of Flickr link and image references that need to be replaced
  3. Extract the original Flickr photo page URL from those references
  4. Use the API endpoint created above to look up the corresponding information on the photo site
  5. Update the post content with the information retrieved from the API

It sounds fairly simple, but I ran into several challenges and opportunities for optimization:

  • Not only were there references to the Flickr URL structure above, there were variations to consider such as the internal version of my Flickr user account ID that was in wider use years ago, or the flic.kr shortened version of their domain.
  • The API lookups could generate a lot of activity on my photo site, so I added some caching since those responses should rarely be changing.
  • I found some photos that I had apparently set to "private" or "contacts only" on Flickr but had left referenced in my blog posts, so I had to manually address those.
  • My Flickr-to-WordPress migration tool didn't handle Flickr "sets" (although it preserved and stored the data needed to handle the), so I had to redirect those references.
  • I had to make sure not to replace Flickr references to other people's photos.
  • Flickr varied its use of http versus https in different embed code it generated over the years.

In the end, I had a working plugin that could do a dry-run operation to see what it was going to change, and then do a "for real" run to actually update the posts as stored in the database.

$ wp flickr-fixer fix-refs --dry_run=false
Getting all posts containing Flickr references...
Found 203 posts to process.
Success: 621 replacement(s) made across all posts

With the API lookup cache primed, on my site it only took a minute or two to run. YAY!

You can see the final plugin code here.

(If you use it, you'll need to find/replace my Flickr username and a few other hardcoded references accordingly.)

Lessons Learned

When I think about the time I put in to first creating the original Flickr-to-WordPress migration tool, and then the time put into this content cleanup tool, it turns out it was a non-trivial project. But it always felt like the right thing to do, since once I was moved fully into WordPress I would have absolute control over my photo collection without depending on the changing services or business model of Flickr or anyone else.

It also highlights a few important lessons for migrations and owning your data online:

  • Try to be consistent in the ways you reference third-party tools and services in your content or workflows. If you have a bunch of variations and inconsistencies in place, any future move to another tool is going to be that much more painful.
  • Hold on to as much metadata as you can. You never know when it's going to come in handy.
  • When tackling big migrations, break hard problems up into smaller, slightly easier problems.
  • Document your thinking and your work along the way. It's too easy to get stuck going in circles on longer projects if you forget where you've already been.
  • APIs are magical and user-facing services that don't have them should be avoided at all costs.

I think this concludes my 14.5 years of being a Flickr user. I've canceled my Pro subscription and plan to delete my account in the weeks ahead.

If you find any of these tools useful, or if you have a different approach, I'd love to hear about it.

Moving photos from Flickr to WordPress

If you're ready to move your own Flickr photo collection to WordPress and feel comfortable on the command line, you can go straight to the Flickr to WordPress tool I built and get started.

Update on Feb 18, 2020: You can now also learn about my tool for finding/replacing old Flickr image references in your WordPress post content.

Here's some backstory:

I used to love Flickr as a place to store photos, and as a community for sharing and discussing photography. But as its ownership changed hands and its future became at times uncertain, I grew reluctant to trust that it could continue to be a permanent home for my own photos. My discomfort increased as I have become more engaged with the need to have full ownership over the things I create online.

So, I set out to migrate my 3.6GB collection of 2,481 Flickr photos, along with their tags, comments and other metadata, into a new home while I still could.

Continue reading "Moving photos from Flickr to WordPress"

Exploring Colorado

Kelly and I were fortunate to be able to spend a week exploring Colorado this month, mostly around Boulder, Vail, Glenwood Springs and Steamboat. It was a great chance to visit some family in the area, see some different landscapes, hike/bike/raft/etc., and just generally enjoy life at 8,000 feet above sea level. Apparently there's a lot more of the state than one can see in a few days, so I expect we'll be back again soon. Some photos and notes from our trip follow.

Chris and Kelly

Vail is like Disneyworld (but cuter) for people who want to be able to pick up an expensive handbag or some sushi right after they get off the slopes, with everything laid out just so for the optimal outdoorsy tourist experience:

Continue reading "Exploring Colorado"

Notes from Moogfest

I took a quick trip to Asheville, North Carolina this past weekend to visit some friends and wander around the area. It's one of my favorite parts of the country, having spent a fair amount of time there as a kid, with my grandparents when they lived in Swannanoa and attending a summer camp for several years in Black Mountain.

But in looking up things to do while I was there this time, I found a whole new great reason to visit, the Moogfest music/art/tech festival.

Moogfest 2014 Wristband

Continue reading "Notes from Moogfest"

A trip to Ecuador and the Galapagos Islands

Cute pairIn May, Kelly and I took an amazing two and a half week trip to Ecuador and the Galapagos Islands in South America.  We spent a little time in the capital city of Quito, but otherwise we were off enjoying the jungle lodge in the cloud forest of Mindo, exploring the Galapagos on a small boat that was our home for seven nights, enjoying whitewater rafting, volcano-heated hot baths and great food in the mountain town of Banos, and checking out the sprawling and lively markets of Otavalo.

The photos and videos I've posted on Flickr capture some of the experience, and while the trip held too much adventure to describe here in great detail, I'll hit some of the highlights below.  (You can also go back and read individual posts written during the trip.)

Our trip was a nice combination of planned itinerary (primarily, the week-long stay on the boat M/Y Eric to tour around the Galapagos) and "wander around once we get there" mode.  The Lonely Planet Guide to Ecuador and the Galapagos Islands provide indispensable for the whole experience, from helping with food to lodging to cultural experiences and everything in between.  We were also visiting in advance of the heavier tourist season, so we were able to get into most any experience without advance reservation.

Continue reading "A trip to Ecuador and the Galapagos Islands"

Links for the Week - February 24, 2009

Things feel kind of in limbo right now, don't they?  What are you in between?  While you think about that, here are some links you might enjoy:

4 reasons to start using Gravatars right now

I've said before that to truly participate in public life, we must do so as ourselves, with our identities revealed.  Online discussions are now a part of the public sphere, and when used well, can bring people together in ways that complement and enhance real-world community.

A related trend I'm appreciating is the increasing number of tools available to help make online conversations more personalized.  A particular tool I'd like to encourage you to start using right now is that of a Gravatar - a "globally recognized avatar" - which displays an image of your choosing (sometimes a photo of you) next to your contributions to online conversations.

Here are 4 reasons why you should: Continue reading "4 reasons to start using Gravatars right now"

Using Stock Photos to Show You Care

Creepy scary stock photoOne of the funniest parts of browsing the Internets is when I come across the funny stock photos of professional people in various professional settings, used by site owners to put a "human face" on their web presence in the most generic way possible. It began with using the headshot of the attentive and waiting customer service representative to show you that "operators are standing by now," and it's just gone crazy from there.

With the photo here, I don't even know what the hell is going on. It's like the creepy older guy is trying to arm wrestle with the maniacally screaming younger dude over who gets to use the laptop, while the two women totally ignore them and instead grin broadly at the hamster dancing on their screen. But I'm like "creepy older dude, BACK OFF!" Why does he need to lunge into younger dude's space like that, using his fingertips as a push-off to further invade? And why won't either of the women help younger dude? This is some messed up stock photography. What was the photographer yelling at them? "Pretend you went to the office holiday party and took Ecstasy!"
Continue reading "Using Stock Photos to Show You Care"

Flickr Photos Featured

I love Flickr, which shows off the world we live in through the eyes and camera lenses of everyday folks. In the last week I've had a few of my photos posted there (most of which are released under the Creative Commons license) appear in some interesting spots:

  • Schmap Nashville Travel Guide: a couple of photos I took at the Tennessee State Museuem are apparently now a part of the fourth edition of this online tourist guide. There may be a print version too, but I'm not sure where to find it.
  • All Around Ohio for November 11: a photo I took of Antioch Hall at Antioch College in Yellow Springs, Ohio was featured in an article about the college staying open.