Unlocking community event information from Facebook

It wasn't always this way.

It might be hard to remember, but there was a time when Facebook wasn't the only place to learn about upcoming events in our communities. Instead of having to scroll past silly memes and political rants to get the details of the next potluck, lecture or book club you cared about, the information was available in lots of other places. Events were available on websites and apps, and they were shared via constantly updated feeds that you could even integrate directly into your own personal calendaring system.

You could learn about what was happening down the street or across town, and you didn't have to give up your online privacy to do it.

At some point, this shifted. Weary of the duplication of efforts maintaining event information in multiple places, and increasing fragmentation of sources that you needed to consult to figure out what's happening, people longed for a more centralized, authoritative spot to enter and learn about community events. I was one of those people.

Enter Facebook

Facebook saw this opportunity and jumped on it.

They made it easy to enter and promote events, and they sprinkled in social features to make it extra compelling to do. Not only could you learn about the event itself, you could see who among your friends was planning on (or interested in) going, and what they had to say about it. If you were an event organizer you could see what kinds of events were catching the interest of your target audience, and you could more easily avoid scheduling your big fundraising gala on the same night as the art museum open house.

At first, Facebook made access to event data one of the most friction-free parts of its platform, and they didn't require you to be "on Facebook" to see it. You could get email notifications, RSS feeds, ICS feeds to bring event information into your own productivity tools and daily life. There was API access so that you could build tools and websites that incorporated the event data where you wanted it to be.

Over time, all of that access was removed. The APIs were shut off, the feeds were shut down, and the message was clear: you have to come to the Facebook website or app to learn about events that you might want to attend.

It's nothing new to note that Facebook's user interface and business model are built around keeping people inside its walled garden. Whatever you might want to contribute to or get from your Facebook experience, they want you to do it on their website or in their mobile app, and on their terms. Their ability to sell advertising space depends on it.

But as with many other aspects of Facebook culture and its grip on our personal and community data, there's a significant downside. Facebook's decision to lock up event information has real implications for how we encounter and experience public life in our real-world communities.

Why is this a problem?

It means that like our exposure to news or updates from our online friends, our awareness of community events is driven by a black-box algorithm optimized around profit over everything else. It means that one company's shifting views on what constitutes an acceptable event, and its sensitivity to the interests of paying advertisers and political organizations could determine whether we see the details of an upcoming protest, demonstration or other exercising of free speech.

And it means that anyone concerned about the privacy implications of having their interest in a given event tracked, sold and monetized may have less exposure to the events that may have traditionally shaped and defined public life in their community, and private life in their circles of friends and neighbors.

There are practical implications too. An organization or business that still chooses to have its own website is having to enter and maintain event information in multiple places, which is time consuming and inefficient. People who want to avoid using Facebook are either left out of the loop or forced into using it again.

I went looking for solutions to this in the context of a website project I was working on recently, where the request was simple: can we bring our Facebook event data into our WordPress site? I had naively assumed that Facebook would have an interest in making this easy: if people could enter their events in one place and have them pushed out to everywhere that mattered, it would be so much easier to see them as the natural place to maintain that information.

But again, they don't make it easy. There's no API available to fetch event data, even for a Facebook page on which I'm an administrator. Event data is not displayed on public-facing Facebook.com pages in any kind of structured ways, and in fact it is rendered in ways that make it resistant to traditional "scraping" tools. There are no other user- or developer-friendly tools for working with event data that I can find.

Yes, Facebook does have a "page widget" that lets you display event information from a Facebook page elsewhere via embedding, if you are an admin on that page. The layout and customization options are pretty limited, but more importantly this is not the same thing as having access to the event information itself for importing, displaying, searching, archiving or other actions that someone might want to take.

Asking page owners to initiate displaying the event information elsewhere also eliminates the chance for other interested parties to "slice and dice" event data in potentially useful or interesting ways. If I want to create a website where you can search and display all of the animal adoption-related events in the Facebook facebook happening within 50 miles of my zip code, I can't. I can't do this even if all of the animal adoption agencies enter their events on Facebook, make them public and put page widgets up on their own website.

Can software fix this?

Being a software developer who works all day long on tools that try to make publishing easier and more interconnected, as I thought about this issue I still felt like there must be some way to extract event data from publicly shared Facebook events. After all, if you can see them in your web browser even while not logged in to Facebook, then that means the data is by definition publicly accessible in some form.

After analyzing the structure of a Facebook web page, its Javascript and the asynchronous HTTP calls made to fully render the content on it, I found that there is a way. Facebook's own event display pages are making a POST to the URL https://www.facebook.com/api/graphql/  to retrieve relevant event details, which are then rendered in HTML and CSS for a normal user to see. I sniffed the request and removed all of the query parameters I could find that might be extraneous. Here's an example of a resulting logged-out POST request that retrieves a batch of the upcoming events for a given Facebook page (my local Parks and Recreation Department):

POST /api/graphql/ HTTP/1.1
Host: www.facebook.com
Content-Type: application/x-www-form-urlencoded
Origin: https://www.facebook.com
User-Agent: PostmanRuntime/7.15.2
Accept: */*
Cache-Control: no-cache
Host: www.facebook.com
Accept-Encoding: gzip, deflate
Content-Length: 255
Connection: keep-alive
cache-control: no-cache

fb_api_req_friendly_name=PageEventsTabUpcomingEventsCardRendererQuery&variables=%7B%22pageID%22%3A%2250650939390%22%7D&__req=8&__user=0&av=0&__a=1&__be=1&dpr=2&fb_api_caller_class=RelayModern&__pc=PHASED%3ADEFAULT&__comet_req=false&doc_id=2343886202319301

I can't tell you what all of the query parameters mean but I think doc_id is the Facebook-internal indicator of a shorthand for which query to run (so apparently "2343886202319301" maps to "all upcoming events" or something like that) and pageID is the internal ID of the Facebook page itself. I can imagine this query breaking later on if Facebook decided to update the doc_id mappings, but for the time being, it works.

The result of the request is a JSON object that contains all the upcoming event info one would need to import Facebook events into another system, display them on a non-Facebook website, and so on.

From this point, I could imagine creating a tool that, given a list of Facebook pages, automatically and regularly grabs all of the upcoming public events available in Facebook and does something useful with them. Here's a proof of concept PHP script that lays the groundwork for that.


<?php
/**
* Proof of concept, retrieve publicly-available Facebook page event data.
* Chris Hardie <chris@chrishardie.com>
*
* To use, first add Guzzle as a dependency:
* $ commposer require guzzlehttp/guzzle
*
*/
require 'vendor/autoload.php';
use GuzzleHttp\Client;
define( 'FB_UPCOMING_EVENT_DOC_ID', 2343886202319301 );
$facebook_pages = array(
'50650939390' => array(
'page_name' => 'Richmond Parks Department',
'page_url' => 'https://www.facebook.com/richmondparks/',
'category' => 'outdoor',
),
);
$facebook_base_params = array(
'fb_api_req_friendly_name' => 'PageEventsTabUpcomingEventsCardRendererQuery',
'__req' => '8',
'__user' => '0',
'av' => '0',
'__a' => '1',
'__be' => '1',
'dpr' => '2',
'fb_api_caller_class' => 'RelayModern',
'__pc:PHASED' => 'DEFAULT',
'__comet_req' => 'false',
);
$client = new Client(
[
// Base URI is used with relative requests
'base_uri' => 'https://www.facebook.com',
// You can set any number of default request options.
'timeout' => 2.0,
]
);
$headers = [
'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36',
'Content-Type' => 'application/x-www-form-urlencoded',
'Origin' => 'https://www.facebook.com',
];
foreach ( $facebook_pages as $facebook_page_id => $facebook_page ) {
$facebook_variable_value = '{"pageID":"' . $facebook_page_id . '"}';
$facebook_form_params = array_merge(
$facebook_base_params,
array(
'variables' => $facebook_variable_value,
'doc_id' => FB_UPCOMING_EVENT_DOC_ID,
)
);
$upcoming_event_response = $client->request(
'POST',
'/api/graphql/',
[
'headers' => $headers,
'form_params' => $facebook_form_params,
]
);
$response_code = $upcoming_event_response->getStatusCode();
if ( ! empty( $response_code ) && ( 200 === $response_code ) ) {
$body = $upcoming_event_response->getBody();
if ( null !== $body ) {
$fb_response = json_decode( $body, true );
$fb_events = $fb_response['data']['page']['upcoming_events']['edges'];
}
}
if ( ! empty( $fb_events ) && ( 0 < count( $fb_events ) ) ) {
foreach ( $fb_events as $fb_event ) {
$event = $fb_event['node'];
echo $event['id'] . PHP_EOL;
echo $event['name'] . PHP_EOL;
echo $event['time_range']['start'] . PHP_EOL;
echo $event['event_place']['contextual_name'] . PHP_EOL;
echo $event['event_place']['city']['contextual_name'] . PHP_EOL;
echo $event['event_buy_ticket_url'] . PHP_EOL;
echo PHP_EOL;
}
}
}
exit;

Am I going to get in trouble for this?

Does Facebook care if we access event data in this way? Yes, yes they do. Their terms of service about Automated Data Collection explicitly states that if you try to extract data from the Facebook platform in an automated way, they can ban you forever. But more likely than anything is that they will change the way their platform works to make this kind of data retrieval even more difficult.

I think there's hope for some shifting expectations about what's considered fair here. A U.S. appeals court ruled just last month that web scraping does not constitute illegal activity, and went so far as to acknowledge that if a scraper is retrieving publicly available data owned not by the platform but by its users, the scraping can't be blocked.

In the case of Facebook event data, it's worth noting again that we're talking about publicly available information shared by organizations that want it to get more exposure online about events that they want the public to attend. Restricting access to that because it's a potential source of advertising revenue seems beyond greedy to me. In the end though, it's up to organizations, businesses and individuals to decide whether they want to have their event data locked up, or out on the open web.

You can help

If you are an event organizer, please consider posting your event data outside of Facebook on a publicly available website!

If you are someone who cares about building a healthy culture of civic engagement in your community, advocate for the organizations you're involved with to move away from tools that make this harder in the long term!

If you are a developer at Facebook or anywhere else that builds tools for people to share information, please don't lock up that information! Show your commitment to the open web. Provide APIs, RSS (or in this case, ICS) feeds, good documentation and facilitate easy exports of user-owned data. (A quick shout-out to the folks at Eventbrite who, at least for now, make available for free a very robust API to access community events shared on their platform.)

I'm glad for any feedback and suggestions about these challenges; please comment below. I've also explored the themes discussed here in various past posts, including:

 

Star Trek values

A few weeks ago I had the pleasure of attending a panel at Silicon Valley Comic Con consisting of various members of the cast of the television show Star Trek: The Next Generation. Brent Spiner, Gates McFadden, Jonathan Frakes, Denise Crosby, Marina Sirtis and Robert O'Reilly were joined by original series cast member William Shatner to talk about the show, their lives as actors, and what the Star Trek universe has taught and can teach us about the real world.

For someone who watched every episode of the show when it originally aired and who has remained a fan since, it was an hour and a half of nearly pure joy. For one, I was just excited to be a part of a whole auditorium full of people reflecting on how much influence the stories, dialogue and creativity of the show had on our lives, the panelists included. Several audience members stood up to say just how strong that influence has been, informing the careers they chose, the people they've become, the kind of lives that they now lead, and I was right there with them.

In watching Star Trek as a young person I remember being invigorated by the complex problem-solving scenarios that the Enterprise crew faced week after week. I learned from the principles of collaboration, mutual respect and cross-species equity that were practiced. I saw strong women in leadership roles, and I saw non-Caucasian characters developed with an unusual (for mainstream TV, anyway) depth and texture. And I was inspired by a vision of the future that offered so many possibilities for exploration, discovery and growth. I'm sure that my real-world evolution as technologist and computer geek was propelled forward significantly by my immersion in that make-believe world of technological wonders.

The panel also highlighted a new angle of my appreciation for the Star Trek universe. I hadn't previously thought about Trek as having a political point of view, because I assumed that the vision of a world where humanity had figured out how to eliminate poverty and hunger, celebrated and built on our various differences, and employed innovation to protect and restore the environment was a vision that anyone would embrace and want to strive for, and not a particularly politically-charged one.

Continue reading "Star Trek values"

WordCamp for Publishers Denver

I'm excited to be a part of the team that is organizing the first WordCamp for Publishers happening in Denver, Colorado this coming August 17th to 19th.

In my work on Automattic's VIP service offering I've been able to see some of the incredible things that journalists and publications of all types are doing with WordPress - really interesting stuff that pushes the software in new directions. So when I saw some of my colleagues discussing an event that would bring engineering, product and editorial teams together around all the things WordPress can do in the publishing world (for publications big and small), I wanted to be a part of making it happen.

We're looking for speakers and sponsors now. If you know someone who might want to be a part of this first-of-a-kind event, please point them in our direction. And if you're involved in publishing with WordPress at any scale, or just want to learn about how media organizations are using it to modernize their publishing  workflow, I hope you'll consider attending. Tickets ($40) will go on sale soon!

Speaking at SupConf NYC

I'll be speaking at the SupConf New York City 2016 conference behind held at the Digital Ocean HQ in Soho on November 15th and 16th.  My talk is on a topic I've enjoyed thinking about and working on over the years, moving customer support interactions from transactions to partnerships. I'm also excited to meet and hear from some other great speakers.

If you're interested and can make it, tickets are available now. I'll also be sure to share the content of my talk here after the event.

Update December 3, 2016: My slides from the talk are below. Continue reading "Speaking at SupConf NYC"

Anna Deavere Smith's questions

Anna Deavere SmithI had the pleasure of seeing actress, playwright and professor Anna Deavere Smith - known for roles on The West Wing and Nurse Jackie among other things - speak and perform today at Earlham College about the role of the artist in society.

She told us of her desire to get people to drop their typical verbal defenses and talk in broken sentences, to interrupt the rhythms of prepared, calculated speech to get to the truth underneath. She said that someone had once given her three questions you can ask someone in a conversation to help make that happen, and I want to share those here:

Continue reading "Anna Deavere Smith's questions"

Powerlessness and Empowerment with Frances Moore Lappé

A week ago I had the opportunity to hear Frances Moore Lappé speak here in Richmond. She's primarily known around the world as author of Diet for a Small Planet, but she's also an Earlham College graduate, so it was great that she came back to her alma mater to give a talk.

Lappé's talk overall was about how we can move from a place of powerlessness to a place of empowerment when it comes to working on addressing various ills that plague the world - from climate change to energy/resource crises to poverty, and all of the other systems and issues that are related.

It's a topic, a question that's been on my mind lately as I think about my own vocation, and where (to borrow from Frederick Buechner) my talents and interests might meet the world's deep needs. The question wasn't answered for me during the talk, but there were a few insights and random bits of wisdom that I want to preserve here:

Continue reading "Powerlessness and Empowerment with Frances Moore Lappé"

Quantitative easing and structural unemployment

Globalization // Coming 2 a mystical cliffside near u - v.2That title really roped you in, huh?  Allow me to explain.

Earlier today I attended the Indiana University 2012 Business Outlook Panel in its visit to Richmond.  It's a group that "has presented national, state, and local economic forecasts for the coming year to business, political, and community leaders of Indiana" for the last 38 years.  I attended the same gathering back in 2005 and I have to say that today's commentary wasn't much different from what it was six years ago: "things are not great with the economy, but there are reasons to be cautiously optimistic."

As I noted in my reflections from the 2005 event, there were a couple of troubling ideas that permeated the remarks, especially from the panelists looking at global and national trends.

Continue reading "Quantitative easing and structural unemployment"

Upcoming speaking events

I have a few upcoming speaking events that you might be interested in:

  1. Capitalism vs. The Environment: A small business perspective on doing well AND doing good. This coming Thursday September 24th at 4 PM at Indiana University East in Whitewater Hall Room 132 the Community Room, free and open to the public, no registration required.  I'll be talking about our experiences at Summersault as we've tried to do the "right thing" when it comes to the environment and nurturing sustainable lifestyles, and examine whether it's even possible to pursue a for-profit technology venture and not be in a harmful relationship with the land and life around us.
  2. Get Techie, Get Social! A workshop to help non-technical people learn more about technical topics, especially social media like Facebook and Twitter.  Monday September 28th from 9:30 AM to 11:30 AM at Morrisson-Reeves Library, free and open to the public, no registration required.  If you're at all intimidated by some of those "newfangled Internet trends" or the Internet in general, I hope you'll bring your questions and frustrations so we can work them out together.  There's a PDF flyer for the event.
  3. TCP/IP topics in Introduction to Computers and Computing.  I'll be speaking at this IU East class on Monday October 12th about things like DNS/Bind, network topologies and routing, e-mail technologies, and web technologies.  This particular class isn't open to the public, but if you have folks interested in discussing these topics in technical detail, let me know and I'd love to speak with your group.
  4. Communicating Through Technology. Friday October 23rd at 9 AM at a conference for women hosted by the Wayne County Foundation.  I'll be speaking along with co-worker Jane Holman about social media and general technology topics.  You can view the conference brochure in PDF, and registration is required.

I hope you can join me for one of these events!

On the Phantoscope Film Festival in Richmond

I'm retroactively (for 2009) and proactively (for 2010) spreading the word about the Phantoscope Film Festival that just concluded its third year here in Richmond at the Art Museum.  It's an event that is just absolutely phenomenal to be happening right here in town, but that is sadly under-promoted and under-appreciated locally.

Every year, high school students around Indiana are encouraged to submit their films for judging and showing at the festival.  The top ten or so films are selected by a panel of judges, and then shown at the screening night (which was tonight).  Before the screening is a panel discussion with professional filmmakers and those involved in the film industry.

Continue reading "On the Phantoscope Film Festival in Richmond"

My upcoming speaking events

A few upcoming speaking events to note:

  • Rising Above the Noise: Online Strategies & Tools for Success: we'll take on blogging, RSS, micro-blogging, social networking, widgets, photo sharing, and more, and talk about how these tools can help businesses and other organizations get their message out there further. See also the article in today's Pal-Item. Thursday September 25th 8-9 AM at the Uptown Innovation Center in Richmond. Free, contact Main Street to register.
  • The Internet as a Political Tool: this is a repeat of the presentation I did in May, but now with more flavor! I'll talk about how the Internet is changing the world of politics, and what it means for local citizens (especially right before the November election). It's a part of the Technology Series at Morrisson-Reeves Library (see the PDF brochure) that is bringing together local experts to talk about various technology issues. Thursday, October 9th 6:30-8 PM in the Bard Room at MRL in Richmond. Free, contact MRL for details or just show up.
  • General Talk about the Technologies of the Internet: I'll be speaking to a group of freelance artists, web developers and consultants in Cincinnati about how to best use the technologies of the Internet to serve their clients. Friday, October 24th. If you're in the Cincinnati area and are interested in joining the group, contact Katie McGuire.

If you're interested in having me speak to your business or organization, learn more about how to get in touch to make a request.