Facebook Archives - Chris Hardie

October 8, 2019October 15, 2019

Unlocking community event information from Facebook

It wasn't always this way.

It might be hard to remember, but there was a time when Facebook wasn't the only place to learn about upcoming events in our communities. Instead of having to scroll past silly memes and political rants to get the details of the next potluck, lecture or book club you cared about, the information was available in lots of other places. Events were available on websites and apps, and they were shared via constantly updated feeds that you could even integrate directly into your own personal calendaring system.

You could learn about what was happening down the street or across town, and you didn't have to give up your online privacy to do it.

At some point, this shifted. Weary of the duplication of efforts maintaining event information in multiple places, and increasing fragmentation of sources that you needed to consult to figure out what's happening, people longed for a more centralized, authoritative spot to enter and learn about community events. I was one of those people.

Enter Facebook

Facebook saw this opportunity and jumped on it.

They made it easy to enter and promote events, and they sprinkled in social features to make it extra compelling to do. Not only could you learn about the event itself, you could see who among your friends was planning on (or interested in) going, and what they had to say about it. If you were an event organizer you could see what kinds of events were catching the interest of your target audience, and you could more easily avoid scheduling your big fundraising gala on the same night as the art museum open house.

At first, Facebook made access to event data one of the most friction-free parts of its platform, and they didn't require you to be "on Facebook" to see it. You could get email notifications, RSS feeds, ICS feeds to bring event information into your own productivity tools and daily life. There was API access so that you could build tools and websites that incorporated the event data where you wanted it to be.

Over time, all of that access was removed. The APIs were shut off, the feeds were shut down, and the message was clear: you have to come to the Facebook website or app to learn about events that you might want to attend.

It's nothing new to note that Facebook's user interface and business model are built around keeping people inside its walled garden. Whatever you might want to contribute to or get from your Facebook experience, they want you to do it on their website or in their mobile app, and on their terms. Their ability to sell advertising space depends on it.

But as with many other aspects of Facebook culture and its grip on our personal and community data, there's a significant downside. Facebook's decision to lock up event information has real implications for how we encounter and experience public life in our real-world communities.

Why is this a problem?

It means that like our exposure to news or updates from our online friends, our awareness of community events is driven by a black-box algorithm optimized around profit over everything else. It means that one company's shifting views on what constitutes an acceptable event, and its sensitivity to the interests of paying advertisers and political organizations could determine whether we see the details of an upcoming protest, demonstration or other exercising of free speech.

And it means that anyone concerned about the privacy implications of having their interest in a given event tracked, sold and monetized may have less exposure to the events that may have traditionally shaped and defined public life in their community, and private life in their circles of friends and neighbors.

There are practical implications too. An organization or business that still chooses to have its own website is having to enter and maintain event information in multiple places, which is time consuming and inefficient. People who want to avoid using Facebook are either left out of the loop or forced into using it again.

I went looking for solutions to this in the context of a website project I was working on recently, where the request was simple: can we bring our Facebook event data into our WordPress site? I had naively assumed that Facebook would have an interest in making this easy: if people could enter their events in one place and have them pushed out to everywhere that mattered, it would be so much easier to see them as the natural place to maintain that information.

But again, they don't make it easy. There's no API available to fetch event data, even for a Facebook page on which I'm an administrator. Event data is not displayed on public-facing Facebook.com pages in any kind of structured ways, and in fact it is rendered in ways that make it resistant to traditional "scraping" tools. There are no other user- or developer-friendly tools for working with event data that I can find.

Yes, Facebook does have a "page widget" that lets you display event information from a Facebook page elsewhere via embedding, if you are an admin on that page. The layout and customization options are pretty limited, but more importantly this is not the same thing as having access to the event information itself for importing, displaying, searching, archiving or other actions that someone might want to take.

Asking page owners to initiate displaying the event information elsewhere also eliminates the chance for other interested parties to "slice and dice" event data in potentially useful or interesting ways. If I want to create a website where you can search and display all of the animal adoption-related events in the Facebook facebook happening within 50 miles of my zip code, I can't. I can't do this even if all of the animal adoption agencies enter their events on Facebook, make them public and put page widgets up on their own website.

Can software fix this?

Being a software developer who works all day long on tools that try to make publishing easier and more interconnected, as I thought about this issue I still felt like there must be some way to extract event data from publicly shared Facebook events. After all, if you can see them in your web browser even while not logged in to Facebook, then that means the data is by definition publicly accessible in some form.

After analyzing the structure of a Facebook web page, its Javascript and the asynchronous HTTP calls made to fully render the content on it, I found that there is a way. Facebook's own event display pages are making a POST to the URL https://www.facebook.com/api/graphql/ to retrieve relevant event details, which are then rendered in HTML and CSS for a normal user to see. I sniffed the request and removed all of the query parameters I could find that might be extraneous. Here's an example of a resulting logged-out POST request that retrieves a batch of the upcoming events for a given Facebook page (my local Parks and Recreation Department):

POST /api/graphql/ HTTP/1.1
Host: www.facebook.com
Content-Type: application/x-www-form-urlencoded
Origin: https://www.facebook.com
User-Agent: PostmanRuntime/7.15.2
Accept: */*
Cache-Control: no-cache
Host: www.facebook.com
Accept-Encoding: gzip, deflate
Content-Length: 255
Connection: keep-alive
cache-control: no-cache

fb_api_req_friendly_name=PageEventsTabUpcomingEventsCardRendererQuery&amp;variables=%7B%22pageID%22%3A%2250650939390%22%7D&amp;__req=8&amp;__user=0&amp;av=0&amp;__a=1&amp;__be=1&amp;dpr=2&amp;fb_api_caller_class=RelayModern&amp;__pc=PHASED%3ADEFAULT&amp;__comet_req=false&amp;doc_id=2343886202319301

I can't tell you what all of the query parameters mean but I think doc_id is the Facebook-internal indicator of a shorthand for which query to run (so apparently "2343886202319301" maps to "all upcoming events" or something like that) and pageID is the internal ID of the Facebook page itself. I can imagine this query breaking later on if Facebook decided to update the doc_id mappings, but for the time being, it works.

The result of the request is a JSON object that contains all the upcoming event info one would need to import Facebook events into another system, display them on a non-Facebook website, and so on.

From this point, I could imagine creating a tool that, given a list of Facebook pages, automatically and regularly grabs all of the upcoming public events available in Facebook and does something useful with them. Here's a proof of concept PHP script that lays the groundwork for that.

	<?php

	/**
	* Proof of concept, retrieve publicly-available Facebook page event data.
	* Chris Hardie <chris@chrishardie.com>
	*
	* To use, first add Guzzle as a dependency:
	* $ commposer require guzzlehttp/guzzle
	*
	*/

	require 'vendor/autoload.php';

	use GuzzleHttp\Client;

	define( 'FB_UPCOMING_EVENT_DOC_ID', 2343886202319301 );

	$facebook_pages = array(
	'50650939390' => array(
	'page_name' => 'Richmond Parks Department',
	'page_url' => 'https://www.facebook.com/richmondparks/',
	'category' => 'outdoor',
	),
	);

	$facebook_base_params = array(
	'fb_api_req_friendly_name' => 'PageEventsTabUpcomingEventsCardRendererQuery',
	'__req' => '8',
	'__user' => '0',
	'av' => '0',
	'__a' => '1',
	'__be' => '1',
	'dpr' => '2',
	'fb_api_caller_class' => 'RelayModern',
	'__pc:PHASED' => 'DEFAULT',
	'__comet_req' => 'false',
	);

	$client = new Client(
	[
	// Base URI is used with relative requests
	'base_uri' => 'https://www.facebook.com',
	// You can set any number of default request options.
	'timeout' => 2.0,
	]
	);

	$headers = [
	'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36',
	'Content-Type' => 'application/x-www-form-urlencoded',
	'Origin' => 'https://www.facebook.com',
	];

	foreach ( $facebook_pages as $facebook_page_id => $facebook_page ) {

	$facebook_variable_value = '{"pageID":"' . $facebook_page_id . '"}';

	$facebook_form_params = array_merge(
	$facebook_base_params,
	array(
	'variables' => $facebook_variable_value,
	'doc_id' => FB_UPCOMING_EVENT_DOC_ID,
	)
	);

	$upcoming_event_response = $client->request(
	'POST',
	'/api/graphql/',
	[
	'headers' => $headers,
	'form_params' => $facebook_form_params,
	]
	);

	$response_code = $upcoming_event_response->getStatusCode();

	if ( ! empty( $response_code ) && ( 200 === $response_code ) ) {
	$body = $upcoming_event_response->getBody();

	if ( null !== $body ) {
	$fb_response = json_decode( $body, true );
	$fb_events = $fb_response['data']['page']['upcoming_events']['edges'];
	}
	}

	if ( ! empty( $fb_events ) && ( 0 < count( $fb_events ) ) ) {
	foreach ( $fb_events as $fb_event ) {
	$event = $fb_event['node'];

	echo $event['id'] . PHP_EOL;
	echo $event['name'] . PHP_EOL;
	echo $event['time_range']['start'] . PHP_EOL;
	echo $event['event_place']['contextual_name'] . PHP_EOL;
	echo $event['event_place']['city']['contextual_name'] . PHP_EOL;
	echo $event['event_buy_ticket_url'] . PHP_EOL;
	echo PHP_EOL;

	}
	}
	}

	exit;

view raw

get-facebook-events.php

hosted with ❤ by GitHub

Am I going to get in trouble for this?

Does Facebook care if we access event data in this way? Yes, yes they do. Their terms of service about Automated Data Collection explicitly states that if you try to extract data from the Facebook platform in an automated way, they can ban you forever. But more likely than anything is that they will change the way their platform works to make this kind of data retrieval even more difficult.

I think there's hope for some shifting expectations about what's considered fair here. A U.S. appeals court ruled just last month that web scraping does not constitute illegal activity, and went so far as to acknowledge that if a scraper is retrieving publicly available data owned not by the platform but by its users, the scraping can't be blocked.

In the case of Facebook event data, it's worth noting again that we're talking about publicly available information shared by organizations that want it to get more exposure online about events that they want the public to attend. Restricting access to that because it's a potential source of advertising revenue seems beyond greedy to me. In the end though, it's up to organizations, businesses and individuals to decide whether they want to have their event data locked up, or out on the open web.

You can help

If you are an event organizer, please consider posting your event data outside of Facebook on a publicly available website!

If you are someone who cares about building a healthy culture of civic engagement in your community, advocate for the organizations you're involved with to move away from tools that make this harder in the long term!

If you are a developer at Facebook or anywhere else that builds tools for people to share information, please don't lock up that information! Show your commitment to the open web. Provide APIs, RSS (or in this case, ICS) feeds, good documentation and facilitate easy exports of user-owned data. (A quick shout-out to the folks at Eventbrite who, at least for now, make available for free a very robust API to access community events shared on their platform.)

I'm glad for any feedback and suggestions about these challenges; please comment below. I've also explored the themes discussed here in various past posts, including:

June 28, 2018June 27, 2018

A year without Facebook

It's been about a year since I left Facebook, and I'm still glad I did. (I guess there were those thirty years before Facebook existed that I somehow managed without it, too.)

Some observations:

People in my circles generally continue to assume that I've seen their event invitations and life updates on Facebook, and so it's still a regular occurrence that I find out about something well after everyone else, or not at all. This is most annoying when it's something really time sensitive that I would have liked to have been a part of.

Some of my published writings have been shared extensively on Facebook, generating hundreds or even thousands of views on my various websites, but I don't have a way of knowing where that activity is coming from or what kind of conversation it might be generating there. I've had people tell me in person that they saw and liked something via Facebook, which is nice, but of course I wish they'd leave their likes and comments on my site where it's closer to the original writing, visible to the world, and not subject to later deletion by some corporate entity. (This comes up for any social network, not just Facebook, but it tends to be the one generating the most traffic for me.)

I won't make a claim that the hours I've saved by not looking at Facebook have freed me up to accomplish some amazing other thing. I will say that I felt a nice release from the self-created pressure to keep up with my interactions and profile there, and that in turn has contributed to an increase in my overall creative energy for other things.

I had one time where I needed to use the Facebook sharing debugger for a work project. I signed up for a new account to do this, but Facebook clearly found my lack of interest in populating a real-looking profile to be suspicious, and closed down the account soon after. In the end it was faster to ask a colleague with an active account to do the debugging for me and share the results. As I've said before, I think it's ridiculous and irresponsible that Facebook doesn't make that tool available to logged-out users.

I'm still surprised at how many organizations and businesses use Facebook as their one and only place for posting content; some even do it in a way that I just can't see it as a logged-out user, and others don't seem to realize that they're giving Facebook 80% of any screen real estate on the links I can see. I am now much more likely to avoid doing business with or offering my support to these entities if they don't bother offering non-Facebook ways for me to engage.

I've accepted that people will not necessarily seek out the open version of the web on their own. Being off Facebook has reinforced that there are big gaps to close in the user experiences that other tools and services offer (the WordPress/blogging ecosystem not least among them). My own efforts to migrate my content that still exists on other services like Flickr into a digital home that I fully control are slow-going, so I don't expect other people to even bother. Facebook is still the path of least resistance for now.

When the actions of Cambridge Analytica were in the news, it was tempting to feel smug about not being an active Facebook user. But I know they still have tons of information about me that is of value to advertisers and others, and that even as I use browser plugins to try to prevent Facebook from accumulating an even larger profile of my online activity, it is a losing battle until there are larger shifts in the culture and business models of technology companies.

August 26, 2017October 31, 2017

Leaving Facebook

A month ago I deactivated my Facebook account and I haven't used it since.

In the larger scheme of things this is a relatively inconsequential decision, and one that I may even eventually reverse. But for the benefit of my future self, and for anyone else who's interested, I thought I'd inventory my thinking behind this change:

1. My time spent on Facebook was minimal, especially since I deleted the app from my phone a few years ago. But it was almost entirely time I was not spending on things that actually matter to me. Any effort I put in to keeping up with my feed felt like it was actively distorting my understanding of how the real world works, how people are actually doing and what they're actually thinking and feeling. Even as distractions from the real world go, it was often more unsatisfactory and harmful than helpful.

2. I continue to be uncomfortable with Facebook's "walled garden" model of connecting people, content and conversations. I've nothing against the company or the people who work there, and actually admire their technical innovation, scale and impact on the world. But as someone who genuinely wants to see people truly own and control their online presence, I can't in good faith spend time contributing to something that undermines the open web. When I tried being just a casual observer without engaging, it didn't work. Which brings me to:

Continue reading "Leaving Facebook" →

August 10, 2014March 21, 2016

Facebook messages autoresponder

I went looking today for tools to create an autoresponder for Facebook's private messaging functions. I try to avoid using Facebook's messaging whenever possible, but that doesn't stop someone who I'm connected to there from sending me a private message, which then most often sits unreplied for weeks or months. Having an autoreply that encouraged message senders to email me instead would save me some time and help make sure the contact attempt got through in a timely manner.

The bottom line is that the options are very limited and I may need to build my own if it feels important enough to pursue. In the meantime I thought I'd post my findings here in case there are others looking for the same, or who have new ideas to share.

Continue reading "Facebook messages autoresponder" →

February 4, 2013

Facebook Appreciation Day?

Idea:

What if Facebook shut down once per day, every year?

Turn it all the way off. No one could get to it. No walls, timelines, profiles, friends, games, apps or messages.

They could call it Facebook Appreciation Day.

Some people would appreciate that Facebook was off for the day and turn their attention to other things.

Some people would appreciate how much they enjoy / like / depend on Facebook the other 364 days of the year.

Facebook's servers and employees could appreciate the day off, or maybe they could do some deep cleaning.

I'm only partly joking here:

A ritual of sabbath from something that has become so engrained in modern culture, something that many people can't imagine NOT using in some form every day, could be useful.

Having everyone who uses Facebook experience it on the same day, together, would just be amazing.

What would you do on Facebook Appreciation Day?

August 12, 2012October 24, 2019

Facebook Likes as protected free speech

Daniel Ray Carter Jr., a sheriff's deputy in Virginia, claims he was fired because he "Liked" a Facebook post belonging to the political rival of his own boss. When he fought the firing in court, the judge ruled against him saying that clicking the "Like" button isn't protected speech: "It is not the kind of substantive statement that has previously warranted constitutional protection."

The case presents an interesting dilema.

On one hand, I hope we're reaching the point where most people understand that clicking the Facebook "Like" on a statement, article or page is not the equivalent of an endorsement of all the things that article/page/group stands for.

Continue reading "Facebook Likes as protected free speech" →

November 23, 2011February 18, 2020

ChrisHardie.com on Facebook

If you're like many people, you've renounced the joys and complexities of face to face socialization in favor of robotic and impersonal displays of feigned consideration on Facebook.

Wait, that didn't come out quite right.

If you're like many people, Facebook plays some part in your daily engagement with friends, family, coworkers and/or your surrounding community.

That's a little better.

Well, this website now has a public Facebook presence that you can add to that engagement.

Continue reading "ChrisHardie.com on Facebook" →

July 12, 2011July 12, 2011

Initial thoughts on Google+

I've had a few days to play around with Google's new social network offering, Google+, and I thought I'd share some initial thoughts.

First of all, kudos to Google for "going for it" in the Facebook era. They're one of few players who actually has the resources and skill to make a serious go at a viable alternative to Facebook, and you've got to admire the effort. If the success of the movie The Social Network tells us anything, it's that Facebook has become mainstream and popular, and as generations of younger people look for ways to establish their identity in the digital age, they'll be looking for alternatives to the place where their parents and now grandparents also hang out online. By the same token, people of all ages and professions are trying to figure out just how to effectively and safely use Facebook, LinkedIn, Twitter and other social media tools in a world where we're being encouraged to blend our personal and professional lives together more publicly.

Is Google+ just the right thing at just the right time?

People are already writing about the high bar that Google+ will have to jump in order to see any significant migration of Facebook users, not the least of which is all the time people have invested in curating their lists of "friends" there. Facebook is going to make it as difficult as possible for its users to do any kind of exporting of account information from their system, and I don't think Google is devious enough to launch an unauthorized workaround. So people will be left to recreate their online identity on Google+, where the number of people you are connected to still largely drives your user experience.

Continue reading "Initial thoughts on Google+" →

October 2, 2010

The Social Network

I saw the movie The Social Network tonight, here are my spoiler-free comments.

The movie was incredibly well made. Aaron Sorkin's writing was as good as the best days of The West Wing, each member of the cast seemed to just nail their role, the editing was some of the best I've seen, and so on.

Perhaps most enjoyably, this is a mainstream movie that is at least in part about the culture and goings-on in the modern world of Internet entrepreneurship, I believe the first of its kind. It fully embraces the geekiness that was and is a part of building a web application like Facebook: in the first 30 minutes, the Apache webserver software project is mentioned at least twice, there are dramatic lines about needing more Linux webservers running MySQL, there are punchlines that involve the emacs text editor, and scenes of glorious code writing marathons - wow.

Continue reading "The Social Network" →