Archiving threatened outlets for the Freedom of the Press Foundation

A little over a month ago, I launched one of the Special Projects I’ve been working on at my new job at the Freedom of the Press Foundation. The Threatened Outlets collection at Archive-It aims to capture the archives of news sites that we deem vulnerable to “the billionaire problem,” wherein wealthy individuals or organizations can eliminate unflattering coverage through litigation or by purchasing media companies altogether. From the launch announcement:

Those efforts help individual journalists. But another important thing we can do to reduce the effectiveness of this kind of attack on press freedom is to commit ourselves to the wholesale preservation of threatened sites.

In this case, we seek to reduce the “upside” for wealthy individuals and organizations who would eliminate embarrassing or unflattering coverage by purchasing outlets outright. In other words, we hope that sites that can’t simply be made to disappear will show some immunity to the billionaire problem.

To my surprise and delight, our launch received some nice and encouraging coverage, in WIRED, TechCrunch, The Next Web, The New York Times, and others. Sometimes when you’ve got your head down on something that feels like a niche topic, it’s nice to come up for air and realize that the general public is interested, too.

“The Florida Project” and questions of control

I had the opportunity to catch the new movie The Florida Project this week, without any background on how it was made or what it is about. That’s my preferred context for seeing movies, but it does sometimes lead to my feeling a bit adrift in the theater, or—in this case—stressed that something catastrophic was about to go down. Still, I feel like I end up with a better idea of what the movie is actually about if I don’t know the plot going into it. Here’s the trailer, if you want a sense of it.

That was certainly the case with The Florida Project, which has attracted nearly universally positive reviews from critics (and some confusion from audiences about the ending). Just about every review I’ve read since seeing the movie describes is as “empathetic,” which is spot on, but none really dive into what the audience is feeling empathy for. This post lays out my explanation, and to do so includes some very minor spoilers. I don’t really think this movie can be “spoiled,” but skip this post if you want to.

To my mind, what the movie does a great job of showing is how people seek spaces where they can be autonomous and in control. It shows children exploring and imagining, but (most importantly) testing out their agency. At the same time, it shows adults, living precarious existences, staking out small pieces of space where they are in charge.

The building manager Bobby is an obvious example, and his arguments with tenants can almost always be framed in this light: whether he’s enforcing the pool rules, confronting unwelcome strangers, or (most tellingly) moving tenants around to prevent them from establishing residency, he is exerting control over his designated space and limiting the autonomy of the people in it. In that last example, too, he alludes to the fact that he’s an agent of another level of power, specifically pointing at the surveillance cameras to explain how his hands are tied.

Halley’s actions in this area are even more illuminating. You can see it best when she gets into conflict with her best friend Ashley. Though Halley doesn’t even know the basis of their argument, she takes it out on her friend by finding a space where Ashley has no autonomy—her workplace—and exercises arbitrary control over her. From that perspective, it’s only a matter of degree when their dispute escalates to violence, and then when Ashley (it’s implied) calls for an intervention by the state.

In that penultimate scene, where Halley is arguing with DCF agents, she makes this unspoken theme somewhat explicit, saying that they’re on her “property.” Throughout the film there are a lot of reasons to think that unit 323 isn’t really Halley’s “property,” but in that moment she reaches for that language to express that she is in control.

The final scene takes that declaration and escalates it to a sort of surreal and metaphorical place. Echoing the film’s tagline (“Find your kingdom”), Moonee and her friend run to a place that feels like their kingdom, where they are the kings. It’s a bit jarring in a film that’s seemed so grounded and “real” to get such a visual metaphor in the closing moments, but it works (and at the same time drives home the juxtaposition between poverty and expensive fantasy that runs under the whole story).

Maybe most movies could be described in these terms—as Octavia Butler said, “all struggles are, essentially, power struggles.” But I think it brings a lot to The Florida Project in particular to think of areas of control and autonomy. The characters shown in the movie are living precarious existences, and seek that kind of control where they can get it. One message from the film is that, where these kinds of spaces aren’t really available, people will find a way to exert control nonetheless. The empathy in this movie is showing the decisions of people who have been rendered powerless in society.

Policies that perpetuate wealth inequality or pervasive surveillance make communities feel that powerlessness. When people talk about those communities “voting against their interests,” it’s worth considering that interest in exercising some kind of control.

Pulling free and open weather data

When I decided to add realtime weather effects to @choochoobot, I knew there were a few qualities I wanted to find in a data source. Ideally I would find something free and reliable that didn’t require me to agree to many developer terms or sign up for an API token. Google shuttered its undocumented Weather API in 2012, and Yahoo’s offering, which has changed a few times over the years, now requires an account and a consumer key and secret.

It took some poking around but I was eventually successful, and now @choochoobot should correctly show clouds, rain, snow, or thunderstorms, depending on whether observations in New York at the moment it’s tweeting.

Current observed weather conditions seems to me like something that should be provided by the government as open data. Fortunately, that intuition was correct: free realtime weather data is openly available, if you know where to look.

The National Weather Service, an agency within NOAA, provides weather observations from stations all over the US in RSS and parseable XML formats. And while parsing XML in Python isn’t exactly pleasant, it’s straightforward enough and I am now able to get qualitative descriptions of the current weather that I can translate into emoji. Here’s the relevant code, and here’s how it works:

  • Each time @choochoobot tweets, it checks to see whether it’s daytime or nighttime. If it’s nighttime, I skip the weather check and instead calculate the phase and placement of the moon in the sky.
  • If it’s during the day, I make download the XML file provided by the KNYC weather station in Central Park. You can load the same file in your browser at any point, but you probably have to view-source to see it in a reasonable human-readable format. That’s one of about 1,800 locations in US states and territories that the NWS provides information for.
  • @choochoobot then parses the XML file using Python’s built-in ElementTree XML API. The relevant field for my purposes is labeled "weather", which contains a text description of the observed conditions.
  • At least in theory, that phrase will always be one of the 250 or so pre-set descriptions provided by the the NWS. These are sort of grouped into categories—there’s a pretty clear thunderstorm grouping, and one for hail—but it seems a bit ad hoc. My use requires classifying the observed weather into just four or five buckets with matching emoji; I just made a big list of terms that I’d take to mean "cloudy," for example, and checked to see whether the observed weather phrase was on that list.
  • Then I pick emoji for the sky, and put the whole tweet together. If the weather is cloudy, I replace the sun emoji with a sun-behind-clouds emoji. Real scientific stuff.

In case it’s useful, I’ve converted the NWS list of weather conditions to JSON and submitted it to Darius Kazemi’s corpora project. Once that gets merged in, those weather conditions will all be available that way.

“A Mind At Play” and Claude Shannon’s grave

As the “father of information theory,” Claude Shannon’s contributions to the development of modern digital technology are hard to overstate. That work puts him in the ranks of the Einsteins, Turings, and Feynmans of the world; somehow, though, he never seemed to get the credit and public recognition that those scientists received.

A great new biography of Shannon, “A Mind At Play,” tells the story of his work and may provide an explanation. In contrast to the stereotype of a single-minded scholar who doggedly pursues a particular theory for an entire lifetime, Shannon seemed to hop around to whatever captured his imagination—and in more than a few cases, advanced the state of the art or provided entirely new frameworks with which to consider its thorniest problems. (I was fortunate enough to be sent a copy of this book by its author because I had previously posted about a long New Yorker profile of Shannon commemorating the centennial of his birth.)

That professional pathway meant he probably never had a dull moment, but also that he wasn’t around to lead the fields he revolutionized. Beyond that, it makes it difficult to distill his work down to an equation, or even a few sentences. If you had to, though, you could do worse than his formulation of information as entropy. One fascinating detail from the end of “A Mind At Play” was that that equation appears on his grave stone at the Mount Auburn Cemetery in Cambridge where he’s buried.

What’s concealed, however, is a message on the reverse: covered by a bush, the open section of the marble on the back of the tombstone holds Shannon’s entropy formula.Shannon’s children had hoped the formula would grace the front of the stone; their mother thought it more modest to engrave it on the back.

And so Claude Shannon’s resting place is marked by a kind of code: a message hidden from view, invisible except to those looking for it.

This struck me as amazing, and I wanted to see how it was rendered. But, perhaps unsurprisingly, it is difficult to find a picture of the back of a tombstone—even one marking the grave of a person as notable as Claude Shannon. So I turned to Twitter, posting that passage and asking if anybody in or around Cambridge was available to take a picture for me.

Fortunately, it seems many people were as intrigued by that detail as I was, and the tweet was pretty widely circulated. To my surprise and delight, a few groups of people reached out to tell me they were available to go, and a few more just set out for there. I had inadvertently prompted a small flashmob at the grave of a scientist who had passed some 16 years earlier.

The equation is in fact pretty well hidden, but a few folks were able to duck into the bush behind the grave and grab a shot. I love the way it looks, and I was very excited that this new book prompted me and a few nerds around the world to share a moment of appreciation for the great Claude Shannon over a network that his work made possible.

Which states hated Wesley?

One of my goals while at Recurse Center has been to improve my ability to manipulate and visualize data sets. To that end, I’ve been toying around with the Social Security Administration’s baby name dataset, which records the number of babies born with each given name every year, both federally and at the state level. Because I’ve also been watching Star Trek: The Next Generation along with the Treks And The City podcast, I chose to dig into information about the name “Wesley.”

On my first pass through the data I noticed that the name’s popularity dramatically spiked around 1976, and then tapered off for a few decades after. Honestly, that spike is the most interesting property of the whole graph, and I can’t explain it very well. But a funny secondary effect is that neither TNG‘s premiere nor the release of The Princess Bride—both in 1987—could prop up the name as it declined in popularity. The effect makes it look like it’s tumbling off a cliff, instead of regressing to the mean. This graph, including the label, was generated in Python’s matplotlib.

After looking at the federal data, I decided to dig into the state-level stuff, to give me a (long-anticipated!) opportunity to generate a choropleth map. Again, I cleaned up the data in Python, and then generated a map using a Javascript library called d3-geomap. For a long time I’ve wanted to get more familiar with its parent library, d3, and this has been a nice opportunity to dip my toe into that.