Introducing: cursewords, a crossword puzzle solving interface for the terminal

I’m releasing new software today for solving crossword puzzles in the terminal. cursewords is a small Python program to open, navigate, and solve puzzles stored as .puz files. If you’re a Mac or Linux user, you can install it today by running pip3 install --user cursewords in your terminal, and then use the cursewords command to open a .puz file on your computer.

In case you’re not a crossword nerd: the .puz file was developed for popular solving software called AcrossLite, and it remains the most popular format for transmitting crosswords online, from independent creators all the way up to the New York Times.

cursewords in action on my terminal

In fact, many independent puzzle creators only distribute their puzzles as digital files. For example, I subscribe to a handful of excellent puzzle outlets—American Values Club, The Inkubator, Crossword Nation, Fireball Crosswords—that don’t offer an online solver or an app like the Times does. As a Linux user, I didn’t have a lot of options to open them: AcrossLite isn’t compatible, Web-based solutions have their limitations, and beyond that, I wanted to be able to introduce fun features like a “downs only” mode that hides the across clues. (You can try it: running cursewordswith the --downs-only flag activates this very challenging mode.)

But also, I liked the challenge of writing my own software as a way of thinking more about how crosswords are built and how we hold them in our head for navigation. I also love the retro-computing aesthetic that comes with terminal applications, and—while I think this is the first ever terminal crossword client—that it’s mostly based on tech that has remained unchanged for decades. In that way I’ve likened it to efforts to, say, imagine what a car built with first-century technology would look like: it’s not necessarily the most useful or the best, but it’s instructive (and in my case, actually works)!

As the name may suggest, cursewords relies on a famed programming library called curses that helps to build text-based user interfaces. It is also heavily indebted to a curses wrapper called blessed, and a library called puz that reads and writes .puz files.

If you’re interested in cursewords, please give it a try and let me know how it works for you. I’ve been very interested in how to solve this problem for over a month now and I am excited to talk about it more publicly.

Shutting down @LinkArchiver, the Twitter link backup bot

After a little over a year of service, @LinkArchiver, the Twitter bot that automatically made Internet Archive backups of the links you tweeted, has archived its last link. In that time it archived somewhere around 7.2 million links total from about 9,000 users.1“Quote tweets” are treated like links to tweets, and constituted about a third of the total links. Something like 4.8 million links backed up were at domains other than Twitter. The last link it archived was this LA Times story about Verizon throttling California firefighters, tweeted on Thursday morning.

LinkArchiver stopped working this week when Twitter turned off the User Stream API that it relied on. Under the hood, LinkArchiver was only looking at its timeline, so it could use Twitter’s built-in following features to make its user list. Since that API change, it can’t pull down a “stream” of its timeline, and so would have to be redesigned to continue to work.

Even as this project is shutting down, I consider it a pretty major success. I am very grateful to Jacob Hoffman-Andrews for pitching me the underlying idea. Writing the code (and seeing it get an enthusiastic reception) was a great way to kick off my time at Recurse Center last summer. I’m also grateful to Ben Cotton who gave it a nice write-up at opensource.com when it launched.

I’ve had a few people ask me about archiving and backup options now that this is no longer available. I’m considering doing something similar for Mastodon, or for plain RSS feeds, but I also don’t want to downplay the fact that Internet Archive does a very good job of running the Wayback Machine crawler, and so the main value I can add is adding a personal layer. In any future work on things like LinkArchiver, I’d want to keep track of that.

There’s also a way to do a Twitter redesign with existing APIs, probably. Instead of getting a stream from Twitter that pinged on new tweet events, it could request new tweets at regular intervals, using an API that’s still operational. If somebody wants to write that, they’re welcome to, but given the way Twitter is, I’m not eager to do so.

   [ + ]

1. “Quote tweets” are treated like links to tweets, and constituted about a third of the total links. Something like 4.8 million links backed up were at domains other than Twitter.

Twitter should allow users to “hide” old tweets: my correct opinion

Twitter should allow users to “hide” old tweets so that they are only visible to that user, and selectively “un-hide” individual tweets from that collection so that they once again become available at their original URL, in quote tweets and threads, and in sites where they are embedded around the web.

The need for a mechanism to remove old tweets is very clear and becoming abundantly moreso with every rightwing campaign trawling for old statements to weaponize by taking out of context. Numerous guides have been written about how and why to delete your tweets, and it’s beyond the scope of the argument I’m making today. Many of these guides, too, address the emotional difficulty of actually pulling the trigger and wiping an archive—in many cases spanning a decade or so of interaction.

One of the best encapsulations of this problem (and that solution) is John Herrman’s Awl 2015 piece “Time Is a Privacy Setting.” Even just the phrase in the headline is an important contribution, and gets at why simple dichotomies of “public” and “private” are so unsatisfying. A lot more goes into our understanding of privacy than a binary toggle, and time is certainly a factor.

And just as interesting, to me, is something Herrman said this week, in a tweet thread that began with his a link to that now three-year-old essay: “You thought you were talking, but you were writing.” It’s true! Twitter has managed, for many people, to clear out a barrier that can otherwise be insurmountable, letting people be comfortable writing down their thoughts and conversations. Something the tweet-deleters flirt with, but never seem to quite put their finger on, is that writing (as opposed to talking) has some real advantages in addition to the drawbacks.1By definition, most of the articles about deleting tweets are written by writers. Writers, again by definition, tend to write more stuff down. So maybe the benefits of having a place to write stuff down if you otherwise wouldn’t isn’t quite as clear to them.

This observation is obviously not new to Twitter. My father, a writer, has always exhorted me to keep a journal or a diary, though I’ve never really been able to. One real value of Twitter, for me, has been breaking through that hesitation by setting the threshold of participation so incredibly low that I can just, y’know, write. It’d be better to keep a journal or to blog, but in many cases tweets are good enough.

Over the decade that I’ve been on Twitter, I’ve gone through periods of prolific writing and total dry spells. I’ve maintained this blog regularly at times, and at others it’s sat dormant for months. And the thing is, I frequently refer to and read the old entries for time periods where they exist! Human memory is fickle and fragile, and it is genuinely interesting and educational for me to look back on old writing and see the germs of ideas that later grew into real passions. Looking up old tweets isn’t quite as good, but it fills in the gaps.

Beyond just lowering the threshold to producing such writing, Twitter did some important things to make online writing even more valuable. Search, the villain in recent sagas, is a powerful tool when used for good. Joanne McNeil recently raised a use case I love:

I do that too! Yes, it’d be great if all my friends blogged about every movie they watched or book they read (and I had a good way of finding those posts when they became relevant to me), but this is a decent runner-up. If you want, too, you can see first interactions with people who later became important to you, or all the people who’ve made a joke you just thought of, or people discussing an article you’ve just read, and Twitter (helpfully, in this case) collapses the time gap between you and those people.

Beyond that, Twitter puts a time stamp and a URL on each of these things. In recent years it even preserves the semantics of threads. I don’t want to downplay the very real harm these attributes can cause, but it’s also true that in some situations they provide a ton of value.

Which brings us back to tweet deletion and why it is emotionally difficult for people to do. For better or for worse, this archive we’ve created of our own writing is valuable, and deleting tweets is permanent. (Emily Dreyfuss’ article at Wired does an especially good job at grappling with it.) Every guide I’ve read to deleting tweets walks through the process of creating an “archive,” but even doing so misses some of these benefits of keeping old tweets online.

Archiving-then-deleting may preserve a copy for you, but it permanently rips each tweet out of its context in threads, at a URL, within quote tweets, embedded in articles, and in the Search index.2Beyond that, there’s no easy way to present your archive on your own terms to anybody but yourself, a feature I’ve argued for in other contexts. Maybe you want to do all those things, and you should have the power to do so! But with a corpus of tens of thousands of tweets, in many cases, it is overwhelming to permanently make that decision for everything all at once.

(Side note: that permanence is in large part a result of Twitter’s centralization. Under a more decentralized model,3Check out Mastodon. where we controlled our own domains, you could maybe just rebuild your database from a backup and point all the right URLs to all the right content. There’s pros and cons to such centralization—and in particular, people seem to love the guarantee that the archive hasn’t been tampered with—but in any case that’s the situation we’re in and what can you do.)

So here’s the solution, referred to up top. Twitter should give users a new option to “hide” old tweets, which would look to every other user like deletion, but which would be a reversible operation. Your hidden tweets should still appear in your searches and views of old threads, but greyed out to indicate they’re not otherwise available. And—and this is probably the hardest part—Twitter should introduce a per-tweet configurability for these settings.

That’s the kicker. Individual tweets could be deleted (fully gone), hidden (just for you), private (for your followers, no retweets), or public. That would be a big change! But it’s important.

(Another side note: when I posted this opinion on Twitter, I got a few responses that such an option should not be available for public figures or politicians or what-have-you. I tend to think that we should not rely on Twitter to preserve these public records, and that their introduction of a feature that has real benefits to real users shouldn’t be hindered by the possible exploitation by shifty officials. And just as with deleting tweets or blocking critics, the behavior can be barred without Twitter needing to intervene.)

Twitter the company has mismanaged Twitter the product in a dozen different ways, and its policies on speech, censorship, harassment, and abuse have often been incoherent. It’s also provided an invaluable platform for many of my favorite writers and thinkers, and done so in a way that is reasonably consistent with the technology of the web.4Two good things that stand out: canonical URLs for every tweet, and the fact that unavailable embedded tweets degrade gracefully into text. On the other hand, the normalization and ubiquitous use of a URL shortener sucks. Giving users more tools—in this case a new privacy setting with more granularity—is the right next step in overcoming the former without diminishing the latter.

   [ + ]

1. By definition, most of the articles about deleting tweets are written by writers. Writers, again by definition, tend to write more stuff down. So maybe the benefits of having a place to write stuff down if you otherwise wouldn’t isn’t quite as clear to them.
2. Beyond that, there’s no easy way to present your archive on your own terms to anybody but yourself, a feature I’ve argued for in other contexts.
3. Check out Mastodon.
4. Two good things that stand out: canonical URLs for every tweet, and the fact that unavailable embedded tweets degrade gracefully into text. On the other hand, the normalization and ubiquitous use of a URL shortener sucks.

Subtitles for Norway’s SlowTV train ride to Oslo

One thing I enjoy is Norway’s public service broadcaster’s production of a train ride from Bergen to Oslo, which was first broadcast in real time, over seven or so hours, in 2009. It’s predictably pretty quiet stuff, but—at least now that it’s on Netflix—there are in fact subtitles of what little dialog there is.

Netflix makes it pretty straightforward to extract the subtitles from a given program, and it stores them according to a very fun standard called the Timed Text Markup Language, or TTML, which just missed adoption by the WHATWG in favor of a lighter-weight, less-XML spec called WebVTT.

Anyway, I pulled out the (very spare) subtitles in that format and wanted to convert them to something a little more usable. So first I converted them to JSON, and produced an array with an object for every subtitle, and then processed it a little further and created a version where “adjacent” subtitles are combined into single objects.

The result is nearly as hypnotic as the original video:


    {
        "begin": "00:17:21",
        "end": "00:17:23",
        "text": [
            "[metallic clang]"
        ]
    },
    {
        "begin": "00:19:40",
        "end": "00:19:44",
        "text": [
            "[indistinct conversation / woman laughs]"
        ]
    },
    {
        "begin": "00:19:52",
        "end": "00:19:54",
        "text": [
            "[woman laughs]"
        ]
    },
    {
        "begin": "00:19:57",
        "end": "00:20:01",
        "text": [
            "[indistinct conversation]"
        ]
    },
    {
        "begin": "00:21:47",
        "end": "00:21:48",
        "text": [
            "[metallic clang]"
        ]
    },
    {
        "begin": "00:22:10",
        "end": "00:22:12",
        "text": [
            "[metallic clang]"
        ]
}

Archiving threatened outlets for the Freedom of the Press Foundation

A little over a month ago, I launched one of the Special Projects I’ve been working on at my new job at the Freedom of the Press Foundation. The Threatened Outlets collection at Archive-It aims to capture the archives of news sites that we deem vulnerable to “the billionaire problem,” wherein wealthy individuals or organizations can eliminate unflattering coverage through litigation or by purchasing media companies altogether. From the launch announcement:

Those efforts help individual journalists. But another important thing we can do to reduce the effectiveness of this kind of attack on press freedom is to commit ourselves to the wholesale preservation of threatened sites.

In this case, we seek to reduce the “upside” for wealthy individuals and organizations who would eliminate embarrassing or unflattering coverage by purchasing outlets outright. In other words, we hope that sites that can’t simply be made to disappear will show some immunity to the billionaire problem.

To my surprise and delight, our launch received some nice and encouraging coverage, in WIRED, TechCrunch, The Next Web, The New York Times, and others. Sometimes when you’ve got your head down on something that feels like a niche topic, it’s nice to come up for air and realize that the general public is interested, too.