S

P

O

I

L

E

R

S

below!

I’ve wanted to make this puzzle for a very long time, but I couldn’t figure out any way to make the theme something that could run in a mainstream publcation. Finally I decided that this was how I wanted to do it and I’d just share it myself, which feels a bit exciting. I’ve long admired the crossword bloggers of the world; maybe now I can start to count myself among them.

The song at the heart of this puzzle is very beautiful, and if you haven’t heard it I recommend listening in both English and Portuguese. I’ve always loved the version by Art Garfunkel and had a very special experience one time seeing Sondre Lerche perform it live. (That’s not the exact performance I saw but I think it was the same tour.) For a laugh, the 1980s Coca-Cola jingle version of it is charmingly insane.

Making the verse into a theme felt like a small act of translation, and it’s always been interesting to me that Jobim wrote both sets of lyrics himself. It’s especially interesting that his translation was also a localization: the English lyrics are written for audiences in the northen hemisphere, who may associate March with rebirth and springtime. In English, it’s about the end of frost and new life, and the images take on a new meaning.

I’ve had occasion to think a lot this year about endings and beginnings, and it felt good to engage with a work that addresses them so beautifully. I hope you enjoyed the puzzle.

See an example grid creation:

We got started on Fill Harmonics towards the end of my batch at Recurse Center. From a tech perspective, it was a fun opportunity to continue building in Javascript, and combine some recent areas of interest: in particular, the CSS grid data structure work I’d done for Malaika’s personal page and the web audio libraries I’d explored for my infinite sea shanty. Similarly, this one is written entirely in vanilla Javascript, though I think at some point it exceeded the complexity where that was a good idea.

Natan and I are both fans of Max Neely-Cohen’s 10,000 Drum Machines project, and we wanted to build something that could be a part of it. Our shared interest in crosswords was a natural starting point, and we pretty quickly landed on the overlap between sequencing instruments and crossword aesthetics.

As we continued to build, we took that early concept and pushed it out in some fun directions. I think the combination of synthesis for melody and sampling for the drums works really well and allows for pretty elaborate musical phrases to emerge from simple components.

Perhaps the most fun addition was “word play mode,” which changes the sequencer loop from operating simultaneously across each row to instead looping over every “across entry” in the grid. (It makes a little more sense when you see it in action, but it’s still a little brain-bendy.) That mode emerged out of a conversation with fellow Recurser Gage Krause, who is working on some extremely cool and polished drum machine stuff right now.

The musical effect of word mode is called polymeter, a term that lives in the shadow of its more famous cousin polyrhythm. Polyrhythm takes a measure and divides it with two or more sets of even subdivisions; polymeter, by contrast, keeps each beat the same size but loops them across measures of different lengths. The neat thing about it is that you can easily create phased patterns that only repeat at their least common multiple number of beats.

Keeping the tempo the same across grid mode and word mode means that you can alternate between the two for a neat musical effect:

One other early musical decision that paid off nicely was making the grid always square and configurable between 8 and 16 units wide. That means that the biggest and smallest grids feel like pretty comfortable 4/4 patterns that are easy to alternate between.

Because the grid is always square, and expanding it adds lower notes to the bottom, you can grab the slider for a cool effect that cuts the loop in half and drops the bass out entirely.

Please send along any sounds you make with Fill Harmonics! We’ve had so much fun making this thing and I can’t wait to see it in other people’s hands.

In my case, there’s a build-and-deploy workflow that’s triggered on every new push to main. Then there’s also a fetch-new-data workflow that runs on a cron schedule and checks an external data source for updates. (Conveniently, that external data source can be a Google Sheet.) If there is new data, it’s formatted correctly and committed to the right place in the repo.

At first I thought that would be the whole story, because the new commit would automatically kick off a new build process. That didn’t happen. It turns out GitHub Actions has a (reasonable!) security feature that prevents tasks performed using the repository’s GITHUB_TOKEN from setting off new workflow runs, in order to prevent accidental infinite loops. My fetch-new-data workflow included a very default set-up of the actions/checkout Action, which uses that default repository token, so I was affected.

As far as I can tell, there are two approaches to fixing this. One would be to reconfigure the fetch-new-data workflow to not use the default token. Generate a personal access token with the appropriate permissions, and tell the checkout action to use that instead. I didn’t go that route, but I think that’s what this tutorial describes.

I opted instead to have the fetch-new-data workflow explicitly call the build-and-deploy workflow if there was a change. I reconfigured the build-and-deploy workflow to be reusable with an on: workflow_call:, added the uses block to my calling workflow, and sat back and waited for it to work.

Once again, it did not. I could get it to trigger the build step, but the deployed site did not reflect the latest change. How annoying! I thought maybe it was some kind of race condition where the new commit hadn’t fully propagated, or there was some kind of caching involved. I tried a lot of silly and complicated things to address those two perceived problem areas.

As it turns out, I was close, and the answer is simple. In my case, the build-and-deploy workflow was also using a very default set-up of the checkout Action. If you don’t provide an explicit ref there, it falls back to whatever the current GITHUB_SHA value is. In the case of a workflow_dispatch/workflow_call setup, the called workflow uses the GITHUB_SHA of the calling workflow. The calling workflow, in turn, uses the ref of the state of the repo when it ran, which did not include the commit it performed.

So the fix was to set an explicit value for ref in the checkout step of the called workflow. I think hard-coding main probably works in almost all cases, but I used github.ref_name for just a touch more class.

I’m using this in two places now. One is in the repo for this very site, where a script that fetches my most recent movies, books, and concerts from various data sources can update a file here that gets incorporated into a Jekyll include. That info is now on my home page, but the “embed” here should also be live data, which is fun.

The data sources for those are my Letterboxd account’s RSS feed, my Goodreads account’s RSS feed, and a Google Doc of concerts that I maintain, but it’s neat to know that I could easily point the script somewhere else if I ever wanted to move away from those sites.

The other is on Malaika’s personal site, where changes in her spreadsheet trigger new builds with updated calendar data.

Writing this out, I realize that there are a zillion ways to solve this problem. Most of them I discarded not because they wouldn’t work, but because they felt inelegant. I could’ve just used a personal access token, but that felt dicey on the least privilege principle. I could’ve just copied and pasted the build workflow steps into the fetch workflow, but that felt like it was unnecessarily repetitive. I’m happy to have landed on a solution where everything works like it feels like it should.

I had a good conversation with some fellow Recursers at about the halfway point of my batch, where we lamented how easy it is to feel like you’re “not getting enough done,” especially because the opportunity to focus on your own projects for 6 or 12 weeks is rare and feels precious. That is true, but it’s missing some critical context. In my opinion, the main project I’ve worked on for each of my two full batches was myself, as a programmer, cultivating my expertise and preferences and experiences.

It’s hard to account for that project, but any reckoning that doesn’t include it is incomplete. Sometimes it presents in more straightforward ways—like in my first batch I committed to using a real text editor, which was new to me and slowed things down, but has paid dividends so many times over—but often it’s intangible improvements to how I think about technology and problem-solving.

That said, I did ship some projects that I’d like to share!

Signal bot for Puzzmo leaderboards

I wanted to write some Go during my batch, which helped shape this first project: a server that processes Puzzmo webhooks containing daily leaderboard updates and sends them to a Signal group. It was fun to dip in to a project that used Tailscale’s tsnet library, and it was also nice to start immediately getting daily benefits of code in production.

At first I felt a little self-conscious that this was not really packaged for public consumption, but that faded. I came across another Recurser’s blog post about “houseplant programming”, a concept that I have come to really love. This was a real houseplant programming job for me, and it made my life a little better while improving my understanding of things like webhooks and Go and Tailscale and Docker and Signal and sysadmin stuff. Hopefully it serves as “an ad to write tiny software just for yourself.”

Surfing the Bluesky Jetstream with Helping Friendly Bot

When I did my first Recurse Center batch in 2017, social media bots—which mostly just meant Twitter bots—were a primary medium for me. In the years since, I’ve watched Twitter (and its API) get destroyed. It’s forced me to think about how to continue to apply the lessons I’d learned and the practices I’d developed in a new context.

So it was especially fun to work on a Bluesky project that would not have been possible with Twitter. This project involved expanding my pre-existing Helping Friendly Bot, which posts context data about Phish songs as the band plays them on-stage, to listen for real-time Bluesky events and do a bunch of finicky replying and reposting stuff.

Beyond the Bluesky and AT Protocol specific parts, I learned a lot about websockets, shell-scripting, job scheduling with the at utility, and more sysadmin stuff.

Personal sites for friends: Calendar of Song and Malaika’s crossword chart

I knew about myself that I love websites and I love helping people publish websites that speak to them. These two projects really worked in that vein. For Michael, I updated a long-running project to each day refresh a page on the personal site I’d previously built for him. For Malaika, I built her a brand new portfolio page that displays her (many, many) publications on a GitHub-style contributions graph.

It’s fun working to a friend’s specifications, and it enforces some discipline about where you won’t cut corners. I think I built things a little nicer for my friends than I would have for myself, and I appreciate the opportunity to show some craftsmanship. I also discovered about myself that I like writing Javascript that runs in the browser! That has not always been the case, but I’ve grown and so has the language and it’s gotten me started on a few projects that will ship in the future.

Maintenance on xword-dl and others

I had told myself coming in to this batch that I would not let myself fall back on Python throughout. I’ve spent much more time writing in Python than in any other language, and it’s fast and comfortable for me. But there are definite benefits to embracing a diversity of languages and approaches, and building the skills and confidence to be able to choose the right tool, not just the one closest at hand.

So it was with some trepidation that I undertook some maintenance on my xword-dl crossword scraping utility—would I be learning, or just doing chores?

That fear was misplaced. There’s so much to learn, and there is a distinct value to continuing to work on a codebase over the course of years. I’ve now watched this project grow and develop as I have grown and developed, and I learned a lot about newer Python features, development best practices, and packaging stuff.

Infinite sea shanty in the browser

A corner of a larger project that I’m still working on, this generative sea shanty was a fun opportunity to catch up on web audio and dip my toe back into the world of generative composition and music theory. I used to be a musician and I have always loved theory, and to be honest I kind of forget that it is a body of knowledge I have spent thousands of hours working on.

(Do you ever do that? Write off skills you have as trivial to fixate on the skills that you don’t?)

So in that way, this was a fun exercise on the first order, and then it was also fun to figure out how to embed it into a (very lightly) interactive blog post, which was new for me. I also found a bug in the library I was using, reported it, and got it fixed! Between the original composition and the post about it, I learned a lot about Javascript, Tone.js, web audio, git and GitHub, and even some music theory. I also got to listen to a lot of shanties.

You should probably go to Recurse Center

This list of projects isn’t complete, because there are a few things I’m still working on and want to keep cooking for a little bit longer. I’ve come to appreciate that Recurse Center is something of a mindset, and that I can chase down those projects and the curiosity that inspires them even as I re-enter the world.

This has been a really hard year for me, in a lot of ways. That has forced me into introspection, and trying to figure out what parts of myself are load-bearing, and how I want to build around them.

Alongside that project, I have been so grateful to have a space and a community like Recurse Center that encourages the pursuit of self-improvement. Obviously at RC, the focus is becoming a better programmer. Developing the volitional skills and the discipline to learn through building at the edge of your abilities, though, helps across the board.

Hands-down, though, the best thing about Recurse Center is the community of Recursers. People that I’ve met during batches have been hugely helpful and inspirational to me, and the alumni network is a constant source of wonder. I am grateful to have met so many of these people, and I want to meet more of them, and I want to learn generously and inspire others the way I’ve been inspired.

I know it is an incredible privilege to be able to focus on that for months at a time. But if you can swing it, I encourage you to apply to do a batch.

I’ve had a longstanding interest in sea shanties, which made their sudden spike in popularity in 2021 first exciting and then somewhat annoying. One nice side effect, though, was for a few weeks there everybody who writes or makes videos about music stuff had to talk about shanties, and it produced a lot of good tutorials and discussions that helped me in this effort.¹

With generative music, the success or failure of the “composition” really comes down to the actual rules you put in place. There’s a narrow path that’s fun and interesting, and danger on both sides. If the rules are too strict, it doesn’t feel generative. Sometimes this looks like overfitting to a particular creative model, which can make variations feel meaningless, leading to an oatmeal problem. But if the rules are too lax, it doesn’t feel composed. You end up making a wind chime. Great if that’s what you’re aiming for, but not what I wanted.

Similarly, when composing in a genre you encounter some of the same overfit/underfit dynamic. If you lean too much into the scène à faire elements, the result feels derivative; but too little and it doesn’t successfully evoke the genre at all.

With that in mind, embedded below is a working draft of my infinite shanty, and here’s the code that powers it. You can toggle each component individually if you want to hear a given instrument’s part more clearly.

Musical thoughts

In order to make a thing that sounded like a shanty, I outlined some musical constraints that I could represent in the code.

Time considerations

Because of the strong historical connection between shanties and repetitive, rhythmic maritime labor tasks, time plays a major role in creating that distinctive sound. There’s a heavy emphasis on the downbeat, usually some “swing” (which means that the emphasized notes have a little longer duration), and often there are grace notes leading in to the start of phrases.

A lot of writing online connects shanties with 6/8 time in particular. (6/8 means that you can count six beats in a measure, usually as 1-2-3 1-2-3.) And there are definitely some notable shanties in 6: Blow The Man Down is a traditional one, or Fathoms Below from the Little Mermaid is a modern example. I think it’s a bit of an oversimplification because lots and lots of shanties are not in 6, but in any case it is the sound I wanted here, so this one is in 6.

And to get more specific: many shanties have a musical connection to Irish traditional music, and can be described as a “jig” (which is usually in 6/8) or “hornpipe” (which is usually in 4/4). I wanted the jig sound, and in particular I wanted a “double jig,” which is characterized by notes on each of the 1-2-3 beats. Maybe the most famous traditional Irish double jig is The Irish Washerwoman, which is not a shanty but has some melodic properties I wanted to emulate.

To get that effect, the timing of the melody I wrote looks like this in the (pretty intuitive!) notation format of the library I used, Scribbletune:

[xxx][x-x] [xxx][x-x] [xxx][x-x] [xxx][x__]

Melody and sound

That is the rhythm of the melody, and the note values vary with each loop. The Scribbletune library allows me to specify particular notes for some beats, and to pull randomly from a specified set for others. In order to give it a classic shanty feel, I used some preset phrases and pull the random notes from a scale in the Dorian mode.

Musical modes are a super interesting way to evoke a particular vibe, and I feel like they’re not well understood by non-musicians. A very simple description is that they determine which notes are available to use in relation to the “tonic” note. For example, this piece uses the D Dorian scale, which, like C Major, is composed of the white piano keys: no sharps or flats. To distinguish this melody from one in C Major we have to establish that the D is our home base. Here, I did that with a droning bassline that just sits on a D note, except for a little flourish every 8 bars, and by populating the melody with some phrases that resolve to D.

Aesthetically, I think this piece is more interesting with synthesized instruments instead of sampled ones, so that’s what I’ve used. I’d initially wanted the melody to be on some kind of wind instrument, but I couldn’t get it to sound good, so I switched to a synthesized steelpan as a nod to some Caribbean sound. Shanties are typically a cappella or close to it, and they have lyrics, so these decisions were really just me vibing. Beyond the melody, there’s the simple bass, and two more percussive instruments that really just keep the rhythm going.

Technical details

It’s possible to do web audio stuff using just vanilla javascript, but it pretty quickly becomes a headache to do anything musical. There’s a framework called Tone.js that is hugely helpful in giving you abstractions like “notes” and “measures” and “instruments,” and my first draft of this used Tone.js directly.

But I found that I was still missing some useful abstractions related to composition, and so I turned to Scribbletune, a javascript library geared towards musical ideation (that uses Tone.js for its browser support). I think in practice that’s mostly for electronic music — it’s possible this is the first Scribbletune shanty. Beyond that, I think my usage pattern, where the Scribbletune playback in the browser is the actual end product, is a little unusual.

So I hit some rough edges of the library but I’m glad I persevered. For one thing, it has no configurable time signature, so I’ve had to sort of fake it by grouping my beats into triplets. This is where I think its roots in electronic music come through: not a lot of EDM uses anything but 4/4.

One thing I love about music theory is that it’s very empirical but still requires a lot of judgment calls. The time signature of a given piece, especially a traditional one, can be subject to some debate, and there are usually multiple ways of representing the same underlying information that differ mostly in terms of what mental model is being applied. In that way, it kind of reminds me of duck typing. It is useful to be able to say: okay, the software may not have a concept of “time signature,” but the property I’m looking for is two groupings of three beats, which I was able to achieve with the triplet function.

Beyond that, the randomness is not very configurable, and I think I found a bug how it’s documented. I think it accidentally made the piece more interesting, so I can’t be too mad.

My goal when starting this project was to make something interesting enough to give your attention for a bit, but ambient enough to fade into the background as necessary. I think I’ve done that, and I’ve learned a good deal about both song composition and javascript along the way.

Why would you even want to fork your own repo? Here’s my situation. In my last blog post, I embedded a small page with some javascript to play the audio I was describing. I’m actively working on that code, and to be able to look at it remotely or share it with friends, I’ve got a small Action that builds the single page and pushes to GitHub Pages.

I could’ve just embedded the resulting page, but it’s under active development and I don’t want to accidentally break something in the blog post. Really, I want a snapshot — but I also want to be able to deliberately pull in changes to that snapshot as necessary.

Usually that would mean you could just make a branch. But in this case I wanted a Pages environment, which is allotted at the repo level.

So instead, I cloned the repository, pushed it to a new GitHub repo, and then used git remote add to add back the Git URL of the original repo as a second remote. That preserves the shared history, and I can even commingle commits. You don’t quite get the interface sugar of pull requests across the two repos, but you can pull upstream changes into a downstream branch and open a PR that way.

One other option I considered: You can fork your own repo into one owned by an organization, so if I were doing this workflow a lot, maybe I’d make an organization that just owns blog-embedding-related repos. For now, though, the duplicated repo did the trick.

I built her that site, and we published it yesterday. I’m really proud of how it turned out, and want to share some thoughts about how it came together.

Programmers in the audience will likely recognize the inspiration immediately. She wanted to display her data in the style of GitHub’s “contributions calendar,” which is displayed by default on every user’s profile. (See my profile, for an example.)

That chart displays days as cells in a 7x53 grid, and colors each cell in a shade of green that depends on how many “contributions” the user made on that day. As a format for visualizing software development data, it has been wildly influential, and that influence has come with lots of backlash. It’s kind of a classic GitHub issue: something they thought was a straightforward measure of some objective metric turned out to be deeply opinionated, and even political, in terms of the values it promotes and actions it encourages. Ah, well.

Some of the criticism boils down to its flattening tendency. By first creating a limited definition of a “contribution,” and then equating all of them in a quantity-only measure, it strips the character of some contributions and outright ignores others. (I can pretend to be above it, but this project began life as a major pull request before being split out into its own repo. The commits for the former were not shown on my graph, and I was annoyed!)

Part of our design spec aimed to avoid that flattening. We wanted to create a customizable set of rules for cell colors, beyond just buckets of counts: Being able to visually distinguish days with entries in a certain category — such as “full-size” 15×15 puzzles — was much more important to us than highlighting days with two or three puzzles published.

The means that, despite the visual similarity, our implementation diverged meaningfully from the inspiration. The general term for the kind of chart GitHub makes is a “calendar heatmap,” and there are some robust libraries for creating them. But given that divergence, and because it would be more fun to build this from scratch, I didn’t use any of them. Instead, the calendar is rendered with a dependency-free vanilla javascript module.

In fact, the first version was one big blob of javascript that did everything by itself. When I shared it with some other folks at the Recurse Center, I got the good feedback that it might be fun to generalize a bit for possible reuse. (I got maybe halfway there in time for launch.) I did realize that moving more configuration from hardcoded properties to initialization parameters would make it easier for Malaika to update and change it in the future, so the calendar, the details panel, and the filters are each separately defined and given specifications in the page’s script tag. Those specifications include the coloring rules, the format of the tooltip presentation, the columns to filter by, that kind of thing.

That pattern of moving configuration up a level of abstraction is one I repeated throughout the development, and eventually I moved a bunch of parameters into a repo-level configuration file. The text on the page, the site’s metadata, and even the theme colors are defined in one file, so she can make updates without having to get into the weeds of the site. Once I’d done that, I was able to ship a bunch of little quality-of-life improvements: the site’s favicon and Open Graph image are generated at build time using the theme colors, for example.

Keen-eyed readers may have put together that Malaika is the “sophisticated programmer” I mentioned in my post last week about Google Sheets as a database. The source of truth for the site is a Sheet she has kept updated for years, which she maintains with a combination of manual data entry and Apps Script automation. Every few hours, a GitHub Action pulls the live data from the sheet, commits if there have been any changes, and then kicks off a new site build. GitHub Actions are famously sort of finicky, and I ran into an issue with chaining workflows, but otherwise it was pretty smooth sailing.

What she didn’t realize when she mentioned the project to me was that this format was a little personal. Years ago, before I had started my programming journey, I had received a cache of time series data in response to a public records request. It showed that a particular surveillance practice had steadily become much more frequent over the period in question. I had this gut feeling that a calendar heatmap would be the best way to display it, but I didn’t know how — and before I could figure it out, the data had become stale and we never did anything with it.

I regret that I never got to do that project, but I’m glad to have gotten a chance to implement something similar. And the site is, I think, quite beautiful.

I’ve helped Michael build and maintain his site over the years because I love reading his writing, and the short captions that accompany these songs are no different. And besides, he pulls from such a wide variety of music, the playlist itself is a great tour across genres, styles, and traditions.

Since we deployed the Calendar of Song, I’ve added it to my phone’s home screen, and tune in each day when I’ve got a quiet moment. It’s been nice.

His statement really captures the project better than I can, but I want to drill down just a bit on his use of the word “site-specific,” which is (both) a great pun and a pithy distillation of the some of the motivation behind my implementation decisions.

The first iteration of the Calendar of Song was (what else?) a Twitter bot, where its atomic unit was an individual post, in a format designed to obliterate context. For the viewer, a Calendar of Song update in that era appeared under a headline, over a shitpost, between selfies and memes and reaction gifs. As the algorithmic timeline gained purchase, it lost even the excuse of simultaneity.

Today’s Calendar of Song is a different beast altogether. Like a tear-off desk calendar, the only context is: today. There is no “previous post” or “see more” or “index,” just the song Michael has picked, and a few words about it. It was fun to build to that spec, especially within the constraints of his (static) site and the delightfully janky CMS I’ve set up for him.

You might describe it as eschewing the promise of the massively parallel for a more serial experience. In that way, from a user experience perspective, it’s as influenced by Wordle as it is the more erudite art antecedents that Michael highlights.

Anyway, go give it a listen. And if you like what you hear, see you again tomorrow.

One good example project is Daily Crossword Links, a newsletter started by Matthew Gritzmacher to compile, as its name suggests, links to each day’s published crosswords. The newsletter always includes mainstream and independent outlets (down to individual bloggers). When Matthew and I first spoke about it, he was putting it together each day by hand, checking and copying text from dozens of sites while referring to a Google Sheet of puzzle sources that he’d made. We started from that Google Sheet, and I was able to automate a lot of that process, ultimately producing a draft each day that he (and eventually other editors) could clean up for publication—and to this day, the source of truth for what goes into those drafts is a Sheet.

It’s a recipe I’ve followed a bunch of times now, and when I’m discussing it with collaborators, I usually come back to the same list of benefits.

• Everybody already has an account, and they are comfortable with the collaboration workflows. That doesn’t mean they’re the ideal workflows for any given situation, or even that they’re very good, but at least they’re familiar. So when the Daily Crossword Links project expanded from one editor to a handful of volunteers, they were able to manage this through invitations to the spreadsheet, without me having to build any new auth or permissions model. Besides collaboration, they also have pretty robust version controls, and other mechanisms in place to prevent data loss.
• Spreadsheets themselves are really powerful tools, including for automation. The crossword newsletter crew now has a bunch of functions to fill in dynamic dates and titles and things like that, and they don’t need “developer time” to expand those. On another project I’m working on, the proprietor (who is a much more sophisticated programmer than I am!) wrote a Google App Script to update her data. And even for less technical users, I can write formulas that are less intimidating—and provide real-time feedback—than the equivalent Python would be.
• There are clients for every device, which means editing is always going to be more-or-less reasonable. Editing a spreadsheet from a cell phone is not great, but it’s not an experience I’m shoving down their throat. Similarly, people know how to read a spreadsheet, and it’s much more inviting than an equivalent table of data.
• Spreadsheets also enforce some data discipline in ways that don’t feel overbearing. With most projects I’m immediately converting the data into JSON or YAML, which to their credit are mostly readable (if intimidating). Creating a new object in either format, though, requires cutting and pasting and sometimes interacting with the vagaries of the specs. On the other hand, a new row in a spreadsheet has labeled columns and example data right next to it. Not to mention, conditional formatting can even highlight missing or malformed data if necessary. All of this makes the upgrade path out of a Sheet clear, if need be; and in most cases, a CSV export would work as a drop-in replacement, in a pinch.
• Combining those last two points: One pattern I’ve used successfully is rigging up a Google Sheets “database” to a Form, which is a totally familiar interface that can do all kinds of validation and provide default values.

There are of course drawbacks. To make one obvious point, a spreadsheet is not really the right data structure for some kinds of projects. Text formatting is a bit of a bear to deal with, and if you need more than two dimensions of data (including nesting or any kind of relational component), things get very hacky fast. (It helps that you can have multiple sheets within a Sheet, and they can refer to each other in formulas.)

And of course, a lot of these advantages would be moot if there weren’t good tooling. For my Python projects, I’ve had a great experience with gspread for both reading and writing data. I haven’t really had to look into what the ecosystem is like elsewhere, but I expect that this many years into Google Sheets there’s pretty good support everywhere.

From a practical standpoint, the existence of good tooling like that means that my exposure to Google Sheets is pretty minimal: I hit the API to download live data, and in some cases to make updates, but the data otherwise feels pretty native. If I ever had to migrate a particular project away, it would be disruptive for my collaborators learning a new workflow, but it would be pretty easy for me to swap out gspread for a regular parser of a standard data format.

In a nutshell, I’m using Docker Compose to spin up this containerized version of signal-cli that exposes a REST API, with Tailscale in a sidecar container for its network mode. From the (cloud) machine where it’s running, the API is accessible at 127.0.0.1:8080, and from my other tailnet devices, it’s at the host signal. From a security perspective this is especially neat, because its ensures that all of the otherwise unencrypted requests are going through the Wireguard tunnels that Tailscale sets up.

The end result is I can send Signal messages with curl or requests or whatever, from any of my devices. If I want to be really slick, I can use just the hostname, for a command like:

curl -X 'GET' 'signal/v1/accounts'

The container offering the API appears as a “machine” in my Tailscale admin console, and access is controlled with my ACLs.

Here’s my full Docker Compose file. A lot of this is pretty standard for a Tailscale sidecar, as seen in the big guide blog post, but I will get into some fun parts below.

---
services:
  ts-signal:
    image: tailscale/tailscale:latest
    container_name: ts-signal
    hostname: signal
    environment:
      - TS_AUTHKEY=tskey-client-n0tr3aLLy474115c413k3y101?ephemeral=false
      - TS_EXTRA_ARGS=--advertise-tags=tag:server
      - TS_STATE_DIR=/var/lib/tailscale
      - TS_AUTH_ONCE=true
    volumes:
      - ${PWD}/state:/var/lib/tailscale
      - ${PWD}/config:/config
    devices:
      - /dev/net/tun:/dev/net/tun
    ports:
      - 127.0.0.1:8080:80
    cap_add:
      - net_admin
      - sys_module
    restart: unless-stopped
  signal-api:
    image: bbernhard/signal-cli-rest-api
    container_name: signal-api
    network_mode: service:ts-signal
    depends_on:
      - ts-signal
    volumes:
      - /path/to/your/.local/share/signal-cli:/home/.local/share/signal-cli
    environment:
      - MODE=native
      - PORT=80
    restart: unless-stopped

volumes:
  signal-api:
    driver: local
  ts-signal:
    driver: local

Before we go through the fun parts, let me lay out the end state I was working towards:

• for compatibility reasons, I want the API to be available from the host machine itself at localhost:8080
• for external access, I want the API to be available over Tailscale at a memorable domain (without requiring a port number)
• I want the API to “play nice” with an existing signal-cli install. I almost didn’t dare to dream that I’d be able to use my existing config, but (spoiler alert) I think that’s possible

I was able to get all those things. Stepping through the interesting portions:

      - TS_AUTHKEY=tskey-client-n0tr3aLLy474115c413k3y101?ephemeral=false
      - TS_EXTRA_ARGS=--advertise-tags=tag:server
      - TS_STATE_DIR=/var/lib/tailscale
      - TS_AUTH_ONCE=true

These are all magic Tailscale environment variables. The one thing I’ll note is that I’m using an OAuth client secret here instead of an authkey so I don’t have to worry about cycling it. That in turn requires the --advertise-tags bit is passed to the TS_EXTRA_ARGS.

(Update 2025-04-25: It turns out I had a misunderstanding about the relationship between Tailscale OAuth clients and auth keys and the struck-out portion above is not exactly correct. The TS_AUTHKEY value is used when creating the node and on initial authorization, but then the machine’s state is preserved and its expiry can be disabled. That’s why we map the container’s state directory to the local machine, and as long as we keep it there, it’s fine to just use an auth key as the AUTHKEY. If you do use a client secret though, it is still the case that you have to provide the tags argument.)

ports:
      - 127.0.0.1:8080:80
#and later, under the signal-api service:
environment:
      - MODE=native
      - PORT=80

This was my tricky approach to making the API available locally at :8080 and remotely without a port number. The PORT variable for signal-api makes it available at :80, which is the standard unencrypted HTTP port; the Tailscale network_mode makes it so that you can hit that container, and its port 80, through normal Tailscale addressing. And because the container is just a machine on the tailnet, you can grant or restrict access to it through your normal ACL configuration.

I thought I was going to use tailscale serve for this, but it kept not working exactly the way I wanted. I think it has to do with the fact that the Signal API container serves (a 404) on port 80 even when it’s using a higher port for the main stuff, but that’s kind of speculation. In any case, this ended up working great. If you skipped the Tailscale container ports line, the API is just available at the Tailscale address, which is not a bad outcome either.

One other note on the ports field: a lot of docs I saw for the Signal API container suggested configurations like 8080:8080, which maps the container’s 8080 to the host machine’s 8080. I would guess most users actually want to bind that to a specific IP address! Without an additional specification, it is set to 0.0.0.0: available on all interfaces. Maybe fine for a network of only trusted devices but I don’t know, man. Bind your ports.

volumes:
      - /path/to/your/.local/share/signal-cli:/home/.local/share/signal-cli

This one surprised me at first, but: as far as I can tell, you can really point a host and container signal-cli at the same config directory and they’ll share state. So if you’re already configured, this plays nicely and just gives you more powers. If you want to keep them separate, then point this volume elsewhere. One small gotcha: by default the config directory is at $HOME/.local/share/signal-cli, but Docker Compose doesn’t expand the $HOME, so you may want to hard-code the path in there.

With this set-up, I can keep the Signal factors separate from my other deployment considerations. For little things like my server to send Puzzmo results to a Signal group, or the personalized messages from my Helping Friendly Bot, it’s so much simpler to be able to put those anywhere I can make a network request, and manage all of my Signal stuff in one place.