live migrations
(an accounting parable in three parts; part one of three)1
In the winter, the birds come in their thousands. On the trash heaps, you can see the storks perch; off the southernmost tip of the Mumbai coastline, in November, you'll see a patch of flamingos, floating like dirty-colored effluent on the waves. We're so used to nasty things on city shores that it takes a moment to notice that these things are alive, alive, alive on the waves. Too young to be pink yet.
When a company wants to move where its services are hosted, here is what we wish would happen:
And here is what it feels like when one actually does it:
(artistic depiction, to be later replaced with lovingly hand-drawn diagrams)
I feel the need to take a moment to marvel that this sort of thing – the movement of massive virtualized machines from host to host, across companies and continents, with the expectation of minimal disruption of service – is possible in the first place. A moment of silence to experience the awe of modern day cloud computing.
Okay cool, let's move on.
Here's the problem:
- We need to keep serving.
- We also need to ensure data integrity.
- We might need to stop halfway through, and so as much as we can, the "halfway through" state also needs to keep serving and not compromise data integrity.
- And, well, something might go wrong, so we need a fallback plan.
You might recognize three of these four as together forming the CAP theorem (famously, "pick any two")2. As for the fourth, well, that just says you also need to be able to do it all backwards.
Fortunately, at least in principle, this is very simple to solve for!
You just need to figure out which two things you want to pick for each step. And then you need to break down the steps until they are small enough that these compromises don't hurt.
(Beware, however, The Danger: more steps means more places things can go wrong.)
First, an audit:
Consider our service: a social media app for birdwatchers to log sightings of birds, tagged with a GPS location and short messages. These "tweets", if you will, are sent out to other birdwatchers in your area, so they can keep an eye out for the same species – or perhaps the same specimen! – of bird.
Get to know your services
- API serves the API request made by the app and the website. Straightforward, cleanly. Deployed as a Docker container.
- Frontend is the website. You get a Github repo when they tell you about this, as is only right. It's served by the load balancer straightforwardly, but you have a tiny little service that tracks user sessions. It's mostly redundant with the app code, really. Probably entirely redundant. Should have been deleted eight months ago, but there was this whole feature push, and then lots happened. Anyway, they're working on it.
- Inside the API code, you have a Terraform script that spins up a Memcached. It's only for testing.
- Production does use Redis, though, to keep track of GPS coordinates. This is how the API decides which collection of tweets to serve to which user. It's quite a clever use of GIS. The logic for it all lives in API, where you note approvingly that the Redis endpoint is configured via an environment variable – and then, less approvingly, note that the environment variable is overridden in the
src/config/vars.go, ifIS_DEVis set to true. A search forIS_DEVturns up no other usage. - There's a photo bucket cached at edges, which the CTO tells you is informed by the Redis GIS data but the logic for which will take you 3 weeks to get somebody to show you where and how. The backing store is an object store. This sort of thing is, if not S3, an S3-like that "speaks" the S3 API. In our case let's say it's just S3.
- bg is a server running in a virtual machine with a homebrewed Github actions deployment pipeline and whose IP address is hardcoded in a dozen places in the code. It calculates thumbnails, recognizes birds, runs the recommender algorithm, and also refers to the Redis. (This is deemed okay and safe because bg doesn't write to any of the production stores unless the call to it is about production data.) You are seized at 11:30PM one Tuesday night with primordial terror, and wake up to check whether Redis is the only place where GIS data is written. You note with relief that of course it's also written to the Postgres database, and are vaguely ashamed for having needed to check. These are competent people. These are your peers. These people have been here since the beginning of this company, and could maybe just buy you? The person, wholesale? – considering the equity they've accrued. You should have more respect.
- There's batch, which is importantly different from bg in that it primarily exists to update the bird-recognition models. It downloads and uploads massive corpora of photo data from the photo bucket on a daily basis – seemingly over the public internet. You breathe deep.
- Two weeks in, you discover the secrets database. This one you've prepared for. You click back out, run a script to grep for every secret-key-like string in every repository you've been given access to, and save the results to a file. You have already decided that outside of this measure, you will avoid the rest of the question for as long as you possibly can.
- And of course, there's the database. Seven copies of three databases, in fact; all on separate database instances, of course, for sanitation reasons. Which for which? Some are for testing or staging environments; for others, we create dummy databases in the build scripts, because they're small and we don't need to do UAT (User Acceptance Testing, otherwise known as it's so hard to fake convincing userdata that we just renamed you Oogle Boogle living at 1234 Fivesix Lane in our testing environment and called it a day) on them. Each in its dignity is possessed of a storied history of schema migrations; there is real design thought put into the shape of their entities and relations. Too bad they all use the default RDS postgres config, but that's easily enough fixed.
And there we have it: our rogues' gallery.
What are your SLOs?
What are your service obligations? What's going to make your employer tsk at the inconvenience, vs stay up all night on the call with you, asking for an update every fifteen minutes? What costs anxiety, vs what costs money? In other words: what is your real priority order for things to go wrong? Get clear on this early.
- Our app cannot lose data: ornithology itself will wither on the vine. Tweets are scientific posterity. They are also a bedrock of user and community faith. Therefore, the Postgres is your baby, the thing you will protect with your life and tuck in at night. This is usually how these things go; data above all.
- There are "critical" or "high-priority" regions where a rare bird or seasonal phenomenon is underway. Downtime in these regions hurts the business.
- Duplicate or corrupted writes are potentially harmful to real user data, but serving incorrect things for brief windows is a somewhat expected risk for our apps' users. So long as any errors are corrected and "tweeted" by the company account, this is at worst an embarrassment.
- Downtime in non-critical regions is probably fine, if scheduled well in advance. Management will grumble but concede when you argue the point in terms of the more important priorities.
What's your current CAP compromise?
Ornithologists want to know what's in the area within seconds, or at worst minutes; even if one time in ten the info is bad, they'd rather hear about it than not.
Therefore our architecture prioritizes Availability. We have network edges, geospatially sharded; aggregation to the central deployment takes minutes verging to hours for photodata.
Where are we going?
Let's say our birdwatching company wants to become GDPR compliant, and is trying to move its center-servers to Europe. That's it, that's the whole thing. (And thank goodness, too, because we've got enough going on just trying to get this all done at all.)
(Well, while we're here, it sure would be nice to standardize bg with our other deployment stories.
Oh, and we've run into timing problems with our Redis/Postgres GIS-writing story three separate times in the past two years. Is there, like, a fix for that?
Of course, our secrets are kind of a mess, and it's going to be a bit of a problem for compliance…
Did I mention we also use Redis for our Bull queues? Yeah, Batch talks to it for that…oh, our Temporal? Yeah wait let me add you to our temporal.io account, go take a look. Just out of curiosity, how hard would it be to self-host Temporal? Just asking because of the licensing fees. You're migrating the Temporal region, right?)
The Gantt chart
Here's the secret to how this all works:
Put the most critical thing in the innermost change window. Treat it as the pivot-point of the change. How long will it take to cut it over?
This is your atomic step. Build outwards from here.
TUNE IN TOMORROW FOR PART 2
Footnotes:
god, Inkhaven, it's down to the damn wire today, isn't it? May I never speak of work related matters again. Let's go back to mourning our dead professors tomorrow, or something.
eliding some detail about network partition, don't be pedantic at me about it before I have a chance to justify myself