Slides from our talk at Percona Live NYC are now available. The presentation covers the design and implementation of Jetpants, Tumblr’s open source toolkit for interacting with hundreds of MySQL database servers.
Interested in working with relational databases at scale? Tumblr is currently hiring database engineers in NYC!
See more posts like this on Tumblr
#MySQL #jetpants #databasesTumblr is a big user of MySQL, and MySQL automation at Tumblr is centered around a tool we built called Jetpants. Jetpants does an incredible job making risky operations safe and reliable, even fairly complex tasks like replacing failed master servers, or splitting a shard.
While Jetpants is an incredibly effective and valuable tool for Tumblr’s day-to-day operation, it has remained very difficult to implement a meaningful testing framework. Integration testing at this level is very challenging. In this article I’ll go through these challenges and how we’ve tackled them at Tumblr.
Jetpants operates under the assumption you’re managing MySQL daemons on a fully functional host, and that it can:
Right away this means we have some challenges with respect to infrastructure testing:
For most of the life of Jetpants, these requirements were fulfilled using actual hardware in a testing pool in our datacenter. This wasn’t ideal, however. Running a test which allocated more replicas, or tested shard splitting means using an extensive amount of real hardware that takes hours to reprovision. Testing changes to the Collins code meant talking to a real Collins deployment. What if we messed up?
This test strategy has all the hallmarks of manual testing. It doesn’t prevent regressions. Test coverage of our featureset is spotty based on what was interesting at the time. Public contributors can’t run the tests.
For a new user to pick up Jetpants and Collins, it can be very difficult to get started. Jetpants requires Collins to be configured it certain ways that aren’t publicly documented. When I first built the testing environment, I had to regularly compare what I had to our actual deployment to figure out why Jetpants wasn’t working correctly.
During a Tumblr hackathon earlier this year, I devoted my time to developing an isolated, automatic testing system. We have since integrated this system directly into Jetpants and are using it in our day-to-day development and testing.
Our test framework is based on the NixOS test framework, the same framework NixOS uses to verify it is safe to release a new version. These tests use QEMU to start an isolated environment of at least one VM, and NixOS configuration to build the VMs.
Our testing framework adds lots of tooling on top to let us create robust tests. By default, a test has a running Collins instance, a master database server, and one replica. Simple options allow provisioning additional spares or additional replicas on that initial master.
Below is an example test we’ve written for performing a dead master promotion. This is where the current master database is dead, and we replace it with one of the existing replicas.
Here you can see what a test looks like, and how easily we can express the components and phases of our tests:
import ../make-test.nix ({ helpers, ... }:
{
name = "shard-dead-master-promotion";
starting-slave-dbs = 2;
test-phases = with helpers; [
(jetpants-phase "shutdown-master" ''
Jetpants.pool("POSTS-1-INFINITY").master.stop_mysql
'')
(phase "jetpants-promotion" ''
(
echo "YES" # Approve for promotion
echo "YES" # Approve after summary output. Confirmation.
) | jetpants promotion --demote=10.50.2.10 --promote=10.50.2.11
'')
(assert-shard-master "POSTS-1-INFINITY" "10.50.2.11")
(assert-shard-slave "POSTS-1-INFINITY" "10.50.2.12")
];
})
Running this test first provisions the base environment, by
Once all this preparation is done, our test phases begin.
First we shut down the current master, to simulate a dead master situation. We then run the jetpants promotion command which will replace the old master (10.50.2.10) with a new master we have selected, 10.50.2.11. jetpants promotion will prompt for confirmations, so we echo approvals to its stdin.
We continue by validating that the jetpants command did what we expected, and verifying the master and slaves.
Through this testing, we have already identified and fixed several race conditions and very old interface bugs. Nix’s functional nature allows us to create and tear down test VMs in minutes, as it isn’t a convergence-based configuration management tool. The stability of the test framework, and consistency of its results have allowed us to more aggressively change the underlying code in Jetpants while remaining confident our tools will work correctly during our day-to-day production maintenance.
Jetpants has been under continuous and vigorous development at Tumblr for many years now, and I’m excited about where the future will be taking MySQL automation at Tumblr.
- @beta
Tumblr is one of the largest users of MySQL on the web. At present, our data set consists of over 60 billion relational rows, adding up to 21 terabytes of unique relational data. Managing over 200 dedicated database servers can be a bit of a handful, so naturally we engineered some creative solutions to help automate our common processes.
Today, we’re happy to announce the open source release of Jetpants, Tumblr’s in-house toolchain for managing huge MySQL database topologies. Jetpants offers a command suite for easily cloning replicas, rebalancing shards, and performing master promotions. It’s also a full Ruby library for use in developing custom billion-row migration scripts, automating database manipulations, and copying huge files quickly to multiple remote destinations.
Dynamically resizable range-based sharding allows you to scale MySQL horizontally in a robust manner, without any need for a central lookup service or massive pre-allocation of tiny shards. Jetpants supports this range-based model by providing a fast way to split shards that are approaching capacity or I/O limitations. On our hardware, we can split a 750GB, billion-row pool in half in under six hours.
Jetpants can be obtained via GitHub or RubyGems.
Interested in this type of work? We’re hiring!
We’re pleased to announce that Tumblr’s Database Engineering team will be attending the Percona Live MySQL Conference next week in Santa Clara, CA!
We’ll be giving a talk on our open source automation software, Jetpants, which has helped us scale to over 175 billion distinct rows of relational data to date. We’re also looking forward to attending a number of amazing sessions from our friends at Percona, Facebook, Oracle, Palomino, Etsy, and more.
If you haven’t registered yet, use code SpeakMySQL to save 15%. Hope to see you there!
Recently we went through an exercise where we moved all of our database masters between data centers. We planned on doing this online with minimal user impact. Obviously when performing this sort of action there are a variety of considerations such as cache consistency and other pieces of shared state in stores like HBase, but the focus of this post will be primarily on MySQL.
During this move we had a number of constraints. As mentioned above this was to be online when serving production traffic with minimal user impact. In aggregate we service hundreds of thousands of database queries per second. Additionally we needed to encrypt all data transferring between data centers. MySQL replication supports encryption, but connections to the servers themselves present several challenges. Specifically, from a performance standpoint the handshake to establish a connection across a WAN can impact latency if there is significant connection churn. Additionally, servicing read queries across a backhaul link adds latency, which is never desirable.
We decided to tackle these issues in several ways. We were able to leverage a number of existing features of our applications and infrastructure, as well as developing new automation to fill gaps in functionality. Our configuration and applications in various runtimes, were able to support a read/write split (which may seem obvious to some, but isn’t always easy to accomplish in every scenario). We used the read/write split, along with encrypted replication, to provide a local read replica. Some runtimes can set up a persistent encrypted connection to a remote master, which serviced read requests in those cases, as the per-connection latency was amortized over a large number of queries. For runtimes which have a high churn rate, such as PHP, we used a MySQL proxy, ProxySQL, which provided persistent, encrypted connections, as well as meeting our performance requirements. We built automation to deploy proxies for numerous database pools, servicing thousands of requests per second, per pool.
When performing the cutover, our workflow was as follows. In each data center, there was a config which pointed to a local read slave, a remote master, and a local proxy with the master (remote or local) as a backend. When moving masters between datacenters, our database automation, Jetpants (new release coming soon!), reparented all replicas, and our automation updated the proxy backend to point to the new master. This resulted in seconds of read-only state per database pool and minimal user impact.
More coming soon!
Tim Ellis of PalominoDB wrote up a great walk-through for installing and using Jetpants on Ubuntu as well as Linux Mint.
We’ll be rolling in some of his changes in the next release, to make Jetpants run on more Linux distros out of the box.
MySQL is an integral component of Tumblr’s architecture. For performance reasons, we run Percona Server, a drop-in replacement for MySQL with major enhancements; we’ve also relied heavily on Percona for their consulting expertise for years. So naturally we’re very excited to attend the New York edition of the Percona Live MySQL Conference, coming up on October 1-2!
On October 2 we’ll be presenting the design and implementation of Jetpants, Tumblr’s open source toolkit for interacting with hundreds of database servers. Our session will delve into the motivations behind our toolkit, explain its internals, and discuss Tumblr’s database growth story as a whole.
If that’s not enough Jetpants for you, how about a live demonstration? Tim Ellis of PalominoDB will be giving a tutorial on October 1 covering the modern ecosystem of MySQL sharding options and tools, including Jetpants.
Several Tumblr engineers will be in attendance at Percona Live. If you’d like to chat about our engineering efforts (or even employment opportunities), come say hello!
Special thanks to Dallas and Matt for helping to formulate our shard-splitting algorithm, Zack for the slide deck visual excellence, and the fine folks at Velocity / O'Reilly Media for running a great conference.
Keeping a site like Tumblr alive and snappy for you to post at a moment’s notice, all day and night, is no small feat. Pesky crabs sneak into our data centers and cut cables all the time…
If you want to help our small but excellent systems team, want to work from anywhere, and are deep into nginx, mysql, kubernetes, and caching, join us in this adventure. Or, if you have a friend or a colleague who’s good with servers, send them our way.
If you need to copy large files between two machines, combining nc (netcat) with pigz (parallel gzip) is an easy and fast option. But what if you need to copy the same set of files to multiple destinations? This is a common need here at Tumblr, for example when we need to spin up several additional replicas of a MySQL instance at once in a fast, automation friendly way.
You could copy from a single source to two destinations serially, but that takes twice as long. Or you could copy from a single source to two destinations in parallel, but that doesn’t end up being faster, due to contention for the network uplink.
Happily, there’s a better way, utilizing some standard UNIX tools. By adding tee and a FIFO to the mix, you can create a fast copy chain: each node in the chain saves the data locally while simultaneously sending it to the next server in the chain.
First, set up the last destination box. This one just listens with nc (we’re using port 1234 for example), and pipe to pigz to decompress, and tar to extract:
nc -l 1234 | pigz -d | tar xvf -
Next, working backwards along the chain, set up the other destination boxes. We do the same listen/decompress/extract combo; but before the decompress/extract stages, we use the tee command to also redirect the output to a FIFO. A separate shell pipeline reads from the FIFO and sends the data – which is already archived and compressed – to the next destination box in the chain:
mkfifo myfifo
nc hostname_of_next_box 1234 <myfifo &
nc -l 1234 | tee myfifo | pigz -d | tar xvf -
Finally, on the source box, kick off the copy chain by sending the files to the first destination in the chain:
tar cv some_files | pigz | nc hostname_of_first_box 1234
In my testing, each box added to the chain only incurred a 3 to 10% performance penalty, which is a huge improvement over copying serially or in parallel from a single source.
We’ve wrapped this technique in some Ruby code that automates the process and verifies every step. Besides using it to spin up database replicas, we also use it as part of our process to split shards that have grown too large – a topic we’ll delve into further in a future post.
Once again it was Hack Week (more than just a day!) at Tumblr!
This is getting repetitive in the best way. A couple of times per year
we slow down our normal work and spend a week working on scratching a
personal itch or features we want as user and see how far we can get
with our hacks. One thing from the last Hack Week in September made it all the way to a new experiment out to some testers: Tumblr Patio!
Here are some of the projects that got built for our most recent Hack Week in January. Some of these things you may also end up seeing on the site…
This one is so obvious and amazing, it’s wild we don’t already have it. For Hack Week, Katie added the ability to select text in a paragraph to be hidden behind a wall of black that can be revealed with a tap. This can be super useful to hide spoilers. And even better: whole spoiler blocks. And while we’re here, the ability to center text!
We haven’t updated our default avatars in several years. (Some of you may remember this one
from 10+ years ago.) They’re feeling a bit stale to us, so why not
update them? And while we’re at it… make a ton more variations! Paul
from the Tumblr Design team came up with a suite of new default avatars,
using our latest Tumblr color palette. Here’s a look at some of them, but there are actually many dozens more using different colors:
This one is for the folks on Tumblr who love numbers and their Activity
page. Daniel, @jesseatblr, and the Feeds & Machine Learning team
worked on some new notifications and emails we could send out to people
about how their posts have been doing lately on the platform, such as
how many views they’ve gotten, and by how many people. We already have
this available (and more) when you Blaze a post, but why not open it up
to more people? It’s really useful to the folks who use Tumblr to help
build an audience for their work!
Some apps we use a lot have a “command palette” accessible via a
keyboard shortcut for quick keyboard-driven access to different parts of
the platform. For example, Slack and Discord have Command + K to access
their quick switchers to hop around conversations. What if Tumblr had
one? Kelly and Paul built one! Press Command/Control + K on Tumblr and
you can use your keyboard to jump to your blog, Activity, your recent
conversations, search, dozens of places!
As always, stay tuned to the @changes blog to see if any of these hacks make it on Tumblr for real!
