The Evolution of Code Deploys at Reddit

That's right. We ran Reddit out of a long-living CMUCL process. Instead of reloading everything, I (I'm always hesitant to use I instead of We, but it was really just me doing the dev at the time) updated the process one function at a time by pasting in newly updated code. Because of this, the instance of Reddit running in production wasn't necessarily documented in any specific code. It resembled more a well-aged cheese culture—impossible to reproduce.

Pfft. files

What are you? Why is your username displayed different (at least in RiF) than a normal user, mod or admin?

Maybe! The only reason I'm certain it wasn't screen is that I distinctly remember learning about screen thinking, "this will make things so much easier!"

My day job is primarily PHP for which that isn't the case. Most sites are served by PHP-FPM and therefore managed by systemd from the outset.

For other stacks like Python and Node.js you may well be right.

My first website was a process running on nohup... It was bad, I had to pkill it.

I've never seen a website hosted by screen or tmux. Most things are going to be either Apache, nginx, or Tomcat. All of these will be running in the background typically from the moment the system started.

There's nohup for that though.

Ah yeah, the still venerable StaleCache :)

It's a small additional cache local to each app server with a very short TTL that has a copy of the very most frequently accessed data like the the current r/all listing or the biggest few subreddits. This way a request can say "it's okay if I get an old copy of this data" for where they know it's okay to be a little out of date and not have to go over the network for it.

Also, hi raldi!

I'd say to start with a hosted/cloud provider but I think you'd have to do the maths on the costs and your team's skill on both. There are huge upsides and downsides for both. With physical hardware you can probably do it for cheaper in the long run, but you're way less flexible. And I think for a startup in particular flexibility is paramount.

The nice thing about physical hardware is that your costs are mostly fixed and you don't have the "noisy neighbour" problem. You can run custom hardware like fancy GPUs or giant RAID arrays and you can guarantee that your servers are physically and topologically close to each other. You can do things like have a separate dedicated network card with a direct gigabit uplink between your app servers and caches. But the downside is that you have to do all of that stuff. Ordering hardware takes a lot of time so you have to do proper capacity planning. Order too many and now you're stuck with them. You can't scale up and down with load. You'll need to tune your databases and run your own DHCP and solve all of that common networking stuff time and time again.

On a hosted provider you're paying someone else to do all of that. Providers like AWS can host things like RDS or elasticache for you so you don't have to be an expert DBA to get a performant database. You don't ever have to put more RAM in a machine because you can just upgrade to the next instance size. You don't have to sell off your obsolete hardware, you just terminate the instances. You don't ever have to drive down to the datacentre to swap out a failed hard drive, you just delete the EBS and make a new one (but you do still have to plan for them failing, and even more so). Of course the downside is that now AWS is doing all of that stuff for you so you don't have a lot of control over it. If an RDS is misbehaving there's nothing you can really do to fix it, it's all opaque to you. You can't have custom configuration, you can't flush the query cache, and you can't go and look at it to see which warning light is blinking on the front of the machine or read the dmesg to see whether it's using the right driver. Paying AWS to do this stuff for you is also really expensive, and dealing with their support when something on their end is broken can be very, very painful.

If you can do it better and cheaper on the team you have, do that instead. But on a small team, I bet you can't.

Comment deleted by user

Nearly always cloud. Especially in the beginning when you are not yet sure what you will even need (and when buying the hardware outright is to incur the cost up front).

We keep persistent connections, it's not the 3-way handshake that's the issue. It's just the regular old packet round trip time.

For instance, let's say you want the rendered versions of links A, B, and C. You can send requests that look like this:

link_a = cache.get('A')
link_b = cache.get('B')
link_c = cache.get('C')

So that turns into network traffic that looks like:

link_a = cache.get('A')  |  > GET A
                         |  < data for A
link_b = cache.get('B')  |  > GET B
                         |  < data for B
link_c = cache.get('C')  |  > GET C
                         |  < data for C

But what's invisible here is that you have to flush the socket in between each of those lines for the result to be in the link_a variable before you go on to get link_b. That means that you're sending one packet containing len("GET A\n") bytes, waiting for it to go through the OS buffers and go out the wire and for them to send you an ACK and then for them to send you a len("data for A\n")-byte response and for you to send them an ACK back again. Then you do all of that two more times for the other two items. Doing the requests async/in parallel helps a little but still carries huge overhead: the TCP packet header alone is ~20 bytes and our example requests are only 6!

On a really fast network all of this additional overhead probably doesn't matter much but if you're on a shared wire with thousands of other AWS customers and your packets go through a bunch of switches and whatever network virtualisation AWS runs, it adds a huge amount of latency even if the throughput is the same.

Instead what you'd rather do is a flow more like:

cache.get_multi(['A', 'B', 'C'])  |  > GET A B C
                                  |  < data for A
                                  |  < data for B
                                  |  < data for C

Then you send one packet, and receive one packet, instead of doing a bunch of unnecessary round trips.

The thing is, the naive version of it with all of the round trips is really easy to write without realising it. You've probably written something like this in an ORM before

class MyOrm:
    @staticmethod
    def get_thing(cls, key)
        cached = cache.get(key)
        if cached:
            return cached
        return cls.get_from_db(key)

So without having a batch interface, you have to call it like

def do_whatever(keys):
    for key in keys:
        thing = MyOrm.get_thing(key)
        thing.do_something(...)

do_whatever here is probably talking about things in terms of your application. In the case of reddit, it's all about Links and Votes and Comments. Network particulars just aren't at the forefront of your mind at that point. Even if they were the details are non-obvious: they're all hidden down the call stack far away from where you even think of things on that level.

Writing it with batching to start isn't any harder, what's harder is turning all of your loops inside out in a program that's already pervasively very loopy. A good rule of thumb is that if your function (or any of its callees) can do I/O, it should probably take a list instead of a single item. It's similar to making matrix/scientific code easier to vectorise.

A very similar thing happens with DB transactions as you alluded to. In something like MySQL or Postgres UPDATE/INSERT is actually pretty cheap, it's the COMMIT that's expensive. So lots of tiny transactions has a very similar effect

Starting around then we started printing every SQL query we run in development mode and that made a big difference at the time. If it adds enough log spam that you can't find what you're looking for, you're probably making too many queries.

An unnamed member of the team had secretly modified his bzflag game client to show him where the best weapons were on the map. He also seemed to win every round. A coincidence, I'm sure

cough don't know if you saw the hiring thing. nudge nudge wink wink know-what-I-mean

You get a punch card when you come back. Free small sundae on third rehire!

Hello Mr. Spicy. We meet again.

Comment deleted by user

Yep we used to do this but it's not so practical now

There's so much to Reddit under the hood that you and I don't know about. I'm gonna throw out a random number and say 90% of the code is probably agnostic to whether it's used by the mobile app, website, or web app. There's an insane amount of work that has to go towards maintaining a site of this scale.

We're also not even considering things like event tracking, A/B testing, performance improvements, site reliability improvements, etc. You can't just launch a website at the scale of Reddit and walk away and let it collect money. It's a very full time operation.

Rip. I understand when companies reject someone, but it sucks not getting a notification of rejection or a notification that the position has been filled.

Is it?

Yep! Depends on the position though. Doesn't hurt to apply!

This is why I haven't applied. This is the work I do in my current position.

Not true! It depends on the position, but we're more open to remote work these days than in the past.

No prob! Also, whoops that's a pretty old one. Here's a more recent AMA: https://www.reddit.com/r/sysadmin/comments/57ien6/were_reddits_infraops_team_ask_us_anything/

Yup! flake8 and pylint feature pretty heavily. You can see the template we use for a new, empty, service here: https://github.com/reddit/baseplate-cookiecutter/tree/master/baseplate_cookiecutter/%7B%7Bcookiecutter.project_slug%7D%7D

My understanding is that upgrading etcd is the main hairy part.