In the ten years that Tumblr’s been around, a lot has changed in web technology. We’ve kept up, of course, but it’s always been a process of addition, layering one new technology on top of another. And what we were working with—a custom framework built on top of Backbone, messily entangled with a PHP backend and its associated templates—was becoming unmanageable. Our piecemeal conversions to new technologies meant we had thousands of ways posts were rendered (only a moderate exaggeration). And each of those had to be updated individually to support new features or design changes.

It was time to step back, survey the world of web technology, and clean house in a big way. That we could finally test some of the new tech we’ve been itching to use was just a little bonus.

We started by laying out our goals:

• A web client codebase fully separated from the PHP codebase that gets its data from the API in the same way our mobile apps do
• A development environment that’s as painless as possible
• Dramatically improved performance
• Isomorphic rendering
• Robust testing tools
• Built on a framework with a healthy and active community, with some critical mass of adoption

With those goals in mind, we spent the beginning of the year on research - figuring out what kinds of things people were building web apps with these days, tooling around with them ourselves, and trying to assess if they would be right for Tumblr. We landed, eventually, on React, with a Node server (running Express) to make isomorphism as easy as possible. On top of that, we’re using Cosmos for developing components, React Router for routing, and TypeScript to make our lives better in general. (My colleague Paul already wrote about what went into our decision to use TypeScript here.)

As if writing an entirely new stack wasn’t enough, we realized along the way that this was our perfect chance to start deploying containerized applications with Kubernetes, a first for Tumblr. We had never previously deployed a node application to production here, and didn’t have the infrastructure for it, so it was a perfect green field on which to build another new and exciting thing. There’ll be more to come later on Kubernetes.

So where are we now? Well, we’ve launched one page powered by this new app - image pages, like this - with more to come very soon. 

Though it may seem simple, there’s a whole new technological world between you clicking that link and seeing that page. There’s a ton more exciting stuff happening now and still to happen in the future, and we’re looking forward to sharing it here. Wanna get in on the action yourself? Come work with us: https://www.tumblr.com/jobs.

- Robbie Dawson / @idiot

One of the Core Web team’s goals at Tumblr is to reduce the number of runtime issues that we see in our React codebase. To help move some of those issues from runtime to compile time, I evaluated the two leading type systems, Flow and TypeScript, to see if they could give us more type safety. I did a bit of background reading about the differences between Flow and TypeScript to see what the community had to say about them.

Background Reading

TypeScript, Flow and the Importance of Toolchains over Tools by Ben Teese

This post claims that Flow and TypeScript are similar enough that you should choose whichever of them is easier to integrate with your other tools. For Angular development, it recommends using TypeScript; for React, Flow.

TypeScript vs. Flow by Marius Schulz

This post claims that both TypeScript and Flow are equally good.

Flow vs. Typescript by Jan Varwig

This post outlines the author’s experience with using Flow in a React codebase. It advocates switching from Flow to TypeScript because of Flow’s unhelpful error messages, bad tooling, and propensity to spread untyped code. It also claims that most of the type annotations are able to be shared between Flow and TypeScript with only minor changes.

Type Systems for JavaScript by Oliver Zeigermann

This slideshow shows many differences around the philosophies and goals of TypeScript and Flow, and it gives detailed explanations in the differences between the two type systems. It explains IDE support and how to get access to third-party type definitions.

Lack of Consensus

It seems like many people have differing opinions about which type system is better for a React codebase. Because there wasn’t a broad consensus across the community, I decided to get some first-hand experience with each of these tools to see which one would be most practical and helpful for use at Tumblr.

Project Setup

I worked with a sample application to vet Flow and TypeScript. The application I used was Microsoft’s TypeScript React Starter. It uses a custom fork of create-react-app to get TypeScript set up. When testing out Flow, I used the standard version of create-react-app and used the source code from this exercise.

For the most part, Flow and TypeScript are basically interchangeable. I was able to reuse most of the source code between both projects with only minor changes. Here are some examples of changes I needed to make to get my TypeScript code working with Flow:

• Flow requires that types are imported using import type where TypeScript re-uses import.
• Some generic type constraints are different in redux’s type declarations between Flow and TypeScript, so I dropped the generic constraint for Flow.
• Types cannot have the same name as constants, so I had to rename a few small things (see below).

Testing

After I got the project prepared I set up the following situations to see which tool performed better. These are my assumptions of the most common situations in which a type checker will help when writing React code on a day-to-day basis.

Handling an Unnecessary Case in a Switch

TypeScript

TypeScript realizes that 'not_real' is not a possible case for the switch.

Flow

Flow does not detect any issue.

Declaring Variables with Same Name as Type

TypeScript

TypeScript allows types to have the same name as constants, and it allows Command-clicking on the types to see their declarations.

Flow

Flow requires types and constants to have different names. In this case, I needed to rename the type to INCREMENT_ENTHUSIASM_T to appease Flow’s type checker.

Returning Incorrect Type from Function

TypeScript

[ts]
    Type '{ enthusiasmLevel: string; languageName: string; }' is not assignable to type 'StoreState'.
      Types of property 'enthusiasmLevel' are incompatible.
        Type 'string' is not assignable to type 'number'.

Flow 0.52

[flow] object literal (This type is incompatible with the expected return type of object type Property `enthusiasmLevel` is incompatible:)

Flow 0.53

[flow] property `enthusiasmLevel` of StoreState (Property not found in number) [flow] property `languageName` of StoreState (Property not found in number)

Missing Required Props When Instantiating a Component

TypeScript

TypeScript shows the error at the site where the properties are missing with the error:

[ts] Type '{}' is not assignable to type 'IntrinsicAttributes & Props'. Type '{}' is not assignable to type 'Props'. Property 'name' is missing in type '{}'.

Flow

Flow shows the error within the component where the property will be used, with no way to discover which call site is missing a property. This can be very confusing in codebases that have lots of reusable components. Flow displays this error:

[flow] property `name` of Props (Property not found in props of React element `Hello`)

Code Safety

TypeScript

TypeScript allows enforcing full type coverage on .ts files with the noImplicitAny flag in the tsconfig.

Flow

Flow provides a code coverage plugin so that you can see which lines are implicitly not typed.

Other Considerations

Flow has the most React community support and tooling, so there is much more documentation about how to get Flow and React working together. TypeScript is more popular with Angular developers. Choosing TypeScript may be breaking from community standards, so we may have more issues that don’t have a simple answer on Google.

Conclusion

I concluded that we should use TypeScript because it seems easier to work with. My experience seems to line up with this blog post. It has better error messages to debug type issues and its integration with VSCode makes coding more pleasant and transparent. If this ends up being the wrong choice later on, our codebase will be portable to Flow with some minor changes.

Shortly after arriving at this conclusion, Flow 0.53 was released and a blog post on Medium published touting it’s “even better support for React”. However, after running through the test cases above, I only found one case where Flow had improved its error messaging. TypeScript still seems like the more reliable, easier to use solution.

Further Reading

To continue our journey with TypeScript, I will need to integrate it into our codebase and teach it to the rest of our frontend developers. Getting started with TypeScript and React and Setting up a new Typescript 1.9 and React project look like they will be helpful articles when integrating TypeScript into our codebase. TypeScript Deep Dive looks like a great book for JavaScript developers that aren’t familar with TypeScript.

– Paul Rehkugler (@pr)

How we wrote our own Service Worker

As we continue the process of reinvigorating Tumblr’s frontend web development, we’re always on the lookout for modern web technologies, especially ones that make our mobile site feel faster and more native. You could have guessed that we are making the mobile dashboard into a progressive app when we open-sourced our webpack plugin to make web app manifests back in August. And you would’ve been right. But to make a high quality progressive web app, you need more than just a web app manifest—you also need a service worker.

What is a service worker?

A service worker is a helper script that a page registers with the browser. After it is registered (some people like to also call it “installed”), the browser periodically checks the script for changes. If any part of the script contents changes, the browser reinstalls the updated script.

Service workers are most commonly used to intercept browser fetches and do various things with them. https://serviceworke.rs has a lot of great ideas about what you can do with service workers, with code examples. We decided to use our service worker to cache some JS, CSS, and font assets when it is installed, and to respond with those assets when the browser fetches any of them.

Using a service worker to precache assets

You might be wondering “why would you want to pre-cache assets when the service worker is installed? Isn’t that the same thing that the browser cache does?” While the browser cache does cache assets after they’re requested, our service worker can cache assets before they’re requested. This greatly speeds up parts of the page that we load in asynchronously, like the notes popover, or blogs that you tap into from the mobile dashboard.

While there are open-source projects that generate service workers to pre-cache your assets (like, for example, sw-precache), we chose to build our own service worker. When I started this project, I didn’t have any idea what service workers were, and I wanted to learn all about them. And what better way to learn about service workers than building one?

How our service worker is built

Because the service worker needs to know about all of the JS, CSS, and font assets in order to pre-cache them, we build a piece of the service worker during our build phase. This part of the service worker changes whenever our assets are updated. During the build step, we take a list of all of the assets that are output, filter them down into just the ones we want to pre-cache, and write them out to an array in a JS file that we call sw.js.

That service worker file importScripts()’s a separate file that contains all of our service worker functionality. All of the service worker functionality is built separately and written in TypeScript, but the file that contains all of our assets is plain JavaScript.

We decided to serve our service worker directly from our node.js app. Our other assets are served using CDNs. Because our CDN servers are often geographically closer to our users, our assets load faster from there than they do from our app. Using CDNs also keeps simple, asset-transfer traffic away from our app, which gives us space us to do more complicated things (like rendering your dashboard with React).

To keep asset traffic that reaches our app to a minimum, we tell our CDNs not to check back for updates to our assets for a long time. This is sometimes referred to as caching with a long TTL (time to live). As we know, cache-invalidation is a tough computer science problem, so we generate unique filenames based on the asset contents each time we build our assets. That way, when we request the new asset, we know that we’re going to get it because we use the new file name.

Because the browser wants to check back in with the service worker script to see if there are any changes, caching it in our CDNs is not a good fit. We would have to figure out how to do cache invalidation for that file, but none of the other assets. By serving that file directly from our node.js application, we get some additional asset-transfer traffic to our application but we think it’s worth it because it avoids all of the issues with caching.

How does it pre-cache assets?

When the service worker is installed, it compares the asset list in sw.js to the list of assets that it has in its cache. If an asset is in the cache, but not listed in sw.js, the asset gets deleted from the cache. If an asset is in sw.js, but not in the service worker cache, we download and cache it. If an asset is in sw.js and in the cache, it hasn’t changed, so we don’t need to do anything.

// in sw.js

self.ASSETS = [

  ‘main.js’,

  ‘notes-popover.js’,

  'favorit.woff’

];

// in service-worker.ts

self.addEventListener('install’, install);

const install = event => event.waitUntil(

  caches.open('tumblr-service-worker-cache’)

    .then(cache => {

      const currentAssetList = self.ASSETS;

      const oldAssets =/* Instead of writing our own array diffing, we use lodash’s */;

      const newAssets =/* differenceBy() to figure out which assets are old and new */;

      return Promise.all([ …oldAssets.map(oldAsset => cache.delete(oldAsset)), cache.addAll(newAssets)]);

  });

);

We launched 🚀

Earlier this month, we launched the service worker to all users of our mobile web dashboard. Our performance instrumentation initially found a small performance regression, but we fixed it. Now our mobile web dashboard load time is about the same as before, but asynchronous bundles on the page load much faster.

We fixed the performance regression by improving performance of the service worker cache. Initially, we naively opened the service worker cache for every request. But now we only open the cache once, when the service worker starts running. Once the cache is opened, we attach listeners for fetch requests, and those closures capture the open cache in their scope.

// before

self.addEventListener('fetch’, handleFetch);

const handleFetch = event =>

  event.respondWith(

    caches.open('tumblr-service-worker-cache’)

      .then(cache => cache.match(request)

      .then(cacheMatch => cacheMatch

        ? Promise.resolve(cacheMatch)

        : fetch(event.request)

      )

    )

  );

// now

caches.open('tumblr-service-worker-cache’)

  .then(cache =>

    self.addEventListener('fetch’, handleFetch(cache));

const handleFetch = openCache => event =>

  event.respondWith(

    openCache.match(request)

      .then(cacheMatch => cacheMatch

        ? Promise.resolve(cacheMatch)

        : fetch(event.request)

      )

  );

Future plans

We have lots of future plans to make the service worker even better than it is now. In addition to pre-emptive caching, we would also like to do reactive caching, like the browser cache does. Every time an asset is requested that we do not already have in our cache, we could cache it. That will help keep the service worker cache fresh between installations.

We would also like to try building an API cache in our service worker, so that users can view some stale content while they’re waiting for new content to load. We could also leverage this cache if we built a service-worker-based offline mode. If you have any interest in service workers or ideas about how Tumblr could use them in the future, we would love to have you on our team.

- Paul / @blistering-pree

Making a progressive web app with webpack just got a little bit easier

Today we are releasing webpack-web-app-manifest-plugin, which generates an app manifest that shows up in your assets manifest.

I heard you like manifests

Turns out, there are a lot of web things called “manifests”. When talking about web app manifests and assets manifests, sometimes it’s hard to keep track. Buckle up, because we made a webpack plugin that deals with both of these types of manifests.

Web app manifests are JSON files that allow your application to specify the way it should be treated when installed as an application on a mobile device. You may want to specify what the application name and icon should be. Maybe you want to tell the browser to tint some of its UI elements to match the color scheme of your page, or even hide the browser chrome entirely. You can do all of that with a web app manifest.

Assets manifests are JSON files that contain paths to assets that are generated by webpack. They’re generated by plugins such as assets-webpack-plugin. If you add hashes to the end of your filenames to allow cache busting, assets manifests can be very useful. For example, we use our assets manifest to add JavaScript and CSS files to our <script> and <link> tags.

So I put a manifest in your manifest

While we were building our web app manifest, we wanted to be able to add a hash to the file path and <link> to it. So we needed to add it to our assets manifest. Unfortunately, we were unable to find any existing open-source plugins that output the file in the correct way to add it to the app manifest. So, we built webpack-web-app-manifest-plugin.

By default, webpack-web-app-manifest-plugin assumes that you will name your icon files in the format manifest/icon_[square dimension].(png|jpeg|jpg). If you name them using that scheme, you can use this plugin just like this:

// in your webpack config
import AppManifestPlugin from ‘webpack-web-app-manifest-plugin’;

…

plugins: [
  new AppManifestPlugin({
    content: {
      name:'Tumblr’,
      short_name:'Tumblr’,
      background_color:’#36465d’,
    },
    destination: ’/manifest’,
  }),
],

…

// in your page template
const manifest = // however you usually access your asset manifest in code
const appManifestPath = manifest['app-manifest’].json;

<link rel=“manifest” href={appManifestPath} />

If you named your icons with some other naming scheme, you can still add them to the web app manifest, it’s just a little more work. That process is detailed in the README.

Please use it

We’re really proud of the work we’ve done to make web app manifests compatible with asset manifests, which is why we’ve decided to open source it and publish it on npm. Please use it.

If this plugin doesn’t meet your needs, we welcome pull requests. And if you have a passion for progressive web applications, webpack, and open source, join our team!

- Paul Rehkugler (@blistering-pree)

A Breath of Life: Welcome the Brand New Mobile Web Dashboard

We rebuilt another piece of the site with React

Background

A few months ago my colleague Robbie wrote a post about how the Core Web team is introducing a new web stack built on Docker, Node, and React. It is a beautiful vision, but we had only launched one tiny page on that infrastructure. This spring, we used that new stack to rebuild the mobile web dashboard from the ground up. And then we launched it.

Why we did it

We decided on building the mobile dashboard next for a few reasons. First, the dashboard is one of the most important parts of Tumblr—for many people it is their primary window into the content here. Because of that, most things that we develop inevitably touch the dashboard. It was time to raise the stakes.

The desktop dashboard and the mobile apps have been moving forward at a blistering pace, but have you seen the mobile web dashboard? It was stuck in 2015. Nobody ever made features for it because it was a completely separate codebase. We avoided that issue with the rewrite. Since the new version we’re building is responsive, our new mobile web dashboard will eventually also power the desktop web dashboard experience. This’ll make it much easier to maintain feature parity between the desktop and mobile web experiences.

Finally, we wanted to make something a little bit more complex than the image page, and the mobile web dashboard seemed like a good next step. It allowed us to start making authenticated API calls and figuring out how to pass cookies through our new server to and from our API; to think about laying the groundwork for a totally responsive dashboard, so we can eventually launch this for the desktop; to make a testing ground for progressive web app features like web app manifests; to start rendering posts that are backed by the new post format; and to figure out how to slowly roll out the new stack to a consistent, small set of users. We also had a pretty bad performance bug and learned a whole lot about profiling Node. (We plan on writing more in-depth posts about some of these topics soon.)

It is good

And the rewrite turned out really well. Even with more features like pull-to-refresh, a new activity popover, and modern audio and video players, we sped up our DOM content loaded time by 35%! The new page has a lot of modern web standards in it too like srcsets, flexbox, asynchronous bundle loading, and a web app manifest. You can install the mobile web dashboard as a standalone app now. It was also way faster and simpler to write using React than it was on our old PHP/Backbone stack.

We’re really proud of the new mobile web dashboard. We look forward to bringing even more new features to it in the near future, and launching even more pages on our new web infrastructure.

If you think this is cool too, come work with us!

– Paul Rehkugler (@blistering-pree)