Tumblr Engineering — OAuth 2 on the Tumblr API

You’ve been asking for an official Golang wrapper for the Tumblr API. The wait is over! We are thrilled to unveil two new repositories on our GitHub page which can be the gateway to the Tumblr API in your Go project.

• API Endpoints Wrapper
• API Client

Why Two Repos

We’ve tried to structure the wrapper in a way that is as flexible as possible so we’ve put the meat of the library in one repo that contains the code for creating requests and parsing the responses, and interacts with an interface that implements methods for making basic REST requests.

The second repo is an implementation of that interface with external dependencies used to sign requests using OAuth. If you do not wish to include these dependencies, you may write your own implementation of the ClientInterface and have the wrapper library use that client instead.

Handling Dynamic Response Types

Go is a strictly typed language including the data structures you marshal JSON responses into. This means that the library could have surfaced response data as a map of string => interface{} generics which would require the engineer to further cast into an int, string, another map of string => interface{}, etc. The API Team decided to make it more convenient for you by providing typed response values from various endpoints.

If you have used the Tumblr API, you’ll know that our Post object is highly variant in what properties and types are returned based on the post type. This proved to be a challenge in codifying the response data. In Go, you’d hope to simply be able to define a dashboard response as an array of posts

type Dashboard struct {
  // ... other properties
  Posts []Post `json:"posts"`
}

However this would mean we’d need a general Post struct type with the union of all possible properties on a Post across all post types. Further complicating this approach, we found that some properties with the same name have different types across post types. The highest profile example: an Audio post’s player property is a string of HTML while a Video post’s player property is an array of embed strings. Of course we could type any property with such conflicts as interface{} but then we’re back to the same problem as before where the engineer then has to cast values to effectively use them.

Doing Work So You Don’t Have To

Instead, we decided any array of posts could in fact be represented as an array of PostInterfaces. When decoding a response, we scan through each post in the response and create a correspondingly typed instance in an array, and return the array of instances as an array of PostInterfaces. Then, when marshalling the JSON into the array, the data fills in to the proper places with the proper types. The end user can then interact with the array of PostInterface instances by accessing universal properties (those that exist on any post type) with ease. If they wish to use a type-specific property, they can cast an instance to a specific post type once, and use all the typed properties afterward.

This can be especially convenient when paired with Go’s HTML templating system:

snippet.go

// previously, we have some `var response http.ResponseWriter`
client := tumblrclient.NewClientWithToken(
    // ... auth data
)

if t,err := template.New("posts").ParseFiles("post.tmpl"); err == nil {
    if dash,err := client.GetDashboard(); err == nil {
        for _,p := range dash.Posts {
            t.ExecuteTemplate(response, p.GetSelf().Type, p.GetSelf())
        }
    }
}

post.tmpl

{{define "text"}}
<div>
    {{.Body | html}}
</div>
{{end}}
{{define "photo"}}
<div>
    Post: {{.Type}}
</div>
{{end}}
{{define "video"}}
<div>
    Post: {{.Type}}
</div>
{{end}}
{{define "audio"}}
<div>
    Post: {{.Type}}
</div>
{{end}}
{{define "quote"}}
<div>
    Post: {{.Type}}
</div>
{{end}}
{{define "chat"}}
<div>
    Post: {{.Type}}
</div>
{{end}}
{{define "answer"}}
<div>
    Post: {{.Type}}
</div>
{{end}}
{{define "link"}}
<div>
    Post: {{.Type}}
</div>
{{end}}

This is a rudimentary example, but the convenience and utility is fairly evident. You can define blocks to be rendered, named by the post’s type value. Those blocks can then assume the object in its named scope is a specific post struct and access the typed values directly.

Wrapping Up

This is a v1.0 release and our goal was to release a limited scope, but flexible utility for developers to use. We plan on implementing plenty of new features and improvements in the future, and to make sure that improvements to the API are brought into the wrapper. Hope you enjoy using it!

At Tumblr, we’re always looking for new ways to improve the performance of the site. This means things like adding caching to heavily used codepaths, testing out new CDN configurations, or upgrading underlying software.

Recently, in a cross-team effort, we upgraded our full web server fleet from PHP 5 to PHP 7. The whole upgrade was a fun project with some very cool results, so we wanted to share it with you.

Timeline

It all started as a hackday project in the fall of 2015. @oli and @trav got Tumblr running on one of the PHP 7 release candidates. At this point in time, quite a few PHP extensions did not have support for version 7 yet, but there were unofficial forks floating around with (very) experimental support. Nevertheless, it actually ran!

This spring, things were starting to get more stable and we decided it was time to start looking in to upgrading more closely. One of the first things we did was package the new version up so that installation would be easy and consistent. In parallel, we ported our in-house PHP extensions to the new version so everything would be ready and available from the get-go.

A small script was written that would upgrade (or downgrade) a developer’s server. Then, during the late spring and the summer, tests were run (more on this below), PHP package builds iterated on and performance measured and evaluated. As things stabilized we started roping in more developers to do their day-to-day work on PHP 7-enabled machines.

Finally, in the end of August we felt confident in our testing and rolled PHP 7 out to a small percentage of our production servers. Two weeks later, after incrementally ramping up, every server responding to user requests was updated!

Testing

When doing upgrades like this it’s of course very important to test everything to make sure that the code behaves in the same way, and we had a couple of approaches to this.

Phan. In this project, we used it to find code in our codebase that would be incompatible with PHP 7. It made it very easy to find the low-hanging fruit and fix those issues.

We also have a suite of unit and integration tests that helped a lot in identifying what wasn’t working the way it used to. And since normal development continued alongside this project, we needed to make sure no new code was added that wasn’t PHP 7-proof, so we set up our CI tasks to run all tests on both PHP 5 and PHP 7.

Results

So at the end of this rollout, what were the final results? Well, two things stand out as big improvements for us; performance and language features.

Performance

When we rolled PHP 7 out to the first batch of servers we obviously kept a very close eye at the various graphs we have to make sure things are running smoothly. As we mentioned above, we were looking for performance improvements, but the real-world result was striking. Almost immediately saw the latency drop by half, and the CPU load on the servers decrease at least 50%, often more. Not only were our servers serving pages twice as fast, they were doing it using half the amount of CPU resources.

These are graphs from one of the servers that handle our API. As you can see, the latency dropped to less than half, and the load average at peak is now lower than it’s previous lowest point!

Language features

PHP 7 also brings a lot of fun new features that can make the life of the developers at Tumblr a bit easier. Some highlights are:

• Scalar type hints: PHP has historically been fairly poor for type safety, PHP 7 introduces scalar type hints which ensures values passed around conform to specific types (string, bool, int, float, etc).
• Return type declarations: Now, with PHP 7, functions can have explicit return types that the language will enforce. This reduces the need for some boilerplate code and manually checking the return values from functions.
• Anonymous classes: Much like anonymous functions (closures), anonymous classes are constructed at runtime and can simulate a class, conforming to interfaces and even extending other classes. These are great for utility objects like logging classes and useful in unit tests.
• Various security & performance enhancements across the board.

Summary

PHP 7 is pretty rad!

As some of you close watchers may have noticed, we recently updated the ID numbers for new posts on Tumblr to be huuuuuge. Post IDs were always 64-bit integers to us at Tumblr, but now they’re actually big enough to push into that bitspace. While this doesn’t change anything for anyone using the official Tumblr apps or website, it did cause some hiccups for third-party consumers using programming languages like Javascript, which support only 53 bits of precision for integers.

To help alleviate this, we’ve added a new field to our post objects via the Tumblr API called id_string, which is a string representation of the numeric post ID. You can use the value of this id_string instead of id in any request to the Tumblr API and it should work just the same. This is the same thing that Twitter did when they moved to big-number “snowflake” identifiers. Starting March 16th, you should see this new field whenever you encounter a post via the Tumblr API.

Why’d we change post IDs to be so huge? Some of you may have noticed there was quite a jump. We recently migrated Tumblr to a new datacenter, and as a part of that migration we updated the system that generates new post IDs. The new system generates much bigger IDs because it uses a different algorithm to generate them more safely.

If you run into trouble with this or don’t see it somewhere you need it, please contact Tumblr Support and we’ll take a look!

It was HACK WEEK at Tumblr and Automattic earlier this month! Just like back in June, we stopped our normal work to focus on some great possible new features, some bug fixing, and more. Here’s a taste of some of the projects that got hacked together. Some of these will hopefully make it to be real new parts of Tumblr, but as always, we can’t guarantee anything here with a timeline or even that they’ll ever see the light of day again – they’re hacks!

Fixing Tag URL issues across Tumblr

L and @superchlorine are working on fixing a lot of issues about how we parse tags in URLs, so that we are more consistent across the web and the apps. This is something that we know has been frustrating for a long time: trying to search for posts tagged with a + sign or with spaces often doesn’t work as you’d expect. But not for much longer! We’ve been slowly fixing these issues one at a time and posting about it on the Changes blog.

Tumblr Ignite: pay to promote posts!

This actually carried over from a previous Hack Day: the ability to click a little 🔥 button on a post, throw down some cash, and promote that post as a Sponsored post across Tumblr. There’d be some kind of moderation queue on our side to ensure nothing unsafe gets sponsored in this way, and any sponsored post would be “frozen” and not editable during the time it’s being promoted. The Ads Core team, lead by @thstall with help from Argyris, Oleg, and Nathan took this pretty far towards being production-ready.

Bringing two-factor authentication settings to the mobile apps

We really want as many people as possible to protect their accounts by leveraging Two Factor Authentication, which you can currently enable by going to your account settings on the web. But it’d be even better to be able to access these settings in the mobile apps themselves! Megs, Adam, and Eduardo are still working to make this a reality.

The Return of Reblog Graphs

For those of you who remember this experiment in Tumblr Labs, this is a popup you can check out on posts which shows you a graph of all of the reblogs on a post and how they’re connected. This is one crazy way you can visualize how posts on Tumblr get reblogged around in huge chains and clusters! @cyle got a working React-based prototype working for Hack Week, and now we’re working to bring this back to the web for real.

Tumblr Fan Funding

Connie and the Post+ team built a prototype of Fan Funding on Tumblr, to help creators set a funding goal and get to that goal, using a lot of the same technology we used to create Post+.

Stay tuned to the @changes blog to find out if any of these end up becoming a part of Tumblr for real!

One distinguishing feature of Automattic’s work culture is a team rotation, through which an individual can move from one team to another. A rotation can happen for a few reasons: to “try out” a new role and gain new skills, to backfill an understaffed team, or to cultivate cross-pollination and diversity across the company. Some rotations are several months long, while others may turn into a permanent team switch.

Tumblr, which is part of Automattic, is currently hiring data scientists and search engineers and machine learning engineers. Thanks to this system of rotations, last year we were able to fill a few of these positions, on the Tumblr Core Data Science team, internally. This team contributes to Tumblr recommendation systems, discovery feeds, targeted push notifications, user-interest profiling, and computational advertisement. Team members work on designing, developing, and maintaining large-scale machine learning algorithms, data pipelines, and backend services to connect users with the content they love. Our data infrastructure is built on top of open source big data frameworks such as Apache Spark and Scalding, orchestrated by Apache Airflow and with a PHP backend layer.

In this post, Adam and Vicki, two Automatticians who experienced a rotation onto the team, tell us about their work in data, their rotations, and being part of Automattic.

Very cool stuff! 👏 Give it a read.

It was Hack Day once again at Tumblr! A couple of times per year we grind everything to a halt and spend 24 hours working on whatever we want and see how far we can get with our hacks. Here are some of the projects that got made for Hack Day! Some of these things you may end up seeing on the site…

Twitch Embeds

Wesley hacked together the ability to post Twitch streams to Tumblr! These can be live streams or clips.

Tumblr to Discord

@cyle put together a very simple webhook integration between Tumblr and Discord so you can send events about your blog to a Discord server:

Custom Tumblr logos on mobile

@mlu, @dakotairene, and friends hacked together the ability for us to put custom Tumblr logos in the mobile apps’ dashboard tab bar, like we do on the web!

Tumblr Time Machine

Lucila constructed an elaborate Tumblr Time Machine, so you can filter search results to a specific year:

Stay tuned to the @changes blog to see if any of these hacks make it on Tumblr for real!

(GIF by @jjjjjjjjjjohn )

It was Hack Day again at Tumblr! A couple of times per year we grind everything to a halt and spend 24 hours working on whatever we want and see how far we can get with our hacks. Here are some of the projects that got made for Hack Day! Some of these things you may end up seeing on the site…

Timestamps on Posts

@superchlorine hacked the website to show timestamps on every post, reblog trail item, and note in the notes view. This is also already a popular XKit extension, and it’s easy to see why!

Mutual Love

Evgeniy hacked the Tumblr Android app to play a cute sound and animation whenever you follow someone who already follows you! Being mutuals on Tumblr is something special, and this celebrates it in such a cute way.

(This example uses this lovely kiss animation.)

Grid and Masonry View for the Dashboard

@cyle hacked the dashboard on web to view it in a grid format (like the Likes page on web) or a masonry format (like the Explore and Tagged pages on web). It creates a dense amount of information!

Post only to a Tagged Page

Tanya hacked the post editor to let you post directly to a tagged page! The posts made this way will only show up on that tagged page, and not your blog, or anywhere else. Great for folks that want to participate in a tag but not have it show up on their blog!

Mobile Rave Room

And it wouldn’t be a Tumblr Hack Day without a rave room hack: @oli took the rave room mobile and did a (completely not officially sponsored) DJ set in NYC and live streamed it.

Stay tuned to the @changes blog to see if any of these hacks make it on Tumblr for real!