Are you an iOS or OS X developer with opinions on how to implement large, performant applications using Core Data? If so, I want to hear what you think.
Core Data, Apple’s object-graph management framework for iOS and OS X, is really powerful but also somewhat complicated, and not always entirely straightforward to use. Sometimes it feels like everyone has their own unique way of using it. We’ve used Core Data for persistence in Tumblr for iOS for years now but are always interested in re-evaluating our approach to make sure that we’re leveraging the SDK as effectively as possible.
In the interest of gathering feedback, we’ve published the CoreDataExample sample project (inventive naming, I know). It displays a little bit of data from Tumblr API using a class called TMCoreDataController to access the underlying database, both from the main queue (user interface) and in the background. Everything in here should be pretty straightforward, but we’d still love for you to have a look and provide any feedback you may have on how we’ve implemented these particular access patterns.
The README contains more background information, so I’d recommend starting there if I’ve successfully piqued your interest. Thanks in advance; looking forward to hearing your thoughts.
Some consider the Tumblr iOS app to be a marvel of modern medicine and science. Others find our custom native HTML rendering to be a telltale sign of our insanity. Other others view our staunch avoidance of Core Data migrations as both prudent and educational. Whichever camp you fall into, you may find something to enjoy in this high-level overview of our iOS application architecture.
Core Data is great but automatic migrations can be tricky. Migrations can take a long time, which could
result in your app being terminated
if it is happening on the main thread during application launch. Performing migrations on a background
thread is also a bad idea, meaning your application really
needs to be able to fully launch without a Core Data stack whatsoever in order to safely migrate.
This can be a huge change to make to an existing app.
If you’re really only using Core Data as a cache, you don’t actually need to perform a migration.
Simply check if the existing store is compatible with your managed object model and if not, delete
and recreate it. Here’s my approach for doing so.
Many thanks to Marcus Zarra for being extremely
helpful both over Twitter and on Stack Overflow. You should totally buy his Core Data book.
John Bunting talks about different services Tumblr has built and how their architecture helps them be fault tolerant as they continue to grow.
John Bunting talks about different services Tumblr has built and how their architecture helps them be fault tolerant as they continue to grow.
You are looking at a Tumblr post, so you love GIFs. You are reading an engineering post, so you love bits. Have you ever wanted to know how Tumblr turns these bits into GIFs? Thanks to QCon, you can watch cyborg hacker codingjester talk about how its done.
We’ve experienced issues with iOS’s dynamic linker increasing our app’s launch time to unacceptable levels. To learn more about the linker’s inner workings and explore possible solutions, we created a Ruby script that generates dependency hierarchies of arbitrary complexity and tested their effect on launch times.
At Tumblr, we’re always looking for new ways to improve the performance of the site. This means things like adding caching to heavily used codepaths, testing out new CDN configurations, or upgrading underlying software.
Recently, in a cross-team effort, we upgraded our full web server fleet from PHP 5 to PHP 7. The whole upgrade was a fun project with some very cool results, so we wanted to share it with you.
Timeline
It all started as a hackday project in the fall of 2015. @oli and @trav got Tumblr running on one of the PHP 7 release candidates. At this point in time, quite a few PHP extensions did not have support for version 7 yet, but there were unofficial forks floating around with (very) experimental support. Nevertheless, it actually ran!
This spring, things were starting to get more stable and we decided it was time to start looking in to upgrading more closely. One of the first things we did was package the new version up so that installation would be easy and consistent. In parallel, we ported our in-house PHP extensions to the new version so everything would be ready and available from the get-go.
A small script was written that would upgrade (or downgrade) a developer’s server. Then, during the late spring and the summer, tests were run (more on this below), PHP package builds iterated on and performance measured and evaluated. As things stabilized we started roping in more developers to do their day-to-day work on PHP 7-enabled machines.
Finally, in the end of August we felt confident in our testing and rolled PHP 7 out to a small percentage of our production servers. Two weeks later, after incrementally ramping up, every server responding to user requests was updated!
Testing
When doing upgrades like this it’s of course very important to test everything to make sure that the code behaves in the same way, and we had a couple of approaches to this.
Phan. In this project, we used it to find code in our codebase that would be incompatible with PHP 7. It made it very easy to find the low-hanging fruit and fix those issues.
We also have a suite of unit and integration tests that helped a lot in identifying what wasn’t working the way it used to. And since normal development continued alongside this project, we needed to make sure no new code was added that wasn’t PHP 7-proof, so we set up our CI tasks to run all tests on both PHP 5 and PHP 7.
Results
So at the end of this rollout, what were the final results? Well, two things stand out as big improvements for us; performance and language features.
Performance
When we rolled PHP 7 out to the first batch of servers we obviously kept a very close eye at the various graphs we have to make sure things are running smoothly. As we mentioned above, we were looking for performance improvements, but the real-world result was striking. Almost immediately saw the latency drop by half, and the CPU load on the servers decrease at least 50%, often more. Not only were our servers serving pages twice as fast, they were doing it using half the amount of CPU resources.
These are graphs from one of the servers that handle our API. As you can see, the latency dropped to less than half, and the load average at peak is now lower than it’s previous lowest point!
Language features
PHP 7 also brings a lot of fun new features that can make the life of the developers at Tumblr a bit easier. Some highlights are:
Scalar type hints: PHP has historically been fairly poor for type safety, PHP 7 introduces scalar type hints which ensures values passed around conform to specific types (string, bool, int, float, etc).
Return type declarations: Now, with PHP 7, functions can have explicit return types that the language will enforce. This reduces the need for some boilerplate code and manually checking the return values from functions.
Anonymous classes: Much like anonymous functions (closures), anonymous classes are constructed at runtime and can simulate a class, conforming to interfaces and even extending other classes. These are great for utility objects like logging classes and useful in unit tests.
Various security & performance enhancements across the board.
Want to embed a tweet? Just paste the URL into a new link block on Tumblr. Or you can use the Share feature in other apps, like Twitter itself, to send the tweet URL to the Tumblr app.
You’ll get a perfect little screenshot of the tweet. You don’t have to flap your device around to help the image develop, but nobody is going to stop you.
Once again it was Hack Week (more than just a day!) at Tumblr!
This is getting repetitive in the best way. A couple of times per year
we slow down our normal work and spend a week working on scratching a
personal itch or features we want as user and see how far we can get
with our hacks. One thing from the last Hack Week in September made it all the way to a new experiment out to some testers: Tumblr Patio!
Here are some of the projects that got built for our most recent Hack
Week in January. Some of these things you may also end up seeing on the
site…
Spoiler text, spoiler blocks, and centered text!
This one is so obvious and amazing, it’s wild we don’t already have
it. For Hack Week, Katie added the ability to select text in a paragraph
to be hidden behind a wall of black that can be revealed with a tap.
This can be super useful to hide spoilers. And even better: whole
spoiler blocks. And while we’re here, the ability to center text!
A plethora of new default blog avatars
We haven’t updated our default avatars in several years. (Some of you may remember this one
from 10+ years ago.) They’re feeling a bit stale to us, so why not
update them? And while we’re at it… make a ton more variations! Paul
from the Tumblr Design team came up with a suite of new default avatars,
using our latest Tumblr color palette. Here’s a look at some of them, but there are actually many dozens more using different colors:
Notifications and emails about engagement on your posts
This one is for the folks on Tumblr who love numbers and their Activity
page. Daniel, @jesseatblr, and the Feeds & Machine Learning team
worked on some new notifications and emails we could send out to people
about how their posts have been doing lately on the platform, such as
how many views they’ve gotten, and by how many people. We already have
this available (and more) when you Blaze a post, but why not open it up
to more people? It’s really useful to the folks who use Tumblr to help
build an audience for their work!
A new way of navigating the web: the Command Palette
Some apps we use a lot have a “command palette” accessible via a
keyboard shortcut for quick keyboard-driven access to different parts of
the platform. For example, Slack and Discord have Command + K to access
their quick switchers to hop around conversations. What if Tumblr had
one? Kelly and Paul built one! Press Command/Control + K on Tumblr and
you can use your keyboard to jump to your blog, Activity, your recent
conversations, search, dozens of places!
As always, stay tuned to the @changes blog to see if any of these hacks make it on Tumblr for real!
Once again it was Hack Week (more than just a day!) at Tumblr! A couple of times per year we slow down our normal work and spend a week working on scratching a personal itch or features we want as user and see how far we can get with our hacks. One thing from the last Hack Day in March made it all the way to production: redesigning how direct messaging looks on Tumblr! Pretty cool!
Here are some of the projects that got made for this most recent Hack Week in September. Some of these things you may also end up seeing on the site…
Tumblr Patio
Maybe this will look familiar to you, but we love this idea of being able to organize Tumblr feeds into many “columns” side-by-side, creating a very dense but lively view of Tumblr. Lenny, Kelly, and Paul hacked this together, and we’re pretty excited to see where it’ll go. Each column can be a different feed on Tumblr, like For You, Following, your Activity, a specific blog, a search, Trending, even a Collection, so many possibilities!
Tumblr Booths
Meanwhile, a separate team of @autoplanes, Katie, @lex, Shaun, and Eve dug into the idea of selling digital and physical goods through blogs on Tumblr, leveraging our sibling platform WooCommerce! Blogs could put whatever they’d like for sale here, and have a dedicated space for it. We know there are so many amazing artists and artisans here who could use this to more easily sell their creations on Tumblr!
Avatar Frames/Hats
This one is a golden oldie, it keeps coming back hack day after hack day, and each time it gets even better. Santi and Maxime hacked together some example avatar “frames” and “hats” that folks on Tumblr could purchase for their blog. Maybe eventually people could create these and sell them or gift them to each other!
As always, stay tuned to the @changes blog to see if any of these hacks make it on Tumblr for real!
He’ll be doing a workshop on how to use the Tumblr API that will hopefully inspire you to use our dataset to create the next big application. If you use the Tumblr API, you’re eligible to win a $500 Amazon Gift Card for you and your team!
If you’re going to be there, give a shout out (John is @codingjester). John will have stickers and t-shirts for anyone who swings by. We hope to see you there!
