Introducing Pinqueue3.0, Pinterest’s next-gen content moderation platform
By James Man | Trust & Safety Engineering
In 2016, we reached a milestone of launching Pinqueue2.0, which has since become the go-to content moderation tool for the engineering, community, and data science teams. One of our top priorities at Pinterest is fostering a safe and trustworthy experience for all Pinners. As Pinterest’s user base and ads business grow, the review volume has been increasing exponentially, and more content types require moderation support. To solve greater engineering and operational challenges at scale, we need a highly-reliable and performant system to detect, report, evaluate, and act on abusive content and users.
Why Pinqueue3.0?
Pinqueue2.0 doesn’t have enough effective features to support daily operations, and it also suffers from subpar performance and usability problems. The design of Pinqueue2.0 makes it too rigid to allow new features like configurable templates or hotkeys. We decide to bring content moderation to the next level with a significant rewrite named Pinqueue3.0. It’s designed and built to achieve the following goals:
- More self-serve functionalities: With minimal engineering effort, team leads should be able to manage things like how the review queue is set up, who can access the reviews and setting any arbitrary hotkey preferences.
- Easy to extend to new review formats: Our primary review source used to be objectionable content, but now we need to adapt to emerging trends of various forms of deception and misinformation. At the same time, more content types and media formats other than images are being introduced to Pinterest (e.g., carousel promoted pins and video pins). Greater coverage of moderation support is needed.
- Efficient and performant review experience: Review time and overall efficiency are key metrics to measure the success of the app. Any productivity improvement would result in lower operational costs and higher user satisfaction.
- Stable and scalable infra with proactive monitoring and alerting: Good infrastructure is the foundation of everything. The platform should stay monitored and available 24/7 to keep all Pinners safe.
Overview
Pinqueue3.0 serves as a generic platform for content moderation and human labeling. Different types of review events are categorized into different groups and queues. Real-time operational metrics are available to help users prioritize and allocate review resources accordingly. The review page is divided into two blocks: the review widgets and the action widget. Though the review UI and action templates vary in different review use cases, they enjoy the same native features like hotkeys and item passing.
Under the hood, Pinqueue3.0 is a Flask + ReactJS app powered by Pinterest’s very own Gestalt UI framework. On the backend, Pinqueue3.0 heavily relies on PinLater, a Pinterest-built reliable asynchronous job execution system, to handle the requests for enqueueing and action-taking. Using PinLater has significantly strengthened Pinqueue3.0’s overall infra with its capability of processing a massive load of events with configurable retry policies.
Pinqueue’s content moderation review cycle involves four major processes:
- Receiving events for review: Pinqueue is agnostic to the source of the reviews. Via Pinqueue’s highly robust enqueue API, any upstream services can send valid review events.
- Fetching and displaying data: Data is fetched from various sources that aren’t necessarily managed by Pinqueue. Online metrics are collected to safeguard and monitor the health of the communications.
- Acting upon decisions: Pinqueue offloads all the action executions to PinLater. Action-related information such as who makes the decision or when the decision is made will be logged securely and appropriately.
- Preserving the data: Ultimately, we dump all the decisions data to Hive. Human review data are valuable assets for offline operational analysis, review quality auditing and further machine learning efforts.
Object-based architecture
Today, Pinterest supports multiple types of user-generated content entities: Pin, board, user, comment, tried-it, and message. Pinqueue must provide content moderation support for all the primary entities, and it also includes other relevant entities (e.g., the owner of the Pin) to make the review more data-informed. For example, the behavior of the owner can be one of the determining factors to decide whether a Pin is spam. The more suspicious the owner’s behavior is, the more spammy the Pin is likely to be, so it’s great to check both together. In Pinqueue3.0, we refer to all the to-be-reviewed entities as “objects”.
In practice, the Trust & Safety review may consist of the reported Pin, the owner of the Pin, the associated board, and the reporter. The merchant-specific review uses a different set of objects: the merchant, its recent Pins and boards. However, a spam review may share a subset of objects (Pin or board) like the Trust & Safety review and another subset of objects (most recent Pins or recent boards) like the merchant review.
As you can see, different review scenarios are essentially reviewing different combinations of “objects”. Each object primarily shares the same aspects throughout the whole review cycle:
- Data Fetching: Regardless of the data sources, each object must have a corresponding data fetcher to know where and how to obtain the data.
- UI Presentation: Fetched data should be displayed in a user-friendly way, ideally presented by a group of standardized, reusable, and extendable UI components.
- Decision Making: Once a decision is made against an object, it should talk to the targeted downstream services and make sure the action is executed and logged.
Pinqueue3.0 provides a modularized framework of object-specific abstractions. It allows developers to easily extend content moderation support for any review formats or objects. The only requirements are defining the UI template, configuring the actions with the Queue Config tool and building the data fetching logic. We’ve built infrastructure to manage objects’ life cycles and connect each object with its corresponding handlers, hence a review event can accommodate any arbitrary combinations of supported objects. Below is a flowchart of a sample review event. We will discuss the Template Engine and the Queue Config in the next section.
Dynamic UI rendering with Template Engine
Building with ReactJS and JSX enables us to pass around the tree structures composed of HTML or React elements as if they were standard JavaScript values. By leveraging JSX, we can transform each review page as individual templates. Pinqueue’s template engine has three components: template Editor, template parser and template renderer. Given the raw template code produced by the editor, the template parser parses the syntax and determines which widgets to display. Ultimately, the renderer pulls up the matching UI widgets from the widgets library and renders them in an organized layout.
Why do we need a template engine? First, it allows us to easily define the layout of the review pages and specify what components to use. Human eyes tend to pay more attention to the center of the page. The positions of the widgets act as a natural guide to pinpoint the most critical information, and we give users the flexibility to manage it. It also makes a per-agent template possible. Secondly, it’s extendable for future improvements. UI widgets can be formed by essential elements like images, video, table or heading. We plan to make components more generalized and abstracted. Users can define their widgets by assembling the basics and configure which fields to display.
Make full self-service possible with Queue Config
As a generic platform, Pinqueue3.0 is built to support all kinds of use cases. When reviewing the same abusive content, an operations team may expect to take the content down while ML engineers may only want to label it. Without sophisticated self-service capabilities, it’ll be hard to accommodate all needs. While the Template Engine controls the UI, Queue Config covers more review-specific specs like which actions or labels will be available, which hotkey and template to use and other miscellaneous configs. Queue Config is reusable and shareable, and it warrants a unified and manageable review experience across queues.
Pinqueue3.0 maintains a list of supported actions for each object. Review leads can easily configure what actions may be taken against the objects on the review page. Even when multiple queues are reviewing the same objects, their decision options may be different.
Solely relying on human power to solve trust & safety issues is not scalable. Future automation and machine learning efforts can benefit from high-quality human-curated labels. We highly value the importance of labeling, so we make labeling a built-in feature with Queue Config. As of now, the label is a free-form string. In addition to the hotkey support, there are other advanced configs like whether labels should be optional or mandatory as well as whether single or multiple labels are allowed.
Behind the scenes, the queue config is saved as a well-structured JSON. Using JSON makes it extendable to add and manage new configs. The review page strictly complies with the queue configuration and renders the actions/labels accordingly on the menu.
We believe the combination of the Template Engine plus Queue Config is the key to unlocking more self-service capabilities. We plan to make the template editing more intuitive by introducing drag-and-drop functionality. Queue Config continues to play a critical role in boosting review efficiency and operational productivity.
Other advanced features
Kitty Mode: Sometimes users want to take a screenshot/video to test or reproduce a bug without capturing graphic content. When the kitty mode is on, all images turn to kittens.
Review Details: For training purposes or further auditing reviewers’ review quality, the review team needs a way to know who made the decision, when the decision was made and which object the action was taken against. The review details page is a one-stop shop that retrieves and displays the full history of decisions related to certain objects or a particular item.
Passing: We often see misrouted items ending up in the wrong queues due to mis-selected report reasons(e.g., a spam review turns out to be a porn review, or users encounter a review in a foreign language). As each queue is managed by a group of well-trained reviewers who know the policy best, users could pass an item to a different queue to ensure proper handling.
What’s next
Hundreds of millions of people around the world use Pinterest to discover and do what they love, and our job is to protect them from abusive and harmful content. We’re committed to providing an inspirational yet safe experience to all Pinners. Solving trust & safety problems is a joint effort requiring expertise across multiple domains. Pinqueue3.0 not only plays a critical role in responsively taking down unsafe content, it also has become an enabler for future ML/automation initiatives by providing high-quality human labels. Going forward, we will continue to improve the review experience, measure review quality and collaborate with our machine learning teams to solve content moderation beyond manual reviews at an even larger scale.
Acknowledgments: Huge thanks to the entire Trust & Safety tooling team who helps in improving Pinqueue and keeping Pinners safe. Special thanks to David Su, Becky Stoneman, Patricia Perozo, Alok Singhal and Dennis Horte for their feedback and editing assistance.
