Messaging User Group Meetup

This is the first meetup around messaging technologies (e.g. queuing, pub/sub, write-ahead logging, data pipeline). It is proposed by Apache BookKeeper community to bring people from different companies and Apache projects together to discuss and share technologies around messaging, share experience using open source projects like Apache BookKeeper and Apache Kafka. Join us for an evening of food, drinks and talks to learn more!

Agenda:

5:30 PM - 6:00 PM // Check-in & Network & Food/Drinks

6:00 PM - 6:05 PM // Opening and Welcome (Twitter)

6:05 PM - 6:35 PM // “Build high-throughput, low-latency DistributedLog” (Twitter)

6:35 PM - 7:05 PM // “Apache Kafka: Leveraging real-time data at scale in practice” (Confluent)

7:05 PM - 7:15 PM // Break

7:15 PM - 7:45 PM // “Cloud Messaging Service, technical overview” (Yahoo!)

7:45 PM - 8:00 PM // Discussions & Network

NOTE: Doors open at 5:30 PM and closed at 6:30 PM.

Build high-throughput, low-latency DistributedLog (Sijie Guo @sijieg / Twitter)

Systems like Databases and Messaging systems require durability. One common way to implement durability while keeping performance high is to use a Log to persist updates to system states. The Log is used to reconstruct the system state in the event of a crash. If the Log is only stored locally, the system state is permanently lost when the server writing log experiences a permanent hardware failure. DistributedLog is a replicated log service that is built on top of Apache BookKeeper, providing infinite ordered append-only streams that could be used for building robust distributed systems. It has been widely used at Twitter, in distributed database, search pipeline and real-time messaging systems.

In this talk Sijie will share details about DistributedLog and describe its production usage.

Sijie Guo is the tech lead of DistributedLog/BookKeeper project at Twitter, and the PMC chair of Apache BookKeeper.

Apache Kafka: Leveraging real-time data at scale in practice (Neha Narkhede @nehanarkhede / Confluent)

Since it was open sourced, Apache Kafka has been adopted very widely from web companies like Uber, Netflix, LinkedIn to more traditional enterprises like Cerner, Goldman Sachs and Cisco. At these companies, Kafka is used in a variety of ways - as a pipeline for collecting high-volume log data for load into Hadoop, a means for collecting operational metrics to feed monitoring and alerting applications, for low latency messaging use cases and to power near realtime stream processing.

Kafka’s unique architecture allows it to be used for real-time processing as well as a bus for feeding batch systems like Hadoop. Kafka is fundamentally changing the way data flows through an organization and presents new opportunities for processing data in real time that were not possible before. The biggest change this had led to is a shift in the way data is integrated across a variety of data sources and systems.

In this talk, I will discuss how companies are using Apache Kafka, where it fits in the Big Data ecosystem and how it serves as a substrate for large-scale stream processing.

Neha Narkhede is co-founder and head of engineering at Confluent. Previously, she was responsible for LinkedIn’s petabyte scale streaming infrastructure supporting hundreds of billions of events per day. She is also one of the initial authors of Apache Kafka and serves as a PMC member and committer for the project. In the past, she has worked on search within the database at Oracle and holds a Masters in Computer Science from Georgia Tech.

Cloud Messaging Service, technical overview (Matteo Merli @merlimat / Yahoo!)

In this talk, Matteo will describe the Yahoo!’s Cloud Messaging Service. He would go through design decisions, how they reached that and how they leverage Apache BookKeeper to implement a multi-tenant messaging service.

Matteo Merli is the tech lead for Cloud Messaging Service project and he has been at Yahoo! for 5 years. Previously maintaining Sherpa (Yahoo!’s distributed database) replication layer, Tribble. He is one of BookKeeper committers.