Skip to content

privacysandbox/protected-auction-key-value-service

 
 

Repository files navigation

Important

The main branch hosts live code with latest changes. It is unstable and is used for development. It is suitable for contribution and inspection of the latest code. The release-* branches are stable releases that can be used to build and deploy the system.


FLEDGE has been renamed to Protected Audience API. To learn more about the name change, see the blog post.


Privacy Sandbox Logo FLEDGE Key/Value service

Background

FLEDGE API is a proposal to serve remarketing and other custom-audience ads without third-party cookies. FLEDGE executes the ad auction between the buyers (DSP) and the sellers (SSP) locally, and receives real-time signals from the FLEDGE K/V servers. To learn more about

When the auction is executed, separate FLEDGE K/V servers are queried for the buyers and sellers. When a buyer is making a bid, the DSP K/V server can be queried to receive real-time information to help determine the bid. To help the seller pick an auction winner, the SSP K/V server can be queried to receive any information about the creative to help score the ad.

State of the project

The current codebase represents the initial implementation and setup of the Key/Value server. It can be integrated with Chrome and Android with the Privacy Sandbox unified origin trial and Privacy Sandbox on Android Developer Preview. Our goal is to present the foundation of the project in a publicly visible way for early feedback. This feedback will help us shape the future versions.

The implementation, and in particular the APIs, are in rapid development and may change as new versions are released. The query API conforms to the API explainer. At the moment, to load data, instead of calling the mutation API, you would place the data as files into a location that can be directly read by the server. See more details in the data loading guide.

Currently, this service can be deployed to 1 region of your choice. Multi-region configuration is up to the service owner to configure.

Current features

Build and deployment

  • Source code is available on Github
  • Releases are done on a regular basis
  • Binaries can be built from source code
    • C++ binary
    • [AWS & GCP] Docker container image
    • [AWS]: Nitro EIF
    • [AWS]: Reference AMI
    • Other tools
  • Server can run as a standalone local process for testing without any cloud dependency or TEE-related functionality
  • Server can be deployed to AWS Nitro enclave
  • Server can be deployed to GCP Confidential Space
  • Reference terraform available for a clean and comprehensive deployment to AWS or GCP
    • Clean: assumes the cloud environment has no preset infrastructure
    • Comprehensive: deploys all dependencies and some recommended (but not necessarily required) configuration
    • Many server behaviors can be configured by parameter flags without having to rebuild

Data loading

  • Server loads key/value data from cloud file storage
  • Server loads key/value data from cloud pub/sub services
  • Server loads data into an in-RAM representation for query serving
  • Server continuously monitors for new data and incrementally updates ("delta files") the in-RAM representation
  • Support independent data ingestion pipelining by monitoring directories in cloud file storage independently
  • Supports Flatbuffers as the data event format
  • Supports Avro and Riegeli as the data file format
  • Supports snapshot files for faster server start up
  • Users can perform compactions of delta files into snapshot files in an offline path

Read request processing

  • Support Protected Audience Key Value Server query spec: can be used as a BYOS server to serve requests from Chrome
  • Support simple key value lookups for queries
  • Users can write "user defined functions" to execute custom logic to process queries
  • User defined functions can be written in JavaScript or WASM
  • User defined functions can call "GetValues" to look up key value from the dataset

Advanced features

  • Set-as-a-value is supported
    • A key "value" pair in the dataset can be a key and a set of values
  • UDF can call "RunQuery" API to run set operations on sets (intersection, union, difference)
  • For GCP, Terraform supports deploying into an existing VPC, such as used by the Bidding and Auction services Non-prod Server logs are persisted after server shutdown
  • Data can be sharded and different servers may load and serve different shards (subset) of the dataset.
  • Sharding supports data locality, where the operator specifies "sharding key" for key value pairs so different key value pairs can have the same sharding key.

Timeline and roadmap

The following sections include the timelines for the Trusted Key Value Server for Protected Auctions. Protected Auctions refer to Protected Audiences and Protected App Signals ad targeting products.

Timelines

Beta testing General availability Enforcement
For Protected Audience

(web browser on desktop)

July 2024

The Privacy-Sandbox-provided Key Value Server implementation can

  • run as a BYOS KV server
  • support production scale traffic and common functionalities
Q4 2024

Opt-in TEE mode will be available to the Adtechs. Opt-in guidance will be published in early Q4 2024.

No sooner than Q3 2025
Beta testing General availability
For Protected Audience

(With Bidding & Auction services for Chrome or Android)

July 2024

The Privacy-Sandbox-provided Key Value Server implementation can be used with the Bidding and Auction services and

  • run as a BYOS KV server
  • support production scale traffic and common functionalities
Dec 2024

The Privacy-Sandbox-provided Key Value Server implementation can be used with the Bidding and Auction services and adtechs can opt-in TEE mode

For Protected App Signals June 2024

The Privacy-Sandbox-provided Key Value Server implementation supports Ad retrieval server functionality and protected communication for live traffic testing

Dec 2024

The implementation supports live traffic at scale

Roadmap

June 2024 Beta release

Deployment and Setup
  • For AWS, Terraform supports deploying into an existing VPC, such as the one that is used by the Bidding and Auction services
  • Internal load balancer is used for servers to send metrics to OpenTelemetry collector
    • In v0.16, the communication goes through a public load balancer
Integration with the Bidding & Auction services
  • The Bidding and Auction services can send encrypted requests to the Key Value Server for Protected App Signals
Debugging support
  • Consented Debugging is supported
  • Diagnose tool to check the cloud environment to warn for potential setup errors before the system is deployed
  • Operational playbook
  • Introduction of unsafe metrics
    • Unsafe metrics have privacy protections such as differential privacy noises
    • More metrics for comprehensive SLO monitoring
Runtime features
  • Data loading error handling
    • The system can be configured to use different error handling strategy for different dataset
Performance/Cost
  • Benchmarking tools
  • Cost explainer
  • Sets-as-values will switch to using bitsets to represent sets for faster RunQuery performance.
Support process
  • Commitment to support window for active releases

Q4 2024 Chrome-PA GA

Chrome integration
  • Update to V2 protocol to support the hybrid mode of BYOS & Opt-in TEE
  • Chrome and Key Value server can communicate in the updated V2 protocol
  • Chrome can send full publisher URL to TEE KV server under V2 protocol

H2 2024 Android-PA GA, PAS GA

User Defined Functions
  • UDF can perform Key/Value lookup asynchronously
  • Flags can be passed from the server parameters into UDF
  • One Key Value Server system can be used for multiple use cases. Multiple UDFs can be loaded. Different UDF can be selected based on the request type.
  • Canaring support for UDF: canary version UDF can be staged in machines with specific tags.
Customization support
  • First class support for customization of the system (without violating the trust model)
Debugging support
  • Diagnose tool to collect standard and necessary debug information for troubleshooting requests
Documentation
  • Complete end to end example as a template to set up the service

Breaking changes

While we make efforts to not introduce breaking changes, we expect that to happen occasionally.

The release version follows the [major change]-[minor change]-[patch] scheme. All 0.x.x versions may contain breaking changes without notice. Refer to the release changelog for the details of the breaking changes.

At GA the version will become 1.0.0, we will establish additional channels for announcing breaking changes and major version will always be incremented for breaking changes.

Key documents

Contribution

Contributions are welcome, and we will publish more detailed guidelines soon. In the meantime, if you are interested, open a new Issue in the GitHub repository.

Feedback

The FLEDGE K/V feature set, API and system design are under active discussion and subject to change in the future. If you try this API and have feedback, we'd love to hear it:

Announcements

General announcements

General announcements will be made on the FLEDGE mailing list on chromium.org and the Privacy Sandbox progress update page on the Android developer site, if needed, an accompanying article will be published on the Chrome Developers blog and Android Developers blog.


OpenSSF Scorecard