Features

• Add cache hit or miss metrics
• Add coorindator specific terraform parameters
• Add data loading prefix allowlist parameter
• Add default PAS UDF
• Add E2E latency for GetKeyValues and GetKeyValueSet in sharded lookup
• Add file groups and file group reader logic
• Add go fmt to pre-commit
• Add key prefix support to blob storage client
• Add LogContext and ConsentedDebugConfiguration proto to v2 API and internal lookup API
• Add prod and nonprod build flag
• Add request context to wrap metrics context
• Add support for configuring directory allowlist
• Add wiring for prefix allowlist (actual impl in follow up cl)
• Allow overrides for coordinators endpoints in nonprod mode
• Allow to disable v1 key not found entry in response
• Create separate metrics context map for internal lookup server
• Deprecate metrics recorder for internal lookup
• Deprecate metrics recorder for internal server
• Deprecate metrics recorder for sharded lookup
• deprecate metrics recorder for V1 server and handler
• Deprecate metrics recorder from cache
• Enable simulation system send realtime udpates
• Enable TCMalloc for KV Server and benchmarks
• Explicitly enable core dumps
• Implement deletion cutoff max timestamp per directory
• Load data files and allow notifications from configured prefix
• Load prefix files on startup and handle prefix blob notifications
• Log common request metrics
• Migrate from glog to absl log
• Partition data loading metrics by delta file name
• Pass request context from hooks to downstream components
• Pass request context to udf hooks
• Read telemetry config from cloud parameter
• Revamp AWS metrics dashboard
• Revamp GCP metrics dashboard
• Set udf_min_log_level from parameter store.
• Support content type proto for v2 api
• Support content type proto for v2 api response
• Update cache interface and blob data location to pass prefix
• Update start_after to use a map from prefix to start_after
• Use file groups for loading snapshots
• Write logs to an Otel endpoint

Bug Fixes

• Actually load all files in a snapshot file group
• AWS: Filter out unavailable zones.
• Correct an error in kokoro_release.
• Correct format for image tag.
• Correct typo for internal dev's service_mesh_address.
• Correct typos in GCP deployment guide.
• Crash server if default UDF fails to load.
• Delete non-active certificate before creating a new one.
• Fix filtering logic for prefixed blobs
• Fix permissions for data-loading-blob-prefix-allowlist
• Make GCP nat optional.
• Parse delta filename from notification before validating it
• Remove glog dependency for record_utils
• Remove temp dir only if it's successfully created.
• Rename class to ThreadManager
• Set retain_initial_value_of_delta_metric flag for aws metrics exporter
• Update a outdated hyperlink.
• Update common repo to pick up the AWS metrics dimension fix
• Update GCP Terraform with ability to delete unhealthy instance.
• Update tf variables to use gorekore instead of kelvingorekore
• Use blob key instead of prefixed basename

GCP: Fixes

• GCP: Make sure server is connected to otel collector before reaching to ready state

GCP: Features

• GCP: Applying Terraform pulls docker image with new tag.
• GCP: Make service mesh address configurable.
• GCP: Make subnet ip cidr configurable.
• GCP: Make xlb/envoy optional.

Documentation

• Add ad retrieval explainer.
• Add docs for directory support
• Add PA and PAS folders
• Add PAS developer guide
• Add public docs for file groups
• Ads retreival explainer update.

Dependencies

• deps: Add clang-tidy bazel config
• deps: Add cpp_nowarn bazel config
• deps: Upgrade bazel to 6.5.0
• deps: Upgrade build-system to 0.55.1
• deps: Upgrade build-system to 0.55.2
• deps: Upgrade build-system to 0.57.0
• deps: Upgrade data-plane-shared repo
• deps: Upgrade data-plane-shared repo to 1684674 2024-02-09
• deps: Upgrade data-plane-shared-libraries to 1fbac46
• deps: Upgrade pre-commit hooks

Bug: N/A

Features

• Add an ability to add default tags
• Add AWS cpu utilization and memory utilization dashboards
• Add bazel configs prod and non_prod
• Add change notifier error count to the AWS metrics dashboard
• Add delta based realtime updates publisher
• Add delta record limited file writer
• Add GCP system metrics dashboard for kv server
• Add gcp_project_id flag when --platform=gcp is specified
• add go script for pushing gcp docker image
• Add optional logical_commit_time param to wasm macros.
• Add sharded realtime message batcher
• Add sharding data locality terraform parameters
• Add sharding support for publishing engine
• Add system metrics to log cpu and memory utilization
• Add UDF timeout parameter value to terraform
• Allow prefix path segments for gRPC http paths.
• Allow setting logging verbosity level through a parameter
• Data locality support
• Enable prettier pre-commit hook for JavaScript
• Expose http_api_paths tf variable to allow prefix matching
• Log error metrics for the change notifier
• Log metrics in the retries with new Telemetry API
• Migrate realtime metrics to new Telemetry API
• Remove envoy health checks
• Remove metrics recorder from blob storage client
• Remove metrics recorder from data orchestrator
• Remove metrics recorder from data reader
• Set UDF timeout from parameters
• Support passing in shard metadata via data_cli flags
• Update default tretyakov aws env settings
• Update v1 to return status per key

Bug Fixes

• Add all record values to switch cases
• add data-bucket-id back as a flag suffix.
• Add run_all_tests bazel config
• bazel config for emscripten no longer needs tool chain resolution
• Check grpc channel connection before sending requests
• deps: Upgrade data-plane-shared-libraries to 44d1d64 2024-01-08
• envoy.yaml permission and pre-commit fix
• Integrate RomaService changes from common repo
• Make some cache logging more precise.
• Missing absl deps
• Prevent Config from being copied into RomaService
• rectify kv_service_port in terraform.
• Remove jaeger as it has been deprecated
• Remove local platform for common repo and fix CloudPlatform import
• Update 'X-allow-fledge' header to 'Ad-Auction-Allowed' according to the spec.
• update GCP tentb environment
• Update UDFs to fix breakages due to FunctionBindingObjectV2
• Upgrade otel collector to the latest version

Dependencies

• deps: Upgrade data-plane-shared to commit f0d0b89 2023-11-14

GCP: Features

• GCP: Add external load balancer and envoy.
• GCP: store Terraform state remotely in GCS bucket

Documentation

• [Sharding] Add info about NUMA clusters
• Add docs for server cpu and memory profiling
• Add info on default wasm macro linkopts and memory limits.
• Add state transition sequence diagrams for aws/gcp sharded/nonsharded configurations
• Data locality
• Doc for aws private communication setup
• Sharding -- add a link to a tool that sets shard id in the delta metadata
• Specify --init flag for docker run commands

Bug: N/A

Features

• [API breaking change] Rename subkey to hostname.
• [Sharding] Add a tool to validate records
• [Sharding] Generate shard specific delta files
• Add base64 encoding to CSV delta cli tool
• Add safe metric definitions
• Add unsafe metric definitions
• Fully statically link the server binary
• Simplified UDF tester.

Bug Fixes

• Add error message when key not found.
• Add proxy visibility back to aws_artifacts
• allow internal ingress for otlp
• Clear errors in driver prior to parsing.
• Fix bash script flag for -v
• Fully qualify RegisterBenchmark calls.
• Package proxify layer on container_image
• Remove unused import httpbody.proto
• Update visibility target to public target in common repo
• Use set for cache key lookups

GCP: Features

• GCP: add capability to use existing service_mesh

Documentation

• Add a top level getting_started directory.
• Add docs about AWS AMI structure
• Add instructions on how to call UDF APIs from C++ WASM.
• Update sharding docs with GCP-specific info

Dependencies

• deps: Update data-plane-shared to b463f16
• deps: Utilize rules_closure deps via data-plane-shared

Bug: N/A

Features

• Add bazel config for code coverage
• Add proto definition for UDF input.
• Add string set support in udf delta tester.
• Add support to output graphviz dot file when playing with queries
• Move server_docker_image.tar to dist/
• Set bazel version to 6.3.2
• Set shard id label for metrics
• Update to latest PublicKeyFetcher

Bug Fixes

• Add seccomp flag to functional tests for local runs
• Allow changelog file to contain non-ascii chars
• bump the terraform version to v1.2.3
• Correct bazel config_setting visibility
• Fix blob_storage_client_gcp's polling frequency
• Fix excessive logging of unmock methods which cause flaky test in local
• Move server_docker_image.tar into dist dir

GCP: Features

• GCP: add GCP blob storage client.

Dependencies

• deps: Upgrade build-system to 0.45.0

Documentation

• [Realtime updates] Update docs to reflect GCP details
• Modify some sentences that I found hard to read.
• Sharding
• Sharding -- metrics

Bug: N/A

Features

• [Coordinators] moving /server/bin to /
• [GCP] Add realtime notifier tool
• [sharding] Add metrics for set query
• A delta based request generator that creates KV requests from delta files
• A generic grpc client that sends a request and returns a response
• Add ability to point to real coordinators through parameters
• Add bazel configs for roma legacy vs sandboxed
• Add bazel macro to generate UDF delta file from wasm binary and JS
• Add delta based request generator to the simulation system
• Add delta file notifier and loader to the request simulation system
• Add Dependencies section to release notes
• Add functions for serializing and deserializing shard mapping records
• Add GCP features and fixes sections in release notes
• add GCP platform and instance
• Add GCP realtime notifier
• Add GCP terraform config for metrics collector
• Add logical sharding config schema and constants
• Add metrics collector endpoint to the parameter
• Add metrics collector to periodically print and publish metrics
• Add NAT gateway for public internet access
• Add OSSF Scorecard badge to top-level README
• Add OSSF Scorecard GitHub Action
• add service mesh to GCP platform
• Add support to execute queries over sharded sets.
• Add tool to convert C++ to JS with inline WASM
• add unit tests to gcp parameter client
• Add version to UdfConfig.
• Adding GCP version of the message service
• Build AMI for request simulation system
• Coordinators: Add missing permission
• Deploy otel with request simulation system to AWS
• Enable metrics and tune the performance for request simulation system
• GCP terraform and parameter client refactor
• Generate synthetic requests at fixed rate
• grpc client worker to send requests at configurable QPS
• Integrating kv value server with the KeyFetcherManagerInterface
• Register get values hook for string and binary output format
• Run request simulation system in local
• Skip delta and snapshot files that belong to other shards
• Support writing shard mapping configs from csv
• Upgrade build-system to v0.33.0
• Upgrade data-plane-shared-libraries to 2023-07-12 commit.
• Upgrade data-plane-shared-libraries to 2023-07-21 commit.
• Upgrade data-plane-shared-libraries to 2023-07-26 commit.
• Upgrade data-plane-shared-libraries to 2023-08-16 commit.
• use local parameter client for gcp platform's local instance

Bug Fixes

• Acquire read lock on the set before iterating over it.
• Add action_env for asan bazel config
• Add noexcept bazel config
• Add seccomp-unconfined flag to build_and_test_all_in_docker
• Check that fb strings are not nullptr.
• Check the metrics collector connection during telemetry initialization
• Do not pass metrics collector endpoint for local and aws instance
• Don't move references in sample udf code.
• Explicitly cast return values of set operations to r-value references
• fix another asan error in the test
• Fix data loading num threads in param client local test
• Fix grpc client error
• Fix the error messages printed in the unit test logs
• Fix UDF function handler name
• logMessage should set an output string.
• make the response outlive grpc client call
• Rearrange bazel config for clarity
• Reduce the number of client workers to 2 in the unit tests to limit the number of threads created
• remove GRPC 4mb payload limit
• remove local variant of component tools.
• Remove redundant docker security-opt
• Set bazel workspace name
• Write data record for set data in the delta test file generator

Dependencies

• deps: Upgrade build-system to 0.42.1
• deps: Upgrade build-system to v0.41.1
• deps: Upgrade emscripten to 3.1.44

GCP: Features

• GCP: Add realtime thread pool manager

Documentation

• Add docs on getValuesBinary API
• Realtime directory is not optional for local dev
• Update docker run instructions to include security-opt flag
• Update inline WASM docs with instructions on how to test it
• Update screenshot of delta file

Bug: N/A

Bug: b/293901782

Features

• [Breaking change] Use UserDefinedFunctionsConfig instead of KVs for loading UDFs.
• [Sharding] Add hpke for s2s communication
• [Sharding] Allow for partial data lookups
• [Sharding] Making downstream requests in parallel
• Add bazel build flag --announce_rc
• Add bool parameter to allow routing V1 requests through V2.
• Add buf format pre-commit hook
• Add build time directive for reentrant parser.
• Add functions to retrieve instance information.
• Add internal run query client and server.
• Add JS hook for set query.
• Add lookup client and server for communication with shards
• Add MessageQueue for the request simulation system
• Add query grammar and interface for set queries.
• Add rate limiter for the request simulation system
• Add second map to store key value set and add set value update interfaces
• Add shard metadata for supporting sharded files
• Add simple microbenchmarks for key value cache
• Add UDF support for format data command.
• Add unit tests for query lexer.
• Adding cluster mappings manager
• Adding padding
• Apply custom lockings on the cache
• Connect InternalRunQuery to the parser
• Extend and simplify collect-logs to capture test outputs
• Extend use of scp deps via data-plane-shared repo
• Implement shard manager
• Move sharding function to public so it's available for file sharding
• Register a logging hook with the UDF.
• Register run query hook with udf framework.
• Sharding - realtime updates
• Sharding read flow fixes
• Simplify work done in set operations. Set operations can be passed by
• Snapshot files support UDF configs.
• Support reading and writing set queries to data files.
• Support reading and writing set values for csv files
• Support reading/writing DataRecords. Requires new DELTA format.
• Support writing sharded files
• Update data_loading.fb to support UDF code updates.
• Update pre-commit hook versions
• Update shard manager mappings continuously
• Upgrade build-system to release-0.28.0
• Upgrade build-system to v0.30.1
• Upgrade scp to 0.72.0
• Use Unix domain socket for internal lookup server.
• Utilize AWS deps via data-plane-shared repo

Bug Fixes

• Add internal lookup client deadline.
• Catch error if insufficient args specified
• Fix aggregation logic for set values.
• Fix ASAN potential deadlock errors in key_value_cache_test
• Proper memory management of callback hook wrappers.
• Specify 2 workers for UDF execution.
• Upgrade pre-commit hooks
• Use shared pointer for UDF absl::Notification.

Build Tools: Fixes

• build: Add scope-based sections in release notes

Documentation

• Add docs for data loading capabilities
• Add explanation that access control is managed by IAM for writes.
• Point readme to a new sharding public explainer

Bug: 290798418

Features

• Add --fast option for running functional tests
• Add a multi-KV SUT
• Add baseline for SSP
• Add bazel macro to generate OpenSSL certificates
• Add brotli compression support
• Add envoy for KV server in baseline SUT
• Add envoy frontend in baseline SUT
• Add optional filename arg to tools/collect-logs
• Add SUT without UDF
• Add terraform support for sharding
• Drop data meant for other shards
• Include tink_cc repository
• Set timestamp for generated delta files for SUTs
• Support selection of single SUT for testing
• Upgrade gRPC to v1.52.1
• Upgrade to functionaltest-system 0.5.1
• Use 443 for http1.1 and 8443 for http2.
• Use scp deps exposed by data-plane-shared-libraries repo
• Use shared control plane skylark functions for transitive dependencies

Bug Fixes

• Address linter issues in production dir
• Address linter issues in tools
• Avoid running prettier on PCR0 files
• Capture test logs into zip archive
• Collect logs from functional testing
• Configure docker compose to pull images quietly
• Correct container deps in multiple-kv-servers
• Defer to data-plane-shared cpp deps
• Remove exposed ports as not needed within docker network
• Remove unused code in tools/get_workspace_status
• Remove workspace parent dirs from logs archive
• Upgrade pre-commit hooks

Documentation

• Add Protected Audience API rename banner

Bug: 280464254

Features

• Add a total realtime QPS metric
• Add aws supplied e2e latency
• Add basic UDF functional tests for v2
• Add error counters for realtime updates
• Add functional test stubs for v2
• Add target to generate delta for sample udf.js
• Add test data artifacts to dist/test_data/deltas
• Add UDF delta file generator tool.
• Add UDF delta file upload through terraform config.
• Add udf.js delta file to test set
• Upgrade to build-system 0.22.0 and functionaltest-system 0.2.0

Bug Fixes

• Add a dashboard for environments
• Add documentation for editing dashboards
• Change envoy log level to debug
• Check that recovery function is valid before calling it.
• Enable docker network cleanup
• Ensure changelog notes use specific version
• ignore interestGroupNames argument
• MetricsRecorder no longer a singleton.
• MetricsRecorder now optional for retry templates.
• Return missing key error status from internal lookup server.
• Upgrade gRPC and make lookup client a singleton
• Use dynamic_cast to get metric_sdk::MeterProvider provider.

Terraform

• Add us-west-1 terraform

Documentation

• Add documentation on roma child processes.
• Add instructions on realtime updates
• Add note to use grpcurl for v2 since http has a bug.
• Add v2 request JSON schema
• AWS realtime update capabilities
• Correct udf target name
• Update documentation for building data cli.
• Update realtime metrics querying docs

Bug: 277075104

Features

• add AWS SQS ReceiveMessage latency histogram
• Add command line flags for parameters
• Add configurable thread count for realtime updater
• Add functional test
• Adding e2e latency measurement for realtime updates
• Allow specifying explicit histogram bucket boundaries
• Allow the blob_storage_util cp command to work with local files.
• Allow the DeltaFileRecordChangeNotifier to read local files as well as from S3
• Batch delete SQS messages
• Build delta files from csv
• clean up realtime queues
• Configure AWS hosted Prometheus.
• Disable the use of exceptions
• Enhance/Simplify local export of telemetry with OTLP.
• Functional testing of local server with delta files
• make AwsSnsSqsManager thread safe
• Make the blob_storage_change_watcher tool work for local files
• Make the blob_storage_util cat and rm commands work for local files
• Make the blob_storage_util ls command work for local files and refactor out common parts from the AWS binary
• Make the delta_file_watcher tool work for local files
• Move the platform specific server configuration logic to a separate file
• multi-threaded realtime notifier
• realtime tester in a container
• Reuse SQS client
• Speed up test updates publisher
• Tools for generating and inserting realtime test data.
• Upgrade build-system to release-0.18.0
• Upgrade build-system to release-0.20.0
• Upgrade debian runtime images to 15 Feb 2023
• Upgrade to build-system 0.17.0
• Use a PlatformInitializer so the data_cli will compile for --platform=local

Bug Fixes

• Add ability to interrupt a SleepFor Duration.
• Add minimum shard size threshold for concurrent reader.
• Launch Envoy first before all other processes.
• Make MetricsRecorder a shared global instance.
• Only run functional tests against local server
• Remove functionaltest/run-server, update docs accordingly
• Remove submodule section from docs
• Run server in background, and reduce noise
• Run server using delta files in run-server-docker
• Update generate_load_test_data to use bazel-debian
• Use symlink for identical test replies
• Use VLOG for concurrent reader debugging logs.
• Wait for envoy to respond before launching enclave service.

Documentation

• Add a playbook template for new alerts
• Add description for backup_poll_frequency_secs
• Add docs about how to run the server locally
• fix /tmp/deltas discrepancy
• remove obsolete service
• Updating instructions on how to copy eif manually.

Terraform

• Convert tfvar files to json
• Support Prometheus service running in a different region