Environment details

1. OS type and version: GCP / GKE
2. Java version: 21+
3. bigtable version(s): 2.47.0

Steps to reproduce

1. Deploy an application to GKE in a common project (e.g. mycompany-pro)
2. Use a custom project for Bigtable resources (e.g. mycompany-myapp-pro)
3. Use a service account that does have monitoring.timeSeries.create permissions on the custom project, but not on the common project
4. Enable client side metrics (enabled by default)

Code example

// Create a default BigtableDataClient
BigtableDataSettings
            .newBuilder()
            .setProjectId(bigtableProperties.projectId)
            .setInstanceId(bigtableProperties.instanceId)
            .setAppProfileId(bigtableProperties.appProfileId)

Stack trace

createServiceTimeSeries request failed for bigtable metrics. Need monitoring metric writer permission on project= ... . Follow https://cloud.google.com/bigtable/docs/client-side-metrics-setup to set up permissions.
com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: Permission monitoring.timeSeries.create denied (or the resource may not exist).

External references such as API reference guides

• https://cloud.google.com/bigtable/docs/client-side-metrics-setup

Any additional information below

This seems to happen because in BigtableCloudMonitoringExporter different metrics are created for Bigtable resource metrics and what it calls application resource metrics. For Bigtable resource metrics, the Bigtable project is used and this is fine. However, the project for the application resource metrics is not customizable. The only metric it tries to record there seems to be per_connection_error_count. If the service account does not have the permission to create the time series in this common project, you receive the error above.

I'm not sure why the application resource metrics are needed for per_connection_error_count and why it defaults to using the project the application is running in instead of the pre-specified Bigtable project (which may be different).

Solving this could be done by either using the specified project, removing the application resource metrics or something else entirely.