Explanation
Why the waste happens and who it affects.
They are disabled by default for all services except some BigQuery services, and Google warns that their volume can be large and that enabling them might result in charges for the additional logs usage. Unlike Admin Activity logs, which go to the free _Required bucket, Data Access logs are stored in the _Default bucket and are billed as regular Cloud Logging storage.
The costly pattern is enabling Data Access logs for all services and all permission types at the organization or folder level, often to satisfy an audit requirement, instead of only for the services and principals that the requirement actually covers. Every read and write by applications and service accounts against high-traffic services then produces a log entry. Google's Bigtable documentation warns that Data Access logging can generate a very high volume of entries and states that managing service account logs is the most important step to reduce log volume; once a parent enables these logs, child folders and projects cannot turn them off.
Billing model
The pricing dimensions that drive this cost.
Data Access audit logs are billed like other logs stored in log buckets.
- Logging storage
- Billed per GiB streamed into log bucket storage, including up to 30 days of retention, after a free allotment of 50 GiB per project per month
- Logging retention
- Logs kept longer than 30 days are billed per GiB per month
- _Required bucket
- Admin Activity and System Event audit logs are stored free of charge; Data Access logs are not stored there
- Routed destinations
- Logs routed to Cloud Storage, BigQuery or Pub/Sub incur the destination's charges instead of log bucket storage
How to detect
4 checks to find it in your estate.
- Inspect the auditConfigs section of the IAM policy at organization, folder and project level (gcloud organizations get-iam-policy, gcloud resource-manager folders get-iam-policy, gcloud projects get-iam-policy) for service allServices with DATA_READ, DATA_WRITE or ADMIN_READ enabled
- In the console IAM, Audit Logs page, review which services have Data Access log types enabled and whether any exempted principals are configured
- Measure the volume of logs with logName containing cloudaudit.googleapis.com%2Fdata_access in the _Default bucket, by service and by principal (protoPayload.authenticationInfo.principalEmail), to find the services and service accounts generating most of the volume
- Compare Cloud Logging storage charges per project in the Cloud Billing export before and after the date Data Access logging was enabled
How to fix
5 ways to remove the waste.
- Replace organization-wide allServices configurations with service-specific configurations covering only the services and permission types your audit policy requires; Google recommends enabling Data Access logs where possible, so the aim is scoping, not switching them off, and a broader parent configuration cannot be disabled at the folder or project level
- Exempt high-volume service accounts that do not need to be audited through the audit configuration's exempted principals, which Google recommends over exclusion filters because it prevents the logs from being generated at all
- Where logs must be generated but not kept in Cloud Logging, add exclusion filters to the _Default sink or route them to a lower-cost destination such as a Cloud Storage bucket with a lifecycle policy
- Estimate the log volume before enabling Data Access logs on high-throughput services such as Bigtable, Cloud Storage or Spanner, using request rates and average entry size
- Set retention on the buckets holding audit logs to what the compliance requirement specifies, since retention beyond 30 days is billed monthly
Documentation
Vendor references for pricing and configuration.