Explanation
Why the waste happens and who it affects.
The Storage Write API (REST), previously known as the legacy tabledata.insertAll method, bills streamed data per 200 MiB and counts every row as at least 1 KB. The Storage Write API over gRPC bills per GiB at a lower rate and includes the first 2 TiB per month free. Google's documentation states that gRPC streaming has a significantly lower cost than REST and recommends it for new workloads; in us-central1 the pricing page lists $0.01 per 200 MiB for REST and $0.025 per GiB for gRPC.
Pipelines written against insertAll, and client code that calls insertAll or insert_rows, still use the REST path. Small events make the gap larger, since each REST row is billed at a 1 KB minimum even when the payload is only a few hundred bytes. For data that does not need to be queryable within seconds, batch load jobs are cheaper still: batch loading is free on the shared default-pipeline slot pool, with only storage billed afterward.
Billing model
The pricing dimensions that drive this cost.
BigQuery ingestion is billed differently by path; prices below are the pricing page's us-central1 (Iowa) rates.
- Storage Write API (REST)
- Streaming inserts billed at $0.01 per 200 MiB for successfully inserted rows, with each row calculated at a 1 KB minimum size
- Storage Write API (gRPC)
- Billed at $0.025 per GiB ingested, with the first 2 TiB per month free
- Batch loading
- Free when load jobs run on the shared default-pipeline slot pool; data is then billed as storage
How to detect
5 checks to find it in your estate.
- Query INFORMATION_SCHEMA.STREAMING_TIMELINE, which reports per-minute statistics for streaming through the Storage Write API (REST), and sum total_input_bytes and total_rows by table over 30 days to size REST streaming volume
- Estimate the effect of the 1 KB row minimum by comparing total_rows multiplied by 1 KB with total_input_bytes; a large gap means small rows are being billed as larger ones
- In the Cloud Billing export, group BigQuery charges by SKU description to compare streaming insert charges with Storage Write API charges per project
- Search pipeline code and connector configuration for tabledata.insertAll or client calls such as insertAll or insert_rows that use the REST path
- Identify streaming writers whose downstream consumers only read the data hourly or daily, which could use batch loads instead
How to fix
4 ways to remove the waste.
- Migrate high-volume streaming writers to the Storage Write API over gRPC; Google recommends the default stream when migrating from the REST path, which gives at-least-once semantics similar to insertAll
- Use committed streams with offsets where exactly-once delivery is needed, which gRPC supports and REST does not
- Move latency-tolerant ingestion to batch load jobs from Cloud Storage, which are free on the shared slot pool, accepting that the shared pool has no throughput guarantees
- Validate the change with a pilot table and compare billing SKUs before and after; the REST path remains fully supported, so migrate the highest-volume writers first
Documentation
Vendor references for pricing and configuration.