Explanation
Why the waste happens and who it affects.
Exactly-once guarantees that records are not dropped or duplicated, which requires Dataflow to deduplicate and checkpoint state along the pipeline. Google states that at-least-once mode, which allows occasional duplicates, can significantly reduce the cost and latency of a pipeline.
Many streaming pipelines do not need the stronger guarantee: map-only pipelines with no aggregations such as log processing, change data capture and ETL, pipelines whose sink deduplicates or is idempotent, and pipelines writing to sinks that cannot guarantee exactly-once delivery anyway, such as Pub/Sub. Reading from Pub/Sub is also significantly optimized in at-least-once mode. These jobs keep paying for exactly-once because it is the default and the mode is chosen once at launch.
Billing model
The pricing dimensions that drive this cost.
Streaming jobs bill per second, per job; rates vary by region.
- Streaming worker vCPU and memory
- Billed per vCPU-hour and GiB-hour for streaming workers
- Streaming Engine Compute Units
- With resource-based billing, Streaming Engine backend resources are metered and billed in Streaming Engine Compute Units
- Mode impact
- Exactly-once processing uses more worker and Streaming Engine resources than at-least-once for the same input
- Persistent Disk
- Billed per worker at normal rates, independent of mode
How to detect
4 checks to find it in your estate.
- Check the Streaming mode shown under Job info on the Dataflow Jobs page or job details panel for each streaming job
- Classify pipelines: flag those without aggregations (no counts, sums, means or windowed combines), those that are pure transformations, and those that write to Pub/Sub or to BigQuery through the Storage Write API
- Confirm whether downstream consumers already deduplicate (for example by a primary key or MERGE in BigQuery) or are idempotent
- Rank candidates by Streaming Engine Compute Unit and worker spend in the Cloud Billing export
How to fix
4 ways to remove the waste.
- Launch eligible pipelines with --dataflowServiceOptions=streaming_mode_at_least_once; the mode cannot be changed in place, so start a replacement job and drain or cancel the old one
- For BigQuery sinks, use the STORAGE_API_AT_LEAST_ONCE write method; at-least-once mode is not compatible with the FILE_LOADS method
- Keep exactly-once for pipelines with aggregations, business-critical results that must not double count, and non-idempotent transforms such as appending timestamps
- Enable Streaming Engine with resource-based billing, which at-least-once mode requires, and compare resource usage before and after the switch
Documentation
Vendor references for pricing and configuration.