Explanation
Why the waste happens and who it affects.
AWS positions on-demand for unpredictable, highly variable traffic and provisioned mode for predictable traffic that is easy to forecast. Many streams are created on-demand for convenience and stay there after their traffic becomes steady and well understood.
For a uniform ingestion rate the per-GB model can cost much more than a right-sized set of provisioned shards; AWS cost optimization guidance states that for a given uniform data ingestion rate, on-demand mode can be many times more expensive than provisioned mode. Accounts with many on-demand streams or large aggregate throughput can also overpay by staying on On-demand Standard when the account-level On-demand Advantage mode, which drops the per-stream charge and lowers per-GB rates in exchange for a minimum usage commitment, would fit their usage.
Billing model
The pricing dimensions that drive this cost.
Rates below are the pricing page figures for US East (N. Virginia).
- On-demand Standard
- Billed per stream-hour (listed at $0.040), per GB of data in (listed at $0.08) and per GB of data out (listed at $0.040), with enhanced fan-out retrievals charged additionally
- On-demand Advantage
- An account-level, per-Region mode with no per-stream charge and lower per-GB rates (listed at $0.032 in and $0.016 out), committing the account to at least 25 MiB/s ingested and 25 MiB/s retrieved; shortfalls are billed at the discounted rate
- Provisioned mode
- Billed per shard-hour (listed at $0.015) plus per million 25 KB PUT payload units (listed at $0.014), regardless of how much of each shard's capacity is used
How to detect
4 checks to find it in your estate.
- List streams and their StreamModeDetails and, for each on-demand stream, review IncomingBytes, IncomingRecords and GetRecords.Bytes over at least 30 days to find streams with flat, predictable throughput
- For those streams, estimate the provisioned shard count with the sizing formula (the higher of write KiB/s divided by 1,024 and read KiB/s divided by 2,048, rounded up, plus headroom for peaks) and compare shard-hour and PUT payload unit cost with the current per-GB and stream-hour charges
- Sum data in and data out across all on-demand streams in each Region and compare with the 25 MiB/s ingest and retrieval commitment of On-demand Advantage; the Kinesis console also indicates whether an account's usage is a good fit
- Count small on-demand streams in On-demand Standard, each paying the fixed stream-hour charge even with little traffic
How to fix
5 ways to remove the waste.
- Switch steady, predictable streams to provisioned mode with a shard count sized for observed peaks plus headroom; switching does not disrupt producers or consumers, and the stream keeps its current shard count at the moment of the switch, so right-size it afterward
- Automate shard scaling for provisioned streams with Application Auto Scaling where traffic follows a daily pattern, since provisioned mode leaves shard management to you
- For accounts whose aggregate on-demand usage is at or above the commitment, or that run many on-demand streams or many enhanced fan-out consumers, evaluate enabling On-demand Advantage; it must stay enabled for at least 24 hours, and warm throughput must be removed before switching back
- Keep on-demand mode for genuinely unpredictable or spiky streams, noting that a stream can switch between on-demand and provisioned modes only twice within 24 hours
- Consolidate very small streams where the architecture allows, to reduce fixed per-stream charges in On-demand Standard
Documentation
Vendor references for pricing and configuration.