Explanation
Why the waste happens and who it affects.
Without an Index State Management (ISM) policy, every older index stays on hot storage, the instance-store or EBS volumes attached to data nodes, until someone deletes it by hand. Hot storage also carries replica copies and reserved space overhead, so months of rarely queried data can drive both the number of data nodes and the size of their volumes.
OpenSearch Service offers UltraWarm and cold storage, both backed by Amazon S3 and billed for the data actually stored, for read-only data that is queried less often or only occasionally. Teams often never enable them because the domain was created before these tiers were needed, because they require dedicated master nodes, or because nobody owns the retention decision. Historical data then stays at hot-storage prices, often for longer than any compliance requirement asks.
Billing model
The pricing dimensions that drive this cost.
The storage tier an index lives on determines how it is billed.
- Hot storage
- Data node instance-hours plus EBS billed per GB-month provisioned, the same price whether the volume is empty or full, with replicas and reserved space consuming part of it
- UltraWarm
- An hourly rate per UltraWarm node plus managed storage billed only for the data stored, counting primary shards only
- Cold storage
- Managed storage billed for data stored, with no compute cost and no charge if cold storage holds no data
- Tier migrations
- No transfer charge for moving data between warm and cold, and only one copy is billed while a migration runs
How to detect
5 checks to find it in your estate.
- List indexes with their creation dates and sizes (for example _cat/indices sorted by creation.date, or GET _hot and GET _warm on UltraWarm domains) and identify time-series indexes older than the window your dashboards and alerts actually query
- Check whether index patterns are covered by an ISM policy with warm, cold or delete states, using the ISM explain API or the Index Management page in OpenSearch Dashboards
- Compare ClusterUsedSpace and FreeStorageSpace with the share of data held in old indexes to estimate how much hot capacity is serving historical data
- On domains with UltraWarm or cold storage enabled, review WarmStorageSpaceUtilization, ColdStorageSpaceUtilization and HotToWarmMigrationSuccessCount to confirm policies are actually moving data
- Compare the oldest retained index with the documented retention requirement for that data set
How to fix
6 ways to remove the waste.
- Attach ISM policies that move indexes from hot to UltraWarm to cold and finally delete them, based on index age, size or document count, and use rollover (for example at a fixed size or age) to keep index growth bounded
- Enable UltraWarm and cold storage on domains that meet the prerequisites: dedicated master nodes, no T2 or T3 data nodes, OpenSearch or Elasticsearch 6.8 or later for UltraWarm and 7.9 or later for cold storage; enabling either one is a configuration change that causes a blue/green deployment
- Move only data that fits the warm tier: warm indexes are read-only unless returned to hot storage, cold indexes must be migrated back to warm before they can be queried, and broad or frequent searches may perform better on hot storage
- Use index rollups to summarize older time-series data into coarser intervals before tiering or deletion, where detailed history is not needed
- After data leaves the hot tier, reduce the number of data nodes or the EBS volume size so the smaller hot footprint actually lowers the bill
- Set delete states that match compliance requirements rather than keeping data indefinitely
Documentation
Vendor references for pricing and configuration.