Explanation
Why the waste happens and who it affects.
Each of those is billed as a custom metric for every hour it receives data. When applications or agents attach dimensions with many possible values, such as request ID, user or customer ID, session ID, pod or container name, IP address or full URL path, one logical metric like Latency turns into thousands or millions of billable metrics.
The problem is easy to create by accident with the embedded metric format, where AWS warns that metrics based on high-cardinality dimensions such as requestId create a custom metric for each unique combination, and with PutMetricData or agent configurations that append per-instance or per-container dimensions. It is common in microservice and container environments, where short-lived tasks and pods keep generating new dimension values, and it shows up as custom metric charges growing faster than the workload.
Billing model
The pricing dimensions that drive this cost.
Classic custom metrics, including those created from embedded metric format logs, are billed per metric per month.
- Custom metric
- Billed per metric-month in volume tiers ($0.30 for the first 10,000, $0.10 for the next 240,000, $0.05 above 250,000 in AWS pricing examples)
- Hourly proration
- Metrics are prorated by the hour and charged only in hours when data is sent, so short-lived dimension values still add cost
- Dimension combination
- Every unique combination of dimension values is a separate billable metric, even with the same metric name
- EMF log charges
- Embedded metric format also incurs log ingestion and archival charges for the log events that carry the metrics
How to detect
4 checks to find it in your estate.
- In Cost Explorer or the Cost and Usage Report, filter CloudWatch usage type MetricMonitorUsage and break it down by operation (MetricStorage for PutMetricData, MetricStorage:AWS/Logs-EMF for embedded metrics) to see which source drives metric count
- Use list-metrics per custom namespace and count distinct values for each dimension name to find dimensions with unbounded values
- Review EMF payloads, client library code and CloudWatch agent configurations for dimensions built from identifiers such as request, user, session, container or pod IDs
- Track the total custom metric count per namespace over time and flag namespaces whose count grows with traffic or deployments rather than with the number of services
How to fix
5 ways to remove the waste.
- Remove unbounded identifiers from dimensions and keep them as log properties instead, so they remain queryable in CloudWatch Logs Insights without creating metrics
- Aggregate before publishing, for example per service, operation or status code instead of per request, per user or per container
- Limit the dimension sets declared in EMF metric directives and in agent configurations to the combinations actually used in dashboards and alarms
- Where per-entity metrics are genuinely needed, evaluate publishing them as OpenTelemetry metrics, which CloudWatch bills per GB ingested ($0.50 per GB on the pricing page) with no charge for the number of unique metric series, instead of as classic custom metrics
- Review the change with dashboard and alarm owners first, because alarms and graphs that reference removed dimension combinations will stop receiving data
Documentation
Vendor references for pricing and configuration.