High-Cardinality Tags Inflating Datadog Custom Metrics

Every new tag value creates a new billable timeseries, so a metric tagged with an unbounded identifier such as user ID, request ID, session ID, container ID or pod name can turn one metric name into thousands of custom metrics. HISTOGRAM and DISTRIBUTION metrics multiply this again: each generates five custom metrics per tag combination by default, and enabling percentiles on a distribution adds five more.

The count is driven by application code, not infrastructure size. A developer adds a tag for debugging, or a library emits per-pod tags, and the monthly average of distinct timeseries jumps above the per-host allotment, often for timeseries no dashboard or monitor queries. The cost lands on the central observability budget while the cause sits in individual services, so attribution needs Datadog's per-metric and per-tag usage tools.

This applies to the cardinality-based Custom Metrics model; contracts on Metric Name pricing SKUs are billed differently.

Custom metric

A unique combination of metric name and tag values, including host; billable usage is the monthly average of distinct custom metrics per hour

Account allotment

100 indexed custom metrics per host on Pro and 200 on Enterprise, pooled across the account

Indexed custom metrics overage

Each 100 indexed custom metrics above the allotment billed at the rate in your contract

Ingested custom metrics

For metrics configured with Metrics without Limits, each 100 ingested custom metrics above the allotment billed at $0.10

Distribution multiplier

5 custom metrics per tag combination for count, sum, min, max and avg, plus 5 more when percentiles are enabled

• On the Plan and Usage Custom Metrics tab, review billable usage, the usage trend and the Top Custom Metrics table, then open the top metric names in Metrics Summary
• In the metric details side panel, use the Custom Metrics Tags Cardinality Explorer to find the tag keys driving the count, such as unbounded user, request, session, container or pod identifiers
• Use Metrics Volume Management to find the largest metric names and those spiking in volume, and chart datadog.estimated_usage.metrics.custom.by_metric and datadog.estimated_usage.metrics.custom.by_tag for attribution by service or team
• In Metrics Summary, filter with the Query Activity and Related Assets facets to find custom metrics that are not queried and not used on any dashboard, notebook, monitor or SLO
• Review distribution metrics with percentile aggregations enabled and HISTOGRAM metrics with extra histogram_aggregates or histogram_percentiles in datadog.yaml

• Remove unbounded tags at the source, or aggregate them before submission, and keep identifiers such as request or user IDs in traces and logs instead of metric tags
• Use Metrics without Limits to set an allowlist of queryable tags on high-cardinality metrics, starting from the recommended configuration based on actively queried tags, and give unqueried metrics an empty tag configuration
• Disable percentile aggregations on distributions that do not need them, and trim histogram_aggregates and histogram_percentiles in the Agent configuration
• Restrict who can change tag configurations with the metrics_tags_write permission, and review changes in the audit trail
• Tradeoff: metrics configured with Metrics without Limits are also billed for ingested custom metrics above the allotment, so check the tag configuration cardinality estimator before saving and do not save a configuration whose indexed count is higher than ingested

• Custom Metrics Billingdocs.datadoghq.com
• Best Practices for Custom Metrics Governancedocs.datadoghq.com
• Metrics without Limitsdocs.datadoghq.com
• Estimated Usage Metricsdocs.datadoghq.com