Explanation
Why the waste happens and who it affects.
Databricks releases runtimes on a regular cadence, and its cost-optimization best practices note that these releases bring performance improvements that often reduce cost through more efficient use of compute. Workloads left on old runtimes do the same work with an older engine and optimizer, so they can run longer and consume more DBUs and cloud instance hours than they would on a current version.
Runtime versions are pinned by default: the spark_version in a job definition, cluster JSON, Terraform module or compute policy stays fixed until someone edits it, and even the auto:latest-lts policy value does not update existing compute when a new runtime ships. Teams avoid upgrades because of library and API changes, so jobs stay on versions that are several releases behind or near end of support. Beyond cost, versions past end of support receive no fixes, and at end of life existing workloads on them fail, which forces a rushed migration.
Billing model
The pricing dimensions that drive this cost.
- DBU consumption
- Classic compute is billed in DBUs at per-second granularity for as long as the cluster runs, at a rate set by the compute type and instance size
- Cloud instance charges
- The virtual machines behind classic compute are billed separately by the cloud provider for the same running time
- Runtime-driven duration
- The runtime version itself has no separate price, but a slower engine lengthens run time, which raises both DBU and instance charges
How to detect
4 checks to find it in your estate.
- Query system.compute.clusters for the latest record of each non-deleted cluster (delete_time is null) and group by dbr_version and cluster_source to find all-purpose, job and pipeline compute on older runtimes
- Compare those versions against the Databricks Runtime release notes and support lifecycle page, flagging versions that are several releases behind the latest LTS or close to end of support
- Join system.compute.clusters to system.billing.usage on usage_metadata.cluster_id to rank outdated-runtime compute by DBUs consumed, so upgrades start with the most expensive workloads
- Review job definitions (Jobs API or asset bundles) and Terraform modules for hard-coded spark_version values, and compute policies whose spark_version is fixed to an old runtime
How to fix
4 ways to remove the waste.
- Upgrade high-cost workloads to the latest LTS runtime in a test run first, and compare run duration and DBUs from system.billing.usage before and after; test library compatibility and review the runtime migration notes for behavior changes
- Update compute policies to allow only supported recent runtimes, using an allowlist or regex on spark_version or special values such as auto:latest-lts, so new compute does not start on old versions
- Remember that policy special values do not auto-update existing compute; schedule a periodic sweep that bumps pinned spark_version values in jobs, bundles and Terraform
- Where workloads fit, move them to serverless compute, where the runtime is managed by Databricks rather than pinned per cluster
Documentation
Vendor references for pricing and configuration.