# Unvacuumed Delta Tables Retaining Unreferenced Data Files Canonical: https://www.pointfive.co/efficiency-hub/inefficiencies/unvacuumed-delta-tables-retaining-unreferenced-data-files Delta Lake never modifies data files in place. Every UPDATE, DELETE, MERGE, overwrite and OPTIMIZE writes new files and marks the old ones as removed... By: PointFive Updated: 2026-09-28 [Cloud Efficiency Hub](https://www.pointfive.co/efficiency-hub) The short version Delta Lake never modifies data files in place. PointFive Research Cloud cost research at PointFive Databricks service [Delta Lake](https://www.pointfive.co/efficiency-hub/cloud-services/delta-lake) Category [Storage](https://www.pointfive.co/efficiency-hub/service-category/storage) Reference CER-0529 Type Excessive Data Retention ## Explanation Why the waste happens and who it affects. Every UPDATE, DELETE, MERGE, overwrite and OPTIMIZE writes new files and marks the old ones as removed in the transaction log, but the old files stay in cloud object storage so earlier table versions remain readable. They are only physically deleted when VACUUM runs and removes unreferenced files older than the retention threshold, 7 days by default. Databricks notes that deleting unused data files reduces cloud storage costs. On tables that are rewritten often - MERGE-heavy CDC targets, tables fully overwritten on each run, tables compacted by frequent OPTIMIZE - unreferenced files accumulate with every rewrite and can exceed the size of the current table if VACUUM never runs. Predictive optimization runs VACUUM automatically, but only on Unity Catalog managed tables where it is enabled, so external tables, legacy Hive metastore tables and managed tables with predictive optimization disabled are the usual sources. The cost lands on the cloud provider's storage bill rather than as a Databricks line item, which is why it tends to go unnoticed. Tables using deletion vectors add a further case: deleted rows are only marked in metadata until the files are rewritten, so VACUUM alone does not reclaim them. ## Billing model The pricing dimensions that drive this cost. Object storage Every data file under the table path, referenced or not, is billed by the cloud provider per GB-month until it is deleted Retention threshold VACUUM removes only unreferenced files older than the retention period, 7 days by default and configurable with delta.deletedFileRetentionDuration Maintenance compute VACUUM run manually uses the cluster or warehouse it runs on; predictive optimization runs it on serverless compute billed under a serverless jobs SKU ## How to detect 5 checks to find it in your estate. - Compare DESCRIBE DETAIL sizeInBytes, which reflects the current table version, with the total size of objects under the table's storage location from a cloud storage inventory report; a large gap indicates unreferenced files - Run DESCRIBE HISTORY on high-churn tables and check for VACUUM START and VACUUM END operations; tables with frequent MERGE, UPDATE, DELETE or OPTIMIZE operations and no recent VACUUM are the main candidates - Check whether predictive optimization is enabled for the catalogs and schemas holding managed tables (DESCRIBE CATALOG EXTENDED or DESCRIBE SCHEMA EXTENDED) and review system.storage.predictive\_optimization\_operations\_history to see which tables it is vacuuming - List external tables and Hive metastore tables, which predictive optimization does not cover, and confirm each has a scheduled VACUUM - Look for table properties that set delta.deletedFileRetentionDuration far above what time travel and recovery needs require ## How to fix 5 ways to remove the waste. - Enable predictive optimization for Unity Catalog managed tables (ALTER CATALOG or ALTER SCHEMA ... ENABLE PREDICTIVE OPTIMIZATION) so VACUUM, OPTIMIZE and ANALYZE run automatically - Schedule VACUUM for external and legacy tables, using VACUUM ... DRY RUN first to preview deletions, and VACUUM LITE (Databricks Runtime 16.4 LTS and above) on very large tables where listing the whole directory is slow - For tables with deletion vectors, run REORG TABLE ... APPLY (PURGE) before VACUUM so soft-deleted rows are rewritten out of data files and can be reclaimed - Set delta.deletedFileRetentionDuration to the time travel window the business actually needs; VACUUM removes the ability to query versions older than the retention period, so confirm recovery and audit requirements first - Size VACUUM compute as Databricks recommends - modest autoscaling workers with a larger driver for tables with many files - rather than running it on a large general-purpose cluster ## Documentation Vendor references for pricing and configuration. - [Remove unused data files with vacuum docs.databricks.com](https://docs.databricks.com/aws/en/delta/vacuum) - [Predictive optimization for Unity Catalog managed tables docs.databricks.com](https://docs.databricks.com/aws/en/optimizations/predictive-optimization) - [Best practices for cost optimization docs.databricks.com](https://docs.databricks.com/aws/en/lakehouse-architecture/cost-optimization/best-practices) - [Databricks Pricing: Flexible Plans for Data and AI Solutions databricks.com](https://www.databricks.com/product/pricing) ## Related inefficiencies [Browse the library](https://www.pointfive.co/efficiency-hub) - Delta Lake CER-0177 ### [Missing Delta Optimization Features for High-Volume Tables](https://www.pointfive.co/efficiency-hub/inefficiencies/missing-delta-optimization-features-for-high-volume-tables) In many Databricks environments, large Delta tables are created without enabling standard optimization features like liquid clustering (or, on older tables, partitioning and Z-Ordering). Without these, queries scanning large datasets may... Storage - Azure Backup CER-0278 ### [Azure Backup Data Retained Beyond Intended Retention Period](https://www.pointfive.co/efficiency-hub/inefficiencies/azure-backup-data-retained-beyond-intended-retention-period) Backup data persists longer than intended due to misaligned or outdated retention policies. It often arises when retention requirements change over time, but older recovery points are not evaluated or cleaned up accordingly. In some cases,... Storage - Azure Backup CER-0286 ### [Retained Azure Backup Data After Resource Decommissioning](https://www.pointfive.co/efficiency-hub/inefficiencies/retained-azure-backup-data-after-resource-decommissioning) A protected resource (such as a virtual machine, database, or file share) is decommissioned without explicitly stopping backup protection. In these cases, Azure Backup keeps applying the retention policy to older recovery points, but the... Storage --- Source: the public page above. Product screenshots and illustrative interfaces are examples, not live customer data.