# Missing Delta Optimization Features for High-Volume Tables

Canonical: https://www.pointfive.co/efficiency-hub/inefficiencies/missing-delta-optimization-features-for-high-volume-tables

[Cloud Efficiency Hub](https://www.pointfive.co/efficiency-hub) 

# Missing Delta Optimization Features for High-Volume Tables

## Explanation

In many Databricks environments, large Delta tables are created without enabling standard optimization features like partitioning and Z-Ordering. Without these, queries scanning large datasets may read far more data than necessary, increasing execution time and compute usage. \*  Partitioning  organizes data by a specified column to reduce scan scope. \*  Z-Ordering  optimizes file sorting to minimize I/O during range queries or filters. \*  Delta Format  enables additional optimizations like data skipping and compaction. Failing to use these features in high-volume tables often results in avoidable performance overhead and elevated spend, especially in environments with frequent exploratory queries or BI workloads.

## Relevant Billing Model

Databricks charges are based on DBUs (Databricks Units) per hour, which correlate directly with compute resource use. Query performance heavily impacts DBU consumption. Inefficient data layout leads to longer scan times, increased cluster runtime, and higher costs.

## Detection

- Tables lacking partitioning on commonly filtered columns

- Absence of Z-Ordering on high-selectivity columns (e.g., timestamps, IDs)

- Slow query performance tied to full-table scans

- High DBU usage by queries reading large volumes of data unnecessarily

- ETL pipelines writing to Delta tables without compaction or OPTIMIZE steps

## Remediation

- Apply partitioning when writing Delta tables, using columns commonly filtered in queries

- Enable Z-Ordering on appropriate columns to improve data skipping efficiency

- Use \`OPTIMIZE\` and \`VACUUM\` to reduce file fragmentation and improve query performance

- Standardize use of Delta Lake format in ETL pipelines

- Automate periodic optimizations for long-lived tables based on size or access patterns

## Relevant Documentation

[Optimize performance with file management](https://learn.microsoft.com/en-us/azure/databricks/delta/optimizations/file-management) 
[Partitioning and Z-Ordering](https://learn.microsoft.com/en-us/azure/databricks/delta/optimizations/optimization-examples) 
[Databricks Delta Table OPTIMIZE Command](https://learn.microsoft.com/en-us/azure/databricks/sql/language-manual/delta-optimize)

[Submit Feedback](https://hub.pointfive.co/inefficiencies/missing-delta-optimization-features-for-high-volume-tables)

## At a glance

Reference

CER-0177

Cloud provider

Databricks

Service

Delta Lake

Category

[Storage](https://www.pointfive.co/efficiency-hub/service-category/storage)

Inefficiency type

Suboptimal Data Layout

## Contributed by

- Scott Shulman 

### Scott Shulman

Automation and FinOps Evangelist @ Arrow Electronics

[LinkedIn](https://www.linkedin.com/in/shulmanscott/)

---
Source: the public page above. Product screenshots and illustrative interfaces are examples, not live customer data.