# Lack of Workload-Specific Cluster Segmentation

Canonical: https://www.pointfive.co/efficiency-hub/inefficiencies/lack-of-workload-specific-cluster-segmentation

[Cloud Efficiency Hub](https://www.pointfive.co/efficiency-hub) 

# Lack of Workload-Specific Cluster Segmentation

## Explanation

Running varied workload types (e.g., ETL pipelines, ML training, SQL dashboards) on the same cluster introduces inefficiencies. Each workload has different runtime characteristics, scaling needs, and performance sensitivities. When mixed together, resource contention can degrade job performance, increase cost, and obscure cost attribution.

ETL jobs may overprovision memory, while lightweight SQL queries may trigger unnecessary cluster scale-ups. Job failures or retries may increase due to contention, and queued jobs can further inflate runtime costs. Without clear segmentation, teams lose the ability to tune environments for specific use cases or monitor workload-specific efficiency.

## Relevant Billing Model

Databricks charges per-node-hour using Databricks Units (DBUs), with different rates based on cluster type and configuration. When disparate workloads share a single cluster - especially in all-purpose clusters - compute is inefficiently allocated, jobs contend for resources, and DBU consumption can spike due to overprovisioning, retry inflation, or queuing delays.

## Detection

- Identify all-purpose or shared clusters that execute a wide range of job types

- Look for clusters with inconsistent scaling or job runtimes

- Review job metadata (e.g., task type, frequency, team owner) and compare to cluster setup

- Examine cluster tags and naming conventions for clarity on intended usage

- Check whether long-lived clusters are being used by multiple teams or pipelines

## Remediation

- Define and enforce separate cluster types for distinct workload categories (e.g., SQL, ML, ETL)

- Encourage the use of job clusters for short-lived, batch-oriented workloads to ensure clean isolation and efficient resource use

- Use job clusters for single-purpose, short-lived jobs to ensure isolation and efficient spin-up

- Apply strict cluster tagging and naming standards to reflect usage intent

- Implement cluster policies that restrict configuration options based on workload class

- Educate platform users on workload characteristics and recommend cluster segmentation best practices

## Relevant Documentation

- [Cluster Policies](https://docs.databricks.com/aws/en/admin/clusters/policies)

[Submit Feedback](https://hub.pointfive.co/inefficiencies/lack-of-workload-specific-cluster-segmentation)

## At a glance

Reference

CER-0189

Cloud provider

Databricks

Service

Databricks Compute

Category

[Compute](https://www.pointfive.co/efficiency-hub/service-category/compute)

Inefficiency type

Inefficient Configuration

## Contributed by

- Nicole Boyd 

### Nicole Boyd

Managing Consultant - FinOps @ Capgemini Invent

[LinkedIn](https://www.linkedin.com/in/nicolecboyd/)

---
Source: the public page above. Product screenshots and illustrative interfaces are examples, not live customer data.