Explanation
Newer AWS Glue versions-such as Glue 5.0-include significant performance optimizations for **Python-based** ETL jobs, often reducing runtime by 10-60%. These improvements do not require any code changes, making version upgrades a simple and impactful optimization. When jobs remain on older runtimes such as Glue 3.0 or 4.0, they execute more slowly, consume more DPUs, and incur unnecessary cost. Additionally, Glue 5.0 offers more worker types (larger standard workers and memory-optimized workers), that can provide additional performance gain for some jobs. This inefficiency does not apply to Scala-based jobs, which do not benefit from the same performance uplift.
Relevant Billing Model
AWS Glue is billed per second based on the number of Data Processing Units (DPUs) used. Faster runtime versions reduce job execution time, lowering total DPU consumption and overall cost. Older versions can extend runtime duration without providing any functional benefit.
Detection
- Review Glue job configurations to identify jobs still running on older Glue versions such as 3.0 or 4.0
- Determine whether the job is Python-based, as only Python workloads benefit from upgrading
- Assess job execution times that remain consistently longer than expected for similar workloads
- Evaluate whether runtime upgrades have been incorporated into standard Glue job governance or deployment templates
Remediation
- Upgrade Python-based Glue jobs to the latest supported Glue runtime version to benefit from performance improvements
- Incorporate runtime version management into IaC templates and deployment pipelines to ensure consistent modernization
- Periodically review job runtimes to confirm alignment with current Glue versions and available optimizations
- Validate job output after upgrading to confirm compatibility with updated library and Spark versions