Platform engineers constantly balance three competing priorities: performance, security, and cost. When it comes to cloud storage, Amazon S3 represents one of the most widely used and potentially expensive services in any AWS environment -- yet optimization efforts typically stop at the bucket level, leaving significant savings on the table.
This work was originally presented at PlatformCon, focusing on a prefix-oriented methodology for S3 cost optimization that goes deeper than traditional bucket-level analysis.
The Problem with Bucket-Level Thinking
S3 provides highly scalable, durable, and available object storage. It is the most popular object storage solution in the cloud ecosystem. However, many organizations accumulate sprawling buckets that serve multiple purposes without clear structure or governance.
Best practice calls for purpose-built buckets with well-defined retention policies and storage class configurations. In reality, existing cloud environments are rarely that clean. Buckets grow organically, accumulating objects across many different prefixes (effectively folder-like structures within a bucket) that may have vastly different access patterns, retention requirements, and cost profiles.
Optimizing at the bucket level treats all objects within a bucket identically -- applying the same storage class, lifecycle policy, and analysis. This misses the reality that different prefix paths within a single bucket may warrant entirely different optimization strategies.
Prefix Oriented Object Management
The methodology introduced here is Prefix Oriented Object Management (POOM) -- analyzing and optimizing S3 storage at the prefix level rather than the bucket level. By examining how objects are distributed across prefix structures, teams can identify specific paths that contain infrequently accessed data suitable for cheaper storage classes, prefixes accumulating unnecessary object versions, hot paths that benefit from caching or CDN placement, and abandoned prefix trees that can be cleaned up entirely.
This granular analysis reveals optimization opportunities that are invisible at the bucket level.
The S3DMap Tool
S3DMap is an open-source tool that implements this prefix-oriented methodology through visual storage maps. Rather than working with spreadsheets or CLI output, S3DMap generates visual representations of how storage is distributed across a bucket's prefix hierarchy.
The visual approach makes it immediately apparent where the largest concentrations of storage cost reside and how data is organized. Engineers can quickly identify prefix paths that warrant attention -- whether for lifecycle policy changes, storage class transitions, or cleanup.
Key characteristics of the tool:
- Prefix-level granularity -- analysis operates at individual prefix paths rather than whole buckets
- Visual output -- storage distribution rendered as navigable maps rather than tabular data
- Cost-focused -- designed specifically to surface optimization opportunities, not just inventory
- Open source -- available on GitHub for teams to adopt and extend
Moving Beyond Bucket-Level Optimization
For organizations with significant S3 footprints, the gap between bucket-level and prefix-level optimization can be substantial. A single large bucket may contain terabytes of data across hundreds of prefix paths, each with different access patterns and cost characteristics. Applying a uniform optimization strategy to such a bucket inevitably leaves money on the table.
S3DMap provides the visibility needed to make informed, targeted decisions about S3 storage management -- turning what was previously an opaque cost center into a structured optimization opportunity.