Explanation
Why the waste happens and who it affects.
User home directories, departmental shares and software build shares tend to hold many copies of the same or similar files, so without deduplication the file system needs far more provisioned storage than the unique data requires. AWS documents typical savings of 50-60 percent for general-purpose shares, 30-50 percent for user documents and 70-80 percent for software development datasets.
Because FSx for Windows bills on provisioned storage capacity, and that capacity can be increased but never decreased, skipping deduplication compounds: teams add capacity as shares fill up, and each increase is permanent for the life of the file system. The pattern is common on file systems migrated from on-premises servers where deduplication was never configured. Deduplication is a poor fit for highly dynamic data, so the savings are concentrated on shares with mostly static content.
Billing model
The pricing dimensions that drive this cost.
FSx for Windows File Server bills for what you provision, prorated by the hour, not for the data actually stored.
- Storage capacity
- Billed per GB-month of provisioned SSD or HDD capacity, by deployment type, whether or not it is used
- Throughput capacity
- Billed per MBps-month of provisioned throughput, which also sets the memory available for deduplication jobs
- Backups
- Billed per GB-month of backup storage consumed
- Capacity increase only
- Storage capacity can be increased but never decreased on an existing file system
How to detect
4 checks to find it in your estate.
- Run Get-FSxDedupStatus through the FSx remote PowerShell endpoint on each Windows file system; file systems with no deduplication status have it disabled
- Use Measure-FSxDedupFileMetadata on representative folders to estimate how much space deduplication would reclaim before enabling it
- Prioritize file systems whose FreeStorageCapacity CloudWatch metric is trending down and that are candidates for a storage capacity increase
- Classify shares by content type: user documents, home directories and software build or binary shares are the strongest candidates, while databases and rapidly changing data are not
How to fix
5 ways to remove the waste.
- Enable deduplication with Enable-FSxDedup and adjust settings with Set-FSxDedupConfiguration, for example restricting it to certain file types or folders, or setting a minimum file age
- Schedule optimization and garbage collection jobs with Set-FSxDedupSchedule during idle periods; space is freed only after garbage collection runs
- Check that the file system's throughput capacity provides enough memory for the logical data size (Microsoft recommends about 1 GB of memory per 1 TB of logical data) and increase it if deduplication jobs fail for lack of memory
- Use the reclaimed space to defer or avoid future storage capacity increases; to actually lower provisioned capacity, create a smaller file system and migrate the data, sizing it with headroom until deduplication has run on the new file system
- Monitor job results with Get-FSxDedupStatus, and avoid Robocopy options that AWS and Microsoft flag as unsafe with deduplication
Documentation
Vendor references for pricing and configuration.