Explanation
Ingesting numerous small files through Snowpipe creates cost inefficiencies. Each file, regardless of its size, incurs the same overhead fee, making small file ingestion less economical. The practice also strains Snowflake's metadata infrastructure.
Relevant Billing Model
Snowpipe charges involve compute resources plus per-file overhead. The platform assesses 0.06 credits per 1,000 files loaded, regardless of file size.
Detection
- Analyze average file sizes to identify those below 10 MB thresholds
- Review total file ingestion volumes
- Evaluate arrival frequency patterns
- Consult data engineering teams about batching feasibility without compromising data freshness
Remediation
- Implement file aggregation mechanisms targeting sizes between 10 MB and 250 MB
- Configure pipelines for regular staging intervals
- Consider Snowpipe Streaming for real-time scenarios
- Establish consistent monitoring practices