# Inefficient Snowpipe Usage Due to Small File Ingestion

Canonical: https://www.pointfive.co/efficiency-hub/inefficiencies/inefficient-snowpipe-usage-due-to-small-file-ingestion

[Cloud Efficiency Hub](https://www.pointfive.co/efficiency-hub) 

# Inefficient Snowpipe Usage Due to Small File Ingestion

## Explanation

Ingesting a large number of small files (e.g., files smaller than 10 MB) using Snowpipe can lead to disproportionately high costs due to the per-file overhead charges. Each file, regardless of its size, incurs the same overhead fee, making the ingestion of numerous small files less cost-effective. Additionally, small files can increase the load on Snowflake's metadata and ingestion infrastructure, potentially impacting performance.

## Relevant Billing Model

## Detection

- Analyze the average file size being ingested via Snowpipe; identify if many files are below the recommended size threshold (e.g., under 10 MB).

- Review the total number of files ingested over a period to assess the impact of per-file overhead charges.

- Evaluate the frequency of file arrivals; high-frequency ingestion of small files may indicate an opportunity for batching.

- Consult with data engineering teams to understand the source systems and whether file batching is feasible without impacting data freshness requirements.

## Remediation

- Implement batching mechanisms to aggregate small files into larger ones before ingestion, aiming for file sizes between 10 MB and 250 MB for optimal cost-performance balance.

Adjust data pipeline configurations to stage data at regular intervals (e.g., every few minutes) to allow for file aggregation.

- Explore using Snowpipe Streaming for real-time ingestion scenarios, as it may offer more cost-effective options for high-frequency, small data loads.

- Monitor Snowpipe usage and costs regularly to identify and address inefficiencies promptly.

## Relevant Documentation

- [Snowpipe Best Practices](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-billing)

- [Snowpipe Streaming Recommendations](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-recommendation)

[Submit Feedback](https://hub.pointfive.co/inefficiencies/inefficient-snowpipe-usage-due-to-small-file-ingestion)

## At a glance

Reference

CER-0019

Cloud provider

Snowflake

Service

Snowflake Snowpipe

Category

[Other](https://www.pointfive.co/efficiency-hub/service-category/other)

Inefficiency type

Inefficient Data Ingestion

## Contributed by

- Simar Arora 

### Simar Arora

Value Engineer - FinOps @ Snowflake

[LinkedIn](https://www.linkedin.com/in/simarpreet-arora/)

---
Source: the public page above. Product screenshots and illustrative interfaces are examples, not live customer data.