In the real world, data is rarely in one big CSV. It’s usually split, and this is where you might want to use Polars to scan multiple files efficiently.
CSV split data :
sales_2024_01.csvsales_2024_02.csv- …
In Pandas, you have to write a loop, read every file into a list, and then concat them. It’s slow and messy. In Polars, you can just use a Glob Pattern.
The “Glob” Pattern
A glob pattern uses * as a wildcard.
data/*.csv= All CSVs in the data folder.data/2024_*.csv= All CSVs starting with 2024.
The Code
Polars handles this natively in scan_csv (and scan_parquet).
import polars as pl
# 1. Scan ALL files in the folder at once
# This creates a single LazyFrame that represents ALL the data combined
df_lazy = pl.scan_csv("sales_data/*.csv")
# 2. Verify it worked
# Let's count rows per file (Polars adds a 'source_file' column automatically!)
result = (
df_lazy
.with_columns(pl.col("_source").alias("filename")) # Capture filename (if supported by version)
.group_by("filename")
.len()
.collect()
)
print(result)Note: If your Polars version doesn’t support _source, simply running df_lazy.collect() will combine all data seamlessly.
This allows you to query a folder containing 100GB of split files as if it were a single, simple table.
Key Takeaways
- Data rarely comes in one large CSV; it’s often split into multiple files like sales_2024_01.csv and sales_2024_02.csv.
- Using Pandas, you must loop through each file and concatenate them, which can be slow and messy.
- In Polars, you can use a Glob Pattern to manage multiple files easily, allowing for efficient data processing.
- Glob patterns use * as a wildcard, such as data/*.csv to select all CSVs in a folder.
- Polars’ scan_csv function lets you query large datasets as if they were a single table, improving performance significantly.
