In Previous Weeks, we introduced partitioned datasets. Now, let’s dive into Hive Partitioning, the industry standard for organizing Big Data (used by Apache Spark, AWS Athena, and Polars).
What is Hive Partitioning?
It’s a specific folder structure where the folder names contain the data values:
/dataset/
/region=US/
/year=2024/
data.parquet
/region=EU/
/year=2024/
data.parquet
Why Polars Loves Hive
When you run a query like filter(pl.col("region") == "US"), Polars doesn’t even open the files in the /region=EU/ folder. It skips them entirely based on the folder name. This is called Partition Pruning.
How to Use It
You don’t need special code. scan_parquet detects Hive partitions automatically if hive_partitioning=True (which is default).
import polars as pl
# 1. Create a Hive-style dataset (Writing)
df = pl.DataFrame({
"region": ["US", "US", "EU", "EU"],
"year": [2024, 2025, 2024, 2025],
"sales": [100, 150, 200, 250]
})
# Write to folder structure
df.write_parquet(
"sales_data",
partition_by=["region", "year"]
)
# 2. Querying (Reading)
# We scan the ROOT folder
lazy_df = pl.scan_parquet("sales_data/**/*.parquet")
# This query will be INSTANT because it only opens 1 folder
# It ignores 'EU' and '2025' folders completely
query = (
lazy_df
.filter(
(pl.col("region") == "US") &
(pl.col("year") == 2024)
)
.collect()
)
print(query)Using Hive partitioning is the single most effective optimization for terabyte-scale datasets.
Polars Hive Partitioning Cuts Query Time by Skipping Folders Entirely
Partition pruning only fires when the filter column matches the partition key exactly — filter(pl.col("region") == "US") skips the EU folder, but filter(pl.col("region").str.starts_with("U")) forces Polars to open every partition and scan row by row. Choose partition keys that appear in high-cardinality equality filters in your actual queries, not columns you filter with ranges or string operations. Partition order in partition_by also matters: Polars writes outer folders first, so put the column your queries filter on most often at position zero. For datasets that grow over time, year or month at the outermost level keeps new writes isolated to a single folder and lets historical partitions stay untouched.
Key Takeaways
- Hive Partitioning is an effective method for organizing Big Data using a specific folder structure based on data values.
- Polars benefits from Hive Partitioning through Partition Pruning, which skips irrelevant folders during query execution.
- Using Hive partitioning offers significant optimisation for large datasets, reducing query time by avoiding unnecessary file access.
- To maximise efficiency, choose partition keys that match high-cardinality equality filters in your queries.
- Place the most frequently filtered column at the outermost level in your partition structure for optimal performance.
