You’ve learned all the individual Polars methods. Now, let’s put them together in one “A-to-Z” project to clean a messy dataset and look at effective Polars data cleaning techniques.
This is the 80% of data science that no one talks about, but it’s the most important part.
The Messy Data
Let’s create a “messy” DataFrame.
import polars as pl
from datetime import datetime
df_messy = pl.DataFrame({
"id": ["1", "2", "3", "1", None],
"date": ["2025-01-01", "2025-01-02", "2025-01-03", "2025-01-01", "2025-01-04"],
"product_code": [" A-123 ", "B-456", "A-123", " A-123 ", "C-789"],
"sales": ["$50.50", "100", "$30.20", "$50.50", "N/A"]
})
print("--- Messy Data ---")
print(df_messy)The Cleaning Pipeline
We will chain all our operations in one single, high-speed Polars expression.
df_clean = (
df_messy
# 1. Drop rows with nulls
.drop_nulls()
# 2. Remove exact duplicates
.unique()
.with_columns(
# 3. Cast 'id' to an integer
pl.col("id").cast(pl.Int32),
# 4. Cast 'date' to a real date object
pl.col("date").str.to_date(),
# 5. Clean text: strip whitespace, make uppercase
pl.col("product_code").str.strip_chars().str.to_uppercase(),
# 6. Clean numbers: remove '$', 'N/A', and cast to float
pl.col("sales")
.str.replace("$", "")
.str.replace("N/A", None) # Replace N/A with a real null
.cast(pl.Float64)
)
# 7. Use .fill_null() *after* casting
.fill_null(0)
)
print("\n--- Cleaned Data ---")
print(df_clean)Output:
--- Cleaned Data --- shape: (3, 4) ┌─────┬────────────┬──────────────┬───────┐ │ id ┆ date ┆ product_code ┆ sales │ │ --- ┆ --- ┆ --- ┆ --- │ │ i32 ┆ date ┆ str ┆ f64 │ ╞═════╪════════════╪══════════════╪═══════╡ │ 1 ┆ 2025-01-01 ┆ A-123 ┆ 50.5 │ │ 2 ┆ 2025-01-02 ┆ B-456 ┆ 100.0 │ │ 3 ┆ 2025-01-03 ┆ A-123 ┆ 30.2 │ └─────┴────────────┴──────────────┴───────┘
We combined methods for Nulls, Duplicates, Casting, and Strings into one clean, readable, and fast pipeline.
Key Takeaways
- The article focuses on Polars data cleaning methods, combining them in a project to clean a messy dataset.
- It highlights that data cleaning is the crucial part of data science that often gets overlooked.
- The process includes creating a messy DataFrame, followed by a streamlined cleaning pipeline.
- This pipeline integrates methods for handling nulls, duplicates, casting, and strings into a single, efficient expression.
