Modern Data Lakes: Reading and Writing Delta Tables with Polars

We’ve used Parquet files because they are fast. But what if you write to a Parquet file while someone else is reading it? The file breaks. This is where Polars Delta Lake comes in, offering a solution for concurrent file access.

Delta Lake wraps Parquet files in a “transaction log” (ACID compliance), allowing for safe concurrent writes, updates, and even “Time Travel” (viewing older versions of data). Polars supports this natively.

Step 1: Installation

You need the deltalake python package.

pip install polars deltalake

Step 2: Writing a Delta Table

It looks just like writing Parquet, but it creates a folder structure with a _delta_log.

import polars as pl

df = pl.DataFrame({
    "id": [1, 2, 3],
    "status": ["open", "closed", "open"]
})

# Write to a Delta Table
df.write_delta("my_delta_table", mode="overwrite")
print("Delta Table created.")

Step 3: Reading a Delta Table

# Read it back
df_read = pl.read_delta("my_delta_table")
print(df_read)

Step 4: Time Travel (The Killer Feature)

Let’s overwrite the table with new data.

df_new = pl.DataFrame({"id": [4], "status": ["new"]})
df_new.write_delta("my_delta_table", mode="append")

Now, we can read the current version, OR look back in time!

# Read version 0 (the original data before the append)
df_v0 = pl.read_delta("my_delta_table", version=0)
print("--- Version 0 ---")
print(df_v0)

# Read current version
print("--- Current ---")
print(pl.read_delta("my_delta_table"))

This creates a robust, enterprise-grade data pipeline system on your local machine.