Data Engineering with Polars: Performing Upserts (Merge) into Delta Tables
In Data Engineering, you rarely just “write” files. You usually have a master dataset (e.g., “All Users”), and every day you get a “Daily Update”…
Modern Data Lakes: Reading and Writing Delta Tables with Polars
We’ve used Parquet files because they are fast. But what if you write to a Parquet file while someone else is reading it? The file…
Big Data in Polars: How to Scan and Process Multiple Files at Once
In the real world, data is rarely in one big CSV. It’s usually split, and this is where you might want to use Polars to…
Big Data in Polars: Reading and Writing Partitioned Parquet Files
When you have 1TB of data, you don’t save it in one giant file. You split it up. Polars Partitioned Parquet is handling large datasets…
