If you try to plot 10 million points with Matplotlib or Seaborn, your computer will freeze. It tries to draw 10 million individual circles, which uses too much RAM.
Datashader solves this. It converts your massive dataset into a single image (rasterization) based on density. It processes millions of points in milliseconds.
When combined with Polars and hvplot, you get an interactive map of massive data.
Step 1: Installation
pip install polars hvplot datashader holoviews
Step 2: Generate Massive Data
Let’s create a dummy dataset with 5 million points.
import polars as pl
import numpy as np
import hvplot.polars # Enables the .hvplot accessor
n = 5_000_000
df = pl.DataFrame({
"x": np.random.normal(0, 1, n),
"y": np.random.normal(0, 1, n)
})
print(f"DataFrame created with {n} rows.")Step 3: The Crash (What NOT to do)
# DON'T DO THIS! # df.hvplot.scatter(x='x', y='y') # This tries to create 5 million interactive HTML elements.
Step 4: The Solution (rasterize=True)
We add one argument: rasterize=True. This tells hvplot to use Datashader backend.
plot = df.hvplot.scatter(
x='x',
y='y',
rasterize=True, # <--- The Magic Switch
cmap='fire', # Color map (like a heat map)
title="5 Million Points with Datashader"
)
# Display (in Jupyter or save to HTML)
# hvplot.show(plot)
plotYou can now zoom in and out of 5 million points instantly. As you zoom, Datashader re-calculates the image dynamically to show you the detail.
Key Takeaways
- Plotting 10 million points with Matplotlib or Seaborn can freeze your computer due to high RAM usage.
- Datashader efficiently converts large datasets into a single image using rasterization, processing millions of points quickly.
- Combining Datashader with Polars and hvplot allows for interactive visualisation of massive datasets.
- To prevent crashes when dealing with large data, use the argument rasterize=True in hvplot.
- This approach enables dynamic recalculation of images as you zoom in and out of the data.
