So far, we’ve used load_dataset to pull pre-made datasets (like “imdb“) from the Hugging Face Hub. But the real power is training an AI on your data—like your company’s support emails or your personal notes. In this guide, you’ll learn how Hugging Face load local data options make it possible to harness your own datasets for custom AI training.
The datasets library makes this easy.
The Goal
We will load our own local CSV file and prepare it, just like we did in Fine-Tuning Part 1.
Step 1: Create Your Local Data
Create a file named my_reviews.csv in your project folder:
my_reviews.csv
text,label "This product is amazing!",1 "I hated this, it broke in one day.",0 "It's an okay product, not bad.",1 "The worst customer service ever.",0
(Here, 1 = Positive, 0 = Negative)
Step 2: The Code
We use load_dataset just like before, but point it at our local files.
from datasets import load_dataset, DatasetDict
from transformers import AutoTokenizer
# 1. Load your local CSV file
# This loads it into a 'Dataset' object
local_ds = load_dataset("csv", data_files="my_reviews.csv")
# 2. (Optional) Create a Train/Test Split
# Our CSV is small, let's split it 80/20
split_ds = local_ds['train'].train_test_split(test_size=0.2)
# 'split_ds' is now a DatasetDict, just like when we loaded "imdb"!
print(split_ds)
# DatasetDict({
# train: Dataset({ ... num_rows: 3
# test: Dataset({ ... num_rows: 1
# })
# 3. Tokenize it (just like before)
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
def tokenize_function(batch):
return tokenizer(batch["text"], padding="max_length", truncation=True)
tokenized_datasets = split_ds.map(tokenize_function, batched=True)
print("\nYour custom data is ready for training!")
print(tokenized_datasets['train'][0])You have now successfully prepared your own custom data. You can feed this tokenized_datasets object directly into the Hugging Face Trainer to build an AI that’s an expert on your specific text.
Key Takeaways
- The article explains how to use Hugging Face load local data to train AI on your datasets.
- First, create a CSV file named my_reviews.csv in your project folder to store your data.
- Then, use the load_dataset function to point at your local files for custom training.
- You can prepare your own data and feed it into the Hugging Face Trainer to enhance AI on your specific text.
