This is one of the original tasks that made models like BERT famous. A “Masked Language Model” (MLM) is trained by having words randomly “masked” (hidden), and its only job is to guess what the hidden word was. This approach is also used in Hugging Face Fill-Mask tools to predict missing words in sentences.
This is what gives the model a deep understanding of language and context. We can use the Hugging Face pipeline to do this ourselves.
Step 1: Installation
pip install transformers torch
Step 2: The Code
We will use the fill-mask pipeline. The special word for the “blank” is [MASK].
from transformers import pipeline
# 1. Load the pipeline
# 'distilroberta-base' is a fast, powerful model
fill_masker = pipeline(
"fill-mask",
model="distilroberta-base"
)
# 2. Create your sentence with a mask
text = "The capital of France is [MASK]."
# 3. Run the model!
results = fill_masker(text)
# 4. Print the top 5 predictions
print(f"--- Top 5 guesses for: '{text}' ---")
for result in results:
print(f"Token: '{result['token_str']}' | Confidence: {result['score']:.4f}")Step 3: The Result
The model will give you a list of the most likely words to fill the blank, based on its understanding of the world.
--- Top 5 guesses for: 'The capital of France is [MASK].' --- Token: ' Paris' | Confidence: 0.9995 Token: ' Paris' | Confidence: 0.0001 Token: ' Bordeaux' | Confidence: 0.0000 Token: ' Lyon' | Confidence: 0.0000 Token: ' Marseille' | Confidence: 0.0000
It knows with 99.95% certainty that the answer is “Paris”. You can use this for error correction, creative writing, and more.
Key Takeaways
- Masked Language Models (MLMs) like BERT guess hidden words to understand language and context.
- You can use the Hugging Face pipeline to implement this functionality.
- The process involves installation, using the fill-mask pipeline, and preparing a masked word with [MASK].
- The model predicts the most likely word with high certainty, such as identifying ‘Paris’.
- This technique is applicable for tasks like error correction and creative writing.
