We’ve used Text Embeddings to search documents. Now, let’s use Image Embeddings to search photos. we’ll walk through how you can perform a Reverse Image Search using Python and image embeddings.
We will use CLIP (Contrastive Language-Image Pre-Training). CLIP understands images and text in the same vector space. This allows us to find images that are semantically similar to a query image.
⚡ Quick Fix: Image Similarity Search with CLIP and FAISS
CLIPModel.get_image_features() converts images into 512-dimensional vectors. Add them to a faiss.IndexFlatIP index, then search with a query image embedding to retrieve the closest visual matches by cosine similarity.
from transformers import CLIPProcessor, CLIPModel
import faiss, torch
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
inputs = processor(images=images, return_tensors="pt", padding=True)
with torch.no_grad():
emb = model.get_image_features(**inputs)
emb = emb / emb.norm(p=2, dim=-1, keepdim=True)
emb = emb.numpy().astype("float32")
index = faiss.IndexFlatIP(512)
index.add(emb)
D, I = index.search(query_emb, k=3)The full two-script walkthrough below covers the indexer, the query flow, and normalization requirements.
Step 1: Installation
pip install transformers torch pillow faiss-cpu
Step 2: The Indexer (Scanning Your Images)
First, we need to look at a folder of images, convert them to numbers (vectors), and save them in a FAISS index.
import os
from PIL import Image
from transformers import CLIPProcessor, CLIPModel
import faiss
import torch
import numpy as np
# 1. Load CLIP
model_id = "openai/clip-vit-base-patch32"
model = CLIPModel.from_pretrained(model_id)
processor = CLIPProcessor.from_pretrained(model_id)
# 2. Load Images from a folder
image_folder = "./my_photos"
image_files = [f for f in os.listdir(image_folder) if f.endswith(".jpg")]
images = [Image.open(os.path.join(image_folder, f)) for f in image_files]
# 3. Generate Embeddings
print(f"Embedding {len(images)} images...")
inputs = processor(images=images, return_tensors="pt", padding=True)
with torch.no_grad():
outputs = model.get_image_features(**inputs)
# Normalize vectors for Cosine Similarity
embeddings = outputs / outputs.norm(p=2, dim=-1, keepdim=True)
embeddings = embeddings.numpy().astype('float32')
# 4. Create FAISS Index
index = faiss.IndexFlatIP(512) # 512 is the dimension of CLIP-base
index.add(embeddings)
print("Index created!")Step 3: The Search (Querying)
Now, let’s take a new image and find the closest match in our folder.
# Load a query image
query_image = Image.open("query_cat.jpg")
# Embed the query
inputs = processor(images=query_image, return_tensors="pt")
with torch.no_grad():
query_emb = model.get_image_features(**inputs)
query_emb = query_emb / query_emb.norm(p=2, dim=-1, keepdim=True)
query_emb = query_emb.numpy().astype('float32')
# Search FAISS
D, I = index.search(query_emb, k=3) # Find top 3 matches
print("--- Top Matches ---")
for idx in I[0]:
print(f"Match: {image_files[idx]}")The script will output the filenames of the images that look most like your query image!
Python Image Similarity Search with CLIP and FAISS: The Production-Grade Pattern
CLIP’s vector space makes image-to-image similarity search viable without any labeled training data on your side. Two details determine whether this holds up at scale: normalization and index choice. Normalize every embedding with / emb.norm(p=2, dim=-1, keepdim=True) before calling index.add() — skipping this step turns cosine similarity into a raw dot product and corrupts your rankings. IndexFlatIP (inner product) is the correct index type for normalized vectors; swapping in IndexFlatL2 after normalization produces equivalent results but adds unnecessary computation. For folders beyond a few thousand images, switch to faiss.IndexIVFFlat with a trained coarse quantizer — it trades a small accuracy margin for search times that stay constant regardless of index size.
Key Takeaways
- Use Image Embeddings and CLIP to find semantically similar images to a query image.
- Start by installing the required tools, then create a FAISS index by converting images to vectors.
- The script will return filenames of images that closely match the query image.
- Ensure to normalise embeddings before adding to the index to maintain accurate cosine similarity rankings.
- For large folders, use faiss.IndexIVFFlat for efficient searches without compromising accuracy.
