In this guide
Your RAG chatbot embeds documents into vectors, and at query time it needs the nearest ones — fast, from millions of candidates. A brute-force scan of every vector works on a laptop for a thousand documents and dies at a million. Vector databases solve the retrieval half of RAG: storing embeddings, indexing them for approximate nearest-neighbor search, and returning the most similar chunks in milliseconds.
This guide explains what vector databases actually do, how the indexing algorithms work, and compares Pinecone, Chroma, and FAISS honestly — including which one to pick for a student project and why the answer is usually not the one with the flashiest marketing.
What a vector database really is
An embedding model converts text into a list of numbers — a vector, typically 384 to 3072 dimensions. Texts with similar meaning get vectors that point in similar directions. A vector database stores these vectors and answers one question efficiently: given a query vector, which stored vectors are closest?
Closeness is measured by a distance metric:
| Metric | Formula intuition | When to use |
|---|---|---|
| Cosine similarity | Angle between vectors; ignores magnitude | Text embeddings (most common) |
| Euclidean (L2) | Straight-line distance | Image embeddings, some models |
| Dot product | Cosine × magnitudes | When magnitude carries meaning |
Most embedding models (sentence-transformers, OpenAI embeddings) are designed for cosine similarity. Normalize your vectors and cosine becomes equivalent to dot product — a handy simplification.
Note: A vector database is not magic. It is an index over high-dimensional vectors plus metadata filtering plus an API. Understanding the index is understanding the product.
The core problem: nearest neighbors at scale
Exact nearest-neighbor search compares the query against every stored vector: O(n·d) per query. For 1M vectors of 768 dimensions, that is ~768M operations per query — too slow for interactive use, though fine for small collections.
Vector databases use Approximate Nearest Neighbor (ANN) algorithms: they trade a small amount of accuracy (recall ~95–99%) for orders-of-magnitude speed. The two dominant families:
HNSW (Hierarchical Navigable Small World)
The workhorse of modern vector search. HNSW builds a multi-layer graph: each vector is a node connected to its nearest neighbors, with upper layers containing sparser "highway" nodes for long jumps. A query starts at the top layer, greedily moves to the closest node, drops to the next layer, and repeats — like navigating from a country map down to a street map.
- Query time is roughly logarithmic in collection size.
- Recall is tunable via the
ef(exploration factor) parameter: higher ef = slower but more accurate. - Memory-heavy: the graph adds overhead per vector. This is why HNSW on 1M vectors wants gigabytes of RAM.
Pinecone, Chroma (default), and Weaviate all use HNSW under the hood.
IVF (Inverted File Index)
FAISS's signature approach: cluster the vectors (k-means, say 1000 clusters), and at query time only search the clusters nearest the query (nprobe of them). Searching 5 of 1000 clusters means scanning ~0.5% of the data.
- Very memory-efficient, especially with product quantization (PQ) compressing vectors to a few bytes each.
- Slightly lower recall than HNSW at the same speed, but scales to billions of vectors on one machine.
- Needs a training step (clustering) before indexing — an extra operational wrinkle.
Pinecone vs Chroma vs FAISS: the honest comparison
| Aspect | Pinecone | Chroma | FAISS |
|---|---|---|---|
| What it is | Managed cloud service | Embeddable open-source DB (+ cloud option) | Library, not a database |
| Cost | Free tier, then usage-based | Free self-hosted | Free (your hardware) |
| Setup | API key, minutes | pip install chromadb, one file |
pip install faiss-cpu |
| Index | HNSW (managed) | HNSW (default) | IVF, HNSW, PQ — your choice |
| Metadata filtering | Yes, server-side | Yes | Manual (you filter IDs yourself) |
| Persistence | Automatic | Persistent client mode | You save/load the index file |
| Scale sweet spot | Millions, production | Thousands–millions, dev/prototype | Millions–billions, research |
The student-project answer: start with Chroma (or FAISS if you want to learn the algorithms). Chroma runs locally, persists to disk, needs no API key, and handles anything up to a few million vectors. Pinecone makes sense when you deploy something real and want to stop thinking about infrastructure — its free tier covers small production use. FAISS is the choice when you need maximum control, maximum scale per machine, or you are studying ANN algorithms themselves.
Minimal working examples
Chroma — the fastest path to a working RAG retriever:
import chromadb
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection("notes")
collection.add(
documents=["The ESP32 has built-in WiFi and Bluetooth.",
"LoRA fine-tuning trains small adapter matrices."],
ids=["doc1", "doc2"],
)
results = collection.query(query_texts=["wireless microcontroller"], n_results=1)
print(results["documents"])
FAISS — when you want explicit control over the index:
import faiss
import numpy as np
vectors = np.random.rand(10000, 384).astype("float32")
index = faiss.IndexHNSWFlat(384, 32) # 384 dims, 32 graph neighbors
index.add(vectors)
distances, ids = index.search(query_vector, k=5)
Chunking and embeddings: what actually determines retrieval quality
The database is rarely the bottleneck in a student RAG project — the chunks and embeddings are. Get these right before tuning the index:
- Chunk size: 200–500 tokens with ~10–20% overlap is a sane default. Too small and chunks lack context; too large and the relevant sentence drowns in noise.
- Chunk boundaries: split on semantic boundaries (paragraphs, sections), not arbitrary character counts, when you can.
- Embedding model: a decent open model (e.g. a sentence-transformers MiniLM or E5 variant) is enough to start. Upgrading the embedding model usually beats upgrading the database.
- Metadata: store source, page, section, date. Filtering by metadata before vector search ("only search 2024 docs") dramatically improves precision.
- Hybrid search: combining vector similarity with keyword (BM25) scores beats pure vector search on many real queries. Chroma and Pinecone both support hybrid patterns.
Common mistakes
- Choosing the database before having data. If you have 500 documents, brute-force NumPy search is fine. Do not build distributed infrastructure for a class project.
- Embedding with one model, querying with another. Query and documents must use the same embedding model — vectors from different models live in different spaces and are not comparable.
- Ignoring the recall/latency knob. HNSW's ef and IVF's nprobe trade speed for recall. If retrieval quality seems bad, raise the exploration parameter before blaming the embedding model.
- No metadata, no filtering. Pure similarity search over mixed content returns plausible-but-irrelevant chunks. Metadata filters are the cheapest quality win.
- Storing vectors without normalization. If your metric is cosine, normalize. Unnormalized vectors with cosine search silently return wrong rankings.
- Paying for Pinecone on day one. Prototype locally with Chroma; migrate when you have real traffic and a reason.
How to evaluate retrieval (before blaming the LLM)
When a RAG answer is wrong, the retriever is guilty more often than the generator. Build a small test set of questions with known-relevant document IDs, then measure:
- Recall@k: what fraction of queries have at least one relevant chunk in the top k?
- MRR (mean reciprocal rank): how high up is the first relevant chunk?
If recall@5 is low, fix retrieval (chunks, embeddings, hybrid). If recall is high but answers are bad, the problem is the generation step — prompt, context assembly, or the model. This separation saves enormous debugging time.
Where to go from here
Vector search is one half of RAG. For the full picture, read the RAG project guide, then learn how to evaluate whether your RAG actually works. If you are choosing the framework around your retriever, see LangChain vs LlamaIndex. For the evaluation-metrics foundations, the ML evaluation metrics guide covers precision, recall, and ranking metrics in depth. More retrieval and LLM topics in the AI & Machine Learning branch hub.