RAG with LLMs: Final-Year Project Guide

A large language model knows a lot, but it does not know your college's placement policy, your company's HR handbook, or the syllabus that changed last semester. Ask it anyway and it will answer confidently — sometimes correctly, sometimes inventing clauses that never existed. Retrieval-Augmented Generation (RAG) is the architecture that fixes this: instead of asking the model to recall facts from its training data, you retrieve the relevant passages from your own documents and hand them to the model with the question, so its answer is grounded in text you provided.

For final-year projects, RAG hits a sweet spot: it is genuinely useful (a campus FAQ bot, a placement assistant, a document Q&A system), it is buildable in a semester with free tools, and it demonstrates real engineering — chunking, embeddings, vector search, prompt design, evaluation — rather than a thin wrapper around a chatbot API. Live examples on this platform include a RAG-based AI placement assistant, a campus FAQ chatbot using RAG, and an AI college enquiry chatbot. This guide walks through the full build: when RAG is (and is not) the right choice, the pipeline piece by piece, the failure modes that make demos embarrassing, and how to evaluate the system honestly for your report.

What RAG is, precisely

A RAG system has two stages. Retrieval: the user's question is converted to a vector embedding, a vector database finds the most similar text chunks from your document collection, and those chunks are returned. Generation: the chunks plus the question are placed into a prompt, and the LLM writes an answer using the provided context. The model's parametric memory becomes a language engine; your documents become the facts.

The naive alternative — pasting the whole handbook into every prompt — fails the moment the corpus exceeds the context window or the API bill matters. The other alternative — fine-tuning the model on your documents — is heavier, slower to iterate on, and worse at citing sources. RAG's advantage for student projects is practical: add a document, re-index, and the bot knows it immediately, with no retraining.

When RAG is the right choice — and when it is not

Situation	RAG?	Why
Q&A over a bounded document set (handbooks, FAQs, syllabi, policies)	Yes — ideal	The corpus is finite, answers should cite sources, documents change over time
Questions need recent or private information the base model never saw	Yes	Retrieval supplies what training data lacks
Open-ended creative writing, brainstorming, coding help	No	No corpus to retrieve from; plain LLM prompting is the tool
Simple keyword lookup ("what is the last date for…")	Probably overkill	A search box over the documents is simpler and more reliable
Tiny corpus (one 2-page FAQ)	Marginal	Paste it into the prompt directly; RAG machinery adds nothing
The "documents" are constantly-changing live data (prices, seats)	RAG + live API	Retrieve the static context, call an API for the live values

Scope discipline: the strongest student RAG projects pick a corpus with a real owner and real questions — the placement cell's actual PDFs, the department's actual lab manuals. A bot over ten generic web articles demonstrates the pipeline but answers nothing anyone needs. Your viva goes better when the corpus is real.

The pipeline, piece by piece

1. Document ingestion

Start with the actual files: PDFs, Word docs, web pages, spreadsheets. Practical notes from real builds:

PDFs are messy. Tables, two-column layouts, headers/footers, and scanned pages all corrupt extraction. Use a proper parser (PyMuPDF/fitz or pdfplumber); for scanned PDFs you need OCR (Tesseract) first — budget time for this, it is routinely the longest step.
Clean aggressively. Strip repeated headers/footers, page numbers, and navigation chrome. Every junk line you leave in becomes a junk chunk that retrieval can surface.
Record metadata per document: title, source file, date/version, section. Metadata powers two things later: citations in answers ("According to the 2024 placement policy, section 3…") and filtering (only search the current year's documents).

2. Chunking: the decision that shapes everything

The LLM cannot read your whole corpus per question, so documents are split into chunks — typically a few hundred tokens each — and each chunk is embedded separately. Chunking strategy is the highest-leverage decision in a RAG build:

Strategy	Chunk size	Overlap	Fit for
Fixed-size with overlap	400–600 tokens, 50–100 token overlap	Yes	The default: simple, works for prose handbooks and policies
Sentence-aware / recursive	~500 tokens, split at sentence boundaries	Small	Cleaner chunks when sentences carry the meaning (FAQs, manuals)
Section-aware	One chunk per heading/section	N/A	Documents with clear structure (policy docs, syllabi) — preserves context
Small chunks	150–250 tokens	Yes	Precise fact lookup; needs more chunks retrieved per question

Rules that matter:

A chunk should be self-contained. If a chunk says "the deadline is 15 March" but the year and the programme it applies to were two chunks away, retrieval will serve confident nonsense. Prefer section-aware chunking for structured documents; add the section heading as a prefix to every chunk as cheap insurance.
Overlap is not optional for fixed-size chunking. Without it, a sentence split across a boundary is lost to both chunks. 10–20% overlap is the standard.
Tables need special handling. A chunked table is usually garbage. Either keep tables as whole chunks (one table = one chunk, with its caption) or convert key tables to sentences before chunking.

3. Embeddings

An embedding model converts each chunk (and each query) into a vector embedding — a list of floats, typically 384 to 1536 dimensions — where similar meanings land near each other. Your options:

Local, free: sentence-transformers models (e.g. all-MiniLM-L6-v2, 384 dimensions) run on CPU, cost nothing, and are plenty for a student corpus. This is the default recommendation.
API-based: commercial embedding APIs give stronger multilingual and domain performance at a per-token cost. Worth it only if your corpus is non-English or highly technical.

The critical constraint: the query embedding and the chunk embeddings must come from the same model. Re-index the whole corpus if you change embedding models — mixed embeddings are meaningless numbers.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
# chunks: list of strings, each one self-contained text chunk
embeddings = model.encode(chunks, show_progress_bar=True)
print(embeddings.shape)  # (num_chunks, 384)

4. The vector database

The vector DB stores chunk vectors and answers "which chunks are nearest to this query vector?" For a student project:

Option	Type	When to pick it
FAISS	Local library	Default for student builds: free, fast, no server, persists to disk
Chroma	Local / embedded DB	Nicest developer experience; metadata filtering built in
Pinecone / Weaviate Cloud	Hosted service	Only if you need multi-user production or your guide requires it

A few thousand chunks — a realistic student corpus — is trivially small for any of these; FAISS on a laptop answers queries in milliseconds. Do not let infrastructure distract from chunking and evaluation, where the actual quality lives.

import faiss
import numpy as np

dimension = embeddings.shape[1]
index = faiss.IndexFlatL2(dimension)   # exact search; fine up to ~1M vectors
index.add(np.array(embeddings).astype("float32"))
faiss.write_index(index, "corpus.faiss")  # persist it

5. Retrieval: top-k and the knobs around it

At query time: embed the question, fetch the top-k nearest chunks, stuff them into the prompt. The knobs:

top-k (how many chunks): 3–5 is the standard starting point. Too few and the answer lacks context; too many and the prompt fills with marginally relevant text that dilutes the good chunks and costs tokens. Tune this on real questions, not guesses.
Similarity threshold: optionally drop chunks below a minimum score — better to say "I don't know" than to answer from a weakly related chunk.
Hybrid search: combine vector similarity with keyword (BM25) search. Vector search finds "what is the leave policy" → chunks about leave; keyword search catches exact terms like form numbers and regulation codes that embeddings sometimes miss. For corpora full of codes, dates, and names, hybrid measurably helps.
Query rewriting: for follow-up questions ("what about for MTech students?"), rewrite the query with conversation history before embedding, so the retrieval sees a standalone question.

6. Generation: the prompt that keeps the model honest

The system prompt is your hallucination firewall. A structure that works:

You answer questions using ONLY the context below.
- If the answer is not in the context, say "I don't have information about that in the provided documents."
- Quote or cite the source section for each factual claim: [Source: <document title>, <section>].
- Do not use knowledge outside the context. Do not guess dates, numbers, or names.

Context:
<retrieved chunks, each labelled with its source>

Question: <user question>

Calling an LLM API from Python (the WAF-safe pattern — plain requests, no browser JS):

import requests

response = requests.post(
    "https://api.openai.com/v1/chat/completions",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "model": "gpt-4o-mini",
        "messages": [
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": "Context:\n" + context + "\n\nQuestion: " + question},
        ],
        "temperature": 0.2,   # low temperature: factual answers, less creativity
    },
    timeout=60,
)
answer = response.json()["choices"][0]["message"]["content"]

Low temperature (0–0.3) is deliberate: you want the model to extract and synthesise, not to invent. High temperature in a RAG system is a hallucination subsidy.

Hallucination checks: proving the answer came from your documents

"Doesn't hallucinate" is a claim your examiners will probe, so build the checks into the system rather than asserting it:

Refuse outside the corpus. The prompt's "say you don't know" instruction, plus a similarity threshold on retrieval, means unknown questions get an honest refusal instead of a fabricated answer. Demo this deliberately — ask the bot something outside its documents and show the refusal. It is a feature, not a failure.
Citations on every factual claim. Each answer names the source document and section. In your demo, click through a citation to the source passage. Examiners remember this.
Faithfulness spot-checks. For your report, take 30–50 real questions, and for each answer mark every factual claim as supported / unsupported / contradicted by the retrieved chunks. Report the supported fraction. This manual audit is honest, reproducible, and far more convincing than a vague "accuracy 95%".
Adversarial questions. "What is the refund policy for 2026?" when the corpus only covers 2024. "Who is the HOD?" when no document names them. Log what the system does — refusal, hedged answer, or hallucination — and fix the failures before the viva, not during it.

What not to claim: never report a single "accuracy %" for a RAG system as if it were a classifier. Answer quality is multi-dimensional (did it retrieve the right chunks? is the answer faithful? is it complete?). Report the dimensions separately — retrieval hit-rate and faithfulness fraction are the two numbers that matter.

Evaluation for your report

Examiners accept evaluation they can reproduce. A student-grade evaluation plan:

Build a test set of 40–60 questions with known answers from the corpus: factual lookups ("what is the minimum attendance?"), multi-hop questions ("which companies allowed ECE students last year?"), and unanswerable questions (correct behaviour: refusal).
Retrieval evaluation: for each question, note which chunks should answer it; measure hit-rate — the fraction of questions where a gold chunk appears in the top-k. Target 0.80+ as a design goal; below that, fix chunking before touching anything else.
Answer evaluation: human-judge each answer on faithfulness (every claim traceable to context), completeness (did it use all relevant retrieved facts?), and refusal correctness on unanswerable questions. A simple 3-column spreadsheet is the artifact; summarise it as a table in your report.
Latency and cost: measure end-to-end response time and tokens per query on your test set. "Average 4.2 s and ~1,800 tokens per query" is a real engineering result that belongs in the report.

This evaluation is also your debugging loop: questions that fail reveal whether the problem is retrieval (wrong chunks — fix chunking/embeddings/top-k) or generation (right chunks, bad answer — fix the prompt). Diagnose at the stage level; the table below helps.

Failure diagnosis table

Symptom	Likely stage	Check and fix
"I don't have information" for questions the corpus answers	Retrieval	Chunks too large or badly split; embedding model mismatch; top-k too small; try hybrid search
Answer cites the wrong document	Retrieval	Overlapping/duplicated content across documents; add metadata filtering by document version
Right chunks retrieved, answer still wrong	Generation	Prompt too permissive; lower temperature; tighten the "use only context" instruction
Answers vary wildly between runs	Generation	Temperature too high; set 0–0.2 and re-test
Good on short docs, bad on the full corpus	Chunking/scale	Chunk quality degrades with boilerplate; clean ingestion, section-aware chunking
Slow responses	Retrieval + API	Reduce top-k; cache embeddings; check whether the bottleneck is the LLM call (it usually is)
Works in English, fails in Hindi/Marathi queries	Embeddings	`all-MiniLM-L6-v2` is English-centric; switch to a multilingual embedding model and re-index

RAG vs fine-tuning: the decision students get wrong

"Should we fine-tune the LLM on our documents instead?" comes up in every RAG viva. The honest answer is a comparison, not a slogan:

Dimension	RAG	Fine-tuning
New/updated documents	Re-index in minutes; bot knows them immediately	Retrain or re-tune; hours and GPU cost per update
Citing sources	Natural — chunks carry metadata	Hard — the knowledge is baked into weights
Refusing unknown questions	Achievable via prompt + retrieval threshold	Difficult — tuned models answer confidently from memory
Behaviour/style changes	Limited — the base model's personality stays	Strong — this is what fine-tuning is actually good at
Cost for a student project	Embedding API or local model: near-zero	GPU hours for training; iteration is slow
Hallucination control	Grounded in retrieved text; auditable	Reduced but not eliminated; harder to audit

The rule: fine-tune for behaviour, retrieve for knowledge. If your documents change (policies update yearly, FAQs grow), RAG wins. If you need the bot to speak in a specific style or follow a rigid output format, consider fine-tuning — or, more practically for a student timeline, achieve most of it with careful prompting and few-shot examples in the system prompt. Many strong final-year builds are RAG plus excellent prompting, with no fine-tuning at all. Say this explicitly in your report's "design decisions" section; it shows you chose the architecture rather than inheriting it.

Cost and latency math for your report

Examiners like numbers that prove you engineered the system, not just assembled it. Measure these on your test set:

Tokens per query: retrieved chunks dominate. With top-5 chunks of ~500 tokens plus prompt overhead, expect ~3,000–4,000 input tokens per query. At typical API pricing for a mini-class model, that is a fraction of a rupee per query — compute your actual figure and state it: "average 3,400 input tokens and 220 output tokens per query."
Latency budget: embedding the query (~100 ms local), FAISS search (~10 ms), LLM generation (the bulk — 2–8 s depending on model and output length). If your demo needs sub-3-second responses, the levers are: smaller/faster model, fewer output tokens (instruct brevity in the prompt), and streaming the response token-by-token so the UI feels instant.
Index build time: embedding a few thousand chunks with a local sentence-transformer takes minutes on CPU. Rebuilds are cheap — which is why "just re-index when documents change" is a credible operations story in your report.

Keeping the index fresh: the unglamorous half of RAG

A campus bot that answers from last year's handbook is worse than no bot. Build the update path before submission:

Version your documents. Filenames or metadata carry the effective date (placement-policy-2025.pdf, not policy-final-v2.pdf). Retrieval filters to current versions; old ones stay for "what was the rule last year?" questions.
Re-indexing is a script, not a ceremony. One command — ingest, clean, chunk, embed, rebuild FAISS — runnable by whoever owns the corpus next year. Document it in your report's deployment section.
Log unanswered questions. Every "I don't have information about that" is a signal: either the corpus has a gap (add the document) or retrieval failed (fix chunking). A month of logs is the most valuable evaluation data you will ever get, and mentioning the logging pipeline in your viva shows production thinking.

What to demo, and in what order

A RAG demo that lands well follows this script: (1) a normal question with the answer and its citations, clicking through to the source passage; (2) a question requiring two chunks combined (shows synthesis, not copy-paste); (3) a question outside the corpus, showing the honest refusal; (4) the same question asked with a document added five minutes ago (shows the update path). Four questions, five minutes, every architectural claim demonstrated live. Prepare a recorded fallback — API outages during demos are a rite of passage, and the recording also becomes your report's appendix.

Putting it together: a sensible build order

Corpus first: collect the real documents, clean them, decide chunking by inspecting actual chunks (read 30 of them — this is not optional).
Minimal pipeline: embed → FAISS → top-3 → prompt → answer, in one Python script. Get one good answer end-to-end before adding features.
Test set: write the 40–60 questions early; they become your regression suite for every later change.
Iterate on retrieval: chunking strategy, top-k, hybrid search — each change measured against the test set.
Harden generation: citations, refusal behaviour, temperature.
Interface last: a simple web UI (Flask/Streamlit) or WhatsApp-style chat page. The deployment guide covers putting the service behind an API; the testing and debugging guide covers the pre-submission discipline.

Related builds to study: the RAG placement assistant for a placement-cell corpus, the campus FAQ chatbot for FAQ-style chunking, and the AI college enquiry chatbot for the enquiry-desk variant. For choosing the project itself, see the machine learning project ideas guide, and for writing it all up, how to write your final-year project report.

A RAG project done this way — real corpus, inspected chunks, measured retrieval, cited answers, honest refusals — is one of the most defensible builds a student can present: every design decision has a test behind it, and every limitation is documented rather than hidden. That is what turns "a chatbot project" into an engineering project.

RAG with LLMs for Final-Year Projects: Chunking, Embeddings, Vector Search and Honest Evaluation