Professor Digital Twin: Trustworthy Answers From an LLM That Knows When to Stop
TL;DR
- Situation: An "AI professor" is only useful if students can trust it. Frontier chatbots hallucinate citations, invent quotes, and confidently answer questions they shouldn't.
- My decision: Fine-tune a small open-weight model (Mistral-7B-Instruct-v0.2) on Prof. James Brusseau's actual writing in two stages — voice first, then citation-grounding — and add a retrieval layer with a calibrated abstention threshold so the system says "I don't know" instead of guessing.
- Outcome: Prof. Brusseau tested the outputs and endorsed them. RAG corpus: ~1,184 chunks from course readings and papers. Abstention triggers at a retrieval distance of 0.85, which we calibrated against a held-out set of "should have refused" prompts.
The problem
A course-facing "digital twin" of a professor has to do one thing that no general chatbot does well: it has to answer only from the professor's actual work, cite where it got the answer, and refuse when the answer isn't in the material. Every mode where it hallucinates a source, paraphrases a stance the professor doesn't hold, or over-answers a question the professor would have declined to answer, it becomes worse than useless — it becomes actively misleading.
Frontier models will happily do all three of those failure modes if you let them. Grounding is the only defense.
The constraint
The system had to run on modest hardware (LoRA on a single consumer GPU), be inspectable end-to-end (no black-box hosted service), and be verifiable by the professor himself. That last constraint is the hard one — the professor has to be able to read a generated answer and either sign off on it or point at a specific failure. If he can't, the twin isn't a twin, it's a costume.
What I killed: pure RAG with a big base model
The obvious path was: take Llama or Mistral out of the box, put the professor's corpus in a vector store, retrieve top-k chunks at query time, stitch them into the prompt, done. The problem: without fine-tuning, the voice is wrong. The model reasons like a general-purpose assistant, not like the professor. Retrieval solves "what did he write about X" but not "how would he frame the answer." I killed that path in favor of two-stage LoRA before adding retrieval.
What I shipped: two-stage LoRA + calibrated RAG
Stage 1 — Voice fine-tune. LoRA adapter over Mistral-7B-Instruct-v0.2, trained on curated excerpts from the professor's published work. The goal here is stylistic fidelity: how sentences start, which framings the professor picks, what he chooses not to answer.
Stage 2 — Citation-grounding fine-tune. A second LoRA layer on top, trained on Q/A pairs where every answer includes a verbatim or near-verbatim citation from the source material. This teaches the model to attribute rather than paraphrase.
Retrieval layer. ChromaDB with
BAAI/bge-large-en-v1.5 embeddings over ~1,184 chunks of course
readings and paper excerpts. A cross-encoder re-ranker sits between
retrieval and generation to promote the chunks that actually answer the
question over the chunks that just mention the query terms.
Calibrated abstention. If the top retrieved chunk has a
distance greater than 0.85 (a threshold I calibrated against a
held-out set of "professor would have refused" prompts), the system does
not generate an answer — it returns a refusal that surfaces the closest
material and lets the user decide whether to press on. This is the single
most important safety feature in the whole system.
Why the abstention threshold matters
The distance threshold is the whole game. Set it too low and the system refuses questions it could have answered well from the corpus. Set it too high and it will confidently answer questions where the retrieval didn't actually find relevant material — the exact failure mode we're trying to avoid.
I calibrated the threshold by building a small set of "should refuse"
prompts — questions the professor's published work does not address —
and finding the smallest distance value where the refusal rate on that set
was still > 95% while the false-refusal rate on genuine questions
stayed low. 0.85 was the number that came out of that
calibration.
What I'd measure
The system fails in two directions: (1) it answers when it should refuse (hallucination risk), or (2) it refuses when it should answer (usefulness loss). I'd track both, with the professor as the source of truth on a sampled review queue. If false-answer rate rises above ~2%, I'd raise the abstention threshold. If false-refusal rate rises above ~10%, I'd re-examine the retrieval side (chunk size, re-ranker, embedding model) before touching the threshold.
What I'd do differently
I'd instrument citation-verification as a runtime check, not a training objective. Right now Stage 2 teaches the model to cite; a runtime pass could programmatically check that every citation the model emits actually appears verbatim in the retrieved chunks, and refuse the answer if it doesn't. Belt-and-suspenders.
Advisors
Research advised by Prof. James Brusseau (Pace University, Applied Ethics of AI). System design discussed with Prof. Sara Falcone (Pace University, Seidenberg School of CSIS).
Read next: NewsScope / VERA — the first-author paper on schema-grounded claim extraction that uses similar LoRA + retrieval techniques.