Grading is one of those tasks that's simultaneously mechanical and irreducibly judgment-heavy: applying a rubric consistently across fifty submissions is tedious in a way that invites shortcuts, but every one of those fifty submissions can be wrong in a slightly different way that a rigid rubric doesn't anticipate. A single LLM call that reads a rubric and a scanned answer sheet and spits out a score doesn't actually solve this — it just moves the inconsistency somewhere less visible. The Auto-Assessment Agent is my attempt at doing better: a pipeline of narrow, single-purpose agents that transcribe, solve, evaluate, and audit a submission in separate steps, so that every score is traceable back to a specific quote from the student's actual work rather than a single opaque judgment call.
The pipeline
The system handles PDFs, images, plain text, and DOCX, and grades either a single submission or a whole batch against the same rubric. The shape of the pipeline is deliberately linear and inspectable:
| Stage | Role | Runs on |
|---|---|---|
| Transcriber | Reads handwritten/scanned PDFs and images into structured text, preserving notation, tables, and page layout | Gemini (vision) |
| Solver | Generates a reference solution when no official model answer is supplied | Gemini |
| Evaluator | Grades criterion-by-criterion with verbatim evidence quotes; for diagram/sketch/construction questions, looks at the original page images directly rather than trusting the transcript alone | Gemini |
| Auditor | Checks score bounds and criterion-total arithmetic | Python |
| Regrade Agent | Re-checks a specific disputed criterion against the stored evidence before changing a score | Gemini |
| Chat Agent | Answers follow-up questions grounded in the saved assessment, favoring Socratic guidance over handing over answers | Gemini |
The row I'd call out first is the Auditor, and specifically that it isn't an LLM call. Score arithmetic — does each criterion's awarded points fall within its stated bounds, does the total actually sum to what's reported — is exactly the kind of thing a language model can get subtly wrong under load, and exactly the kind of thing a few lines of Python can check with certainty. Whenever a task decomposes into "requires judgment" and "requires arithmetic," it's worth asking whether the arithmetic half needs a model at all.
Design principles
- Evidence before assertion. Every grading decision cites the student's actual work — a verbatim quote from the transcript, or for diagram questions, the actual page image. A score without an attached quote isn't trustworthy on its own terms; it's just an assertion.
- Separate responsibilities. Transcription, solving, evaluation, auditing, regrading, and chat are distinct steps with their own contracts. If the evaluator hallucinates a criterion score, that's traceable to one stage rather than buried inside one long, undifferentiated LLM call.
- Deterministic where possible. Score arithmetic is checked in Python, not by another LLM call, for the reason above.
- Explicit uncertainty. Illegible or ambiguous handwriting sets a
needs_human_reviewflag rather than the system silently guessing and reporting a confident-looking number. - Actionable feedback. Every question gets a concrete "what to do differently next time," not just right/wrong.
The diagram-question handling under "evidence before assertion" is worth dwelling on, because it's a real, specific failure mode rather than a hypothetical one: OCR-style transcription of a hand-drawn circuit diagram or geometric construction is unreliable in a way that transcription of handwritten prose usually isn't — a transcript can describe a diagram as "roughly correct" while missing the one connection that makes it wrong. So for exactly those question types, the Evaluator is given the original page image alongside the transcript instead of trusting the transcript's description of it. It's a narrow fix, but it's the kind of fix you only find by looking at where a general-purpose pipeline actually breaks on a specific input type.
Agentic memory: two feedback loops, not one
The part of this project I find most interesting isn't the per-submission pipeline — it's that the Evaluator isn't stateless across runs. Two lightweight memory stores feed back into every grading call:
Per-student weak-area memory
After each assessment, recurring weak concepts are tracked per signed-in student — strengthening on repeat misses, fading once mastered — and fed back into the next grading pass. This is what lets feedback say a mistake persisted or improved, rather than repeating the same generic tip on every submission regardless of whether the student already acted on it.
Cross-submission grading corrections
When a "Request re-evaluation" confirms a genuine grading mistake, that correction is remembered against the exact question paper. The next student graded on the same test doesn't get the same mistake repeated — a dispute from one student improves grading consistency for everyone who took the same assessment.
Both of these matter for the same underlying reason: a grading system that treats every submission as a fresh, context-free judgment call will make the same category of mistake over and over, on the same test, for different students, and will give the same generic feedback to a student who's already fixed the gap it's pointing at. Identity for this memory is the signed-in Google account, proven through a server-issued, HMAC-signed session token — not a self-reported header, and not the anonymous per-browser ID used for cosmetic session state elsewhere in the app. That distinction matters because weak-area memory is exactly the kind of state you don't want to be spoofable by just changing a client-side value.
Disputing a grade
The regrade flow is designed to avoid both failure modes you'd expect from an automated dispute system: blindly trusting the original score, and blindly trusting whatever the student claims. A disputed criterion is re-checked against the stored evidence quote before any score changes, and the request itself has to point at something concrete:
curl -X POST "http://127.0.0.1:8000/api/regrade" \
-H "Content-Type: application/json" \
-d '{
"assessment_id": "YOUR_ASSESSMENT_UUID",
"question_id": "Question 1",
"claimed_mistake": "You said I did not show 2x = 12, but it appears in my solution.",
"evidence_quote": "2x = 12"
}'If the Regrade Agent confirms the mistake, the correction feeds the cross-submission memory described above. If it doesn't, the original score stands with an explanation — the point isn't to make disputes always succeed, it's to make the resolution process itself evidence-anchored rather than another opaque judgment call.
Stack and shape of the code
The backend is FastAPI with Pydantic contracts between agent stages, SQLite for persistence, and the Google Gen AI SDK for the Gemini calls (plus the OpenAI SDK for Bodhan's text-to-speech in Agent Chat). The frontend is React and Vite, with react-markdown and KaTeX for rendering the math that shows up in a lot of these submissions. Both ship together via Docker Compose, with the frontend served through nginx and proxying /api and /ws to the backend container.
Model choice is centralized rather than hard-coded per call: GET /api/models reports the live configuration for every pipeline stage, and the frontend's own Models page reads that endpoint directly instead of duplicating a static list that can drift out of sync with what's actually deployed.
What I'd still change
- The weak-area memory currently strengthens on repeat misses and fades on mastery, but the exact decay schedule is a heuristic, not something validated against how students actually retain and re-lapse on a concept — it would be worth checking against real longitudinal grading data rather than intuition.
- History currently keeps the five most recent assessment runs per login (a batch counts as one run). That's a reasonable default for a small deployment, but it means the cross-run weak-area signal is bounded by how much history survives, which matters more for a semester-long course than for a one-off quiz.
- The diagram/sketch fallback to raw page images is a targeted fix for one known failure mode; there are probably other question types with similarly specific transcription failure modes I haven't hit yet.
This is a personal project write-up, not a paper. The repository is at github.com/aayushmanda/da7016_project. An LLM was used for organizing this write-up from the project's own README and code; the architecture, design decisions, and code described are my own.