Grading is one of those tasks that's simultaneously mechanical and irreducibly judgment-heavy: applying a rubric consistently across fifty submissions is tedious in a way that invites shortcuts, but every one of those fifty submissions can be wrong in a slightly different way that a rigid rubric doesn't anticipate. A single LLM call that reads a rubric and a scanned answer sheet and spits out a score doesn't actually solve this — it just moves the inconsistency somewhere less visible. The Auto-Assessment Agent is my attempt at doing better: a pipeline of narrow, single-purpose agents that transcribe, solve, evaluate, and audit a submission in separate steps, so that every score is traceable back to a specific quote from the student's actual work rather than a single opaque judgment call.
The pipeline
The system handles PDFs, images, plain text, and DOCX, and grades either a single submission or a whole batch against the same rubric. The shape of the pipeline is deliberately linear and inspectable:
| Stage | Role | Runs on |
|---|---|---|
| Transcriber | Reads handwritten/scanned PDFs and images into structured text, preserving notation, tables, and page layout | Gemini (vision) |
| Solver | Generates a reference solution when no official model answer is supplied | Gemini |
| Evaluator | Grades criterion-by-criterion with verbatim evidence quotes; for diagram/sketch/construction questions, looks at the original page images directly rather than trusting the transcript alone | Gemini |
| Auditor | Checks score bounds and criterion-total arithmetic | Python |
| Regrade Agent | Re-checks a specific disputed criterion against the stored evidence before changing a score | Gemini |
| Chat Agent | Answers follow-up questions grounded in the saved assessment, favoring Socratic guidance over handing over answers | Gemini |
The row I'd call out first is the Auditor, and specifically that it isn't an LLM call. Score arithmetic — does each criterion's awarded points fall within its stated bounds, does the total actually sum to what's reported — is exactly the kind of thing a language model can get subtly wrong under load, and exactly the kind of thing a few lines of Python can check with certainty. Whenever a task decomposes into "requires judgment" and "requires arithmetic," it's worth asking whether the arithmetic half needs a model at all.
Layers under the agentic construction
The six-stage table above is the pipeline's public shape. What actually makes it hold together under real input — illegible handwriting, a student quoting their own answer sheet dishonestly, a rubric with an embedded "ignore previous instructions" line, a schema with too many optional fields — sits one level below that, in four mechanisms that recur across every agent rather than living in any one of them. None of these is a new algorithm; each is a known idea from the broader agents literature. What's specific to this project is applying all four consistently to the same pipeline, and using plain code instead of a second model call wherever the check can be made deterministic.
Layer 1 — schema-constrained generation, with rejection as the retry signal
Every agent call goes through one function, json_response(), which hands Gemini's structured-output mode a Pydantic model's JSON schema and parses the reply straight back into that model. The schemas carry their own invariants as model_validator hooks — a criterion's score can't exceed its weight, an AssessmentReport can't come back with zero evaluations — and a violation raises before the pipeline ever sees the malformed object. call_with_retries treats that raised ValidationError as just another retryable failure alongside a 429 or a 500, so a broken JSON object doesn't fail the request; it silently asks the model again. This is generate-validate-retry rather than generate-and-hope, and the "validate" step is ordinary Pydantic, not a second LLM call grading the first one's output.
The decomposition of the pipeline itself is downstream of the same idea. Reference-diagram generation for questions asking for a sketch or construction used to live inside the main grading schema, as one more optional field per question. It was pulled out into its own isolated call after it measurably increased the rate of the model returning an empty evaluations list — a bigger, more heavily-conditioned schema made the one call that matters most (the actual score) less reliable. That's a concrete, observed instance of a general pattern: schema complexity trades against structured-output reliability, and the fix isn't a smarter prompt, it's fewer optional fields per call.
Layer 2 — grounding checked after the fact, not just requested up front
Asking a model to cite evidence is not the same as verifying it did. The Evaluator is instructed to copy a verbatim evidence_quote for every criterion, but the AuditAgent doesn't take that on faith: it normalizes whitespace and case, then substring-checks every quote against the actual transcript. A quote that isn't literally present flags the question needs_human_review instead of being trusted because it looked plausible. The same substring check is reused on the other side of the system, to validate a human's own evidence_quote in a regrade request before the Regrade Agent is even called — a dispute pointing at text that was never written doesn't get to consume a model call.
The Auditor also catches a second, subtler failure that has nothing to do with the transcript: a question scored above zero with an empty student_answer field is a contradiction between two things the same model call generated, and it gets flagged rather than passed through. This is closer to a self-consistency check across a single generation's own fields than to retrieval-style grounding, and it costs nothing beyond an if statement — the recurring theme is that a lot of what looks like it needs a second opinion from another model call can instead be checked in a few lines of Python, because the thing being checked (is this string present, do these two fields agree) doesn't require judgment, just comparison.
Layer 3 — every upload is adversarial input, on both the way in and the way out
A single constant, UNTRUSTED_DATA_RULE, is prepended to every agent prompt that touches uploaded content: the question paper, rubric, answer key, and student submission are reference data, never instructions, and any embedded directions inside them that try to change behavior, leak the prompt, or dictate a score are to be ignored. Every one of those sections is also wrapped in explicit === BEGIN X === / === END X === delimiters before it reaches the model. This is the instructional-hierarchy idea — keep untrusted content clearly fenced off from instructions — applied as one shared helper rather than as a patch on whichever endpoint happened to get a bad upload first.
The same posture holds on the way back out. The Evaluator can generate reference-diagram SVGs that get rendered directly in the browser, and sanitize_reference_svg strips anything with a <script>, an event-handler attribute, a <foreignObject>, or an external href/src, even though the prompt already tells the model not to produce any of those — the code comment is blunt about why: "a prompt instruction is not a security boundary." The regrade path applies the same distrust to the model's own edits: after a confirmed dispute, the code forcibly restores the question's question_text, student_answer, and reference_diagram_svg from the original record and clamps the score to the original max_score, so a regrade can correct a score and its feedback but can never quietly rewrite what was actually asked or what the student actually wrote.
Layer 4 — correction memory shared across requests, not just within one
Most agent memory patterns operate within a single conversation or a single user's history. This one deliberately crosses both boundaries: a regrade dispute confirmed for one student becomes a known_corrections note injected into the Evaluator's prompt for every other student subsequently graded against that exact question paper, and per-student weak-area memory is keyed to a server-verified Google identity rather than to a session or browser fingerprint that a client could spoof. The mechanics of both stores are covered in the next section; the architectural point here is narrower — this is state shared across concurrent invocations of the pipeline serving different end users, not state private to one agent's own reasoning loop, which is also why the orchestrator class is written to hold no per-request instance state at all: a single shared instance serves every request, so anything like "the current assessment" would leak between users if it were stashed on self instead of passed explicitly through every method call.
Design principles
- Evidence before assertion. Every grading decision cites the student's actual work — a verbatim quote from the transcript, or for diagram questions, the actual page image. A score without an attached quote isn't trustworthy on its own terms; it's just an assertion.
- Separate responsibilities. Transcription, solving, evaluation, auditing, regrading, and chat are distinct steps with their own contracts. If the evaluator hallucinates a criterion score, that's traceable to one stage rather than buried inside one long, undifferentiated LLM call.
- Deterministic where possible. Score arithmetic is checked in Python, not by another LLM call, for the reason above.
- Explicit uncertainty. Illegible or ambiguous handwriting sets a
needs_human_reviewflag rather than the system silently guessing and reporting a confident-looking number. - Actionable feedback. Every question gets a concrete "what to do differently next time," not just right/wrong.
The diagram-question handling under "evidence before assertion" is worth dwelling on, because it's a real, specific failure mode rather than a hypothetical one: OCR-style transcription of a hand-drawn circuit diagram or geometric construction is unreliable in a way that transcription of handwritten prose usually isn't — a transcript can describe a diagram as "roughly correct" while missing the one connection that makes it wrong. So for exactly those question types, the Evaluator is given the original page image alongside the transcript instead of trusting the transcript's description of it. It's a narrow fix, but it's the kind of fix you only find by looking at where a general-purpose pipeline actually breaks on a specific input type.
Agentic memory: two feedback loops, not one
The part of this project I find most interesting isn't the per-submission pipeline — it's that the Evaluator isn't stateless across runs. Two lightweight memory stores feed back into every grading call:
Per-student weak-area memory
After each assessment, recurring weak concepts are tracked per signed-in student — strengthening on repeat misses, fading once mastered — and fed back into the next grading pass. This is what lets feedback say a mistake persisted or improved, rather than repeating the same generic tip on every submission regardless of whether the student already acted on it.
Cross-submission grading corrections
When a "Request re-evaluation" confirms a genuine grading mistake, that correction is remembered against the exact question paper. The next student graded on the same test doesn't get the same mistake repeated — a dispute from one student improves grading consistency for everyone who took the same assessment.
Both of these matter for the same underlying reason: a grading system that treats every submission as a fresh, context-free judgment call will make the same category of mistake over and over, on the same test, for different students, and will give the same generic feedback to a student who's already fixed the gap it's pointing at. Identity for this memory is the signed-in Google account, proven through a server-issued, HMAC-signed session token — not a self-reported header, and not the anonymous per-browser ID used for cosmetic session state elsewhere in the app. That distinction matters because weak-area memory is exactly the kind of state you don't want to be spoofable by just changing a client-side value.
Disputing a grade
The regrade flow is designed to avoid both failure modes you'd expect from an automated dispute system: blindly trusting the original score, and blindly trusting whatever the student claims. A disputed criterion is re-checked against the stored evidence quote before any score changes, and the request itself has to point at something concrete:
curl -X POST "http://127.0.0.1:8000/api/regrade" \
-H "Content-Type: application/json" \
-d '{
"assessment_id": "YOUR_ASSESSMENT_UUID",
"question_id": "Question 1",
"claimed_mistake": "You said I did not show 2x = 12, but it appears in my solution.",
"evidence_quote": "2x = 12"
}'If the Regrade Agent confirms the mistake, the correction feeds the cross-submission memory described above. If it doesn't, the original score stands with an explanation — the point isn't to make disputes always succeed, it's to make the resolution process itself evidence-anchored rather than another opaque judgment call.
Stack and shape of the code
The backend is FastAPI with Pydantic contracts between agent stages, SQLite for persistence, and the Google Gen AI SDK for the Gemini calls (plus the OpenAI SDK for Bodhan's text-to-speech in Agent Chat). The frontend is React and Vite, with react-markdown and KaTeX for rendering the math that shows up in a lot of these submissions. Both ship together via Docker Compose, with the frontend served through nginx and proxying /api and /ws to the backend container.
Model choice is centralized rather than hard-coded per call: GET /api/models reports the live configuration for every pipeline stage, and the frontend's own Models page reads that endpoint directly instead of duplicating a static list that can drift out of sync with what's actually deployed.
What I'd still change
- The weak-area memory currently strengthens on repeat misses and fades on mastery, but the exact decay schedule is a heuristic, not something validated against how students actually retain and re-lapse on a concept — it would be worth checking against real longitudinal grading data rather than intuition.
- History currently keeps the five most recent assessment runs per login (a batch counts as one run). That's a reasonable default for a small deployment, but it means the cross-run weak-area signal is bounded by how much history survives, which matters more for a semester-long course than for a one-off quiz.
- The diagram/sketch fallback to raw page images is a targeted fix for one known failure mode; there are probably other question types with similarly specific transcription failure modes I haven't hit yet.
This is a personal project write-up, not a paper. The repository is at github.com/aayushmanda/da7016_project. An LLM was used for organizing this write-up from the project's own README and code; the architecture, design decisions, and code described are my own.