← All selected work
01 / Applied AI

Policy Copilot

Policy questions answered against source documents, with paragraph citations and a clear abstention when the evidence is insufficient.

PythonRAGBM25StreamlitEvaluation
Academic dissertation
Context
Academic dissertation
My contribution
Designed, built and evaluated the retrieval and verification workflow.
Evidence
63 synthetic benchmark questions
THE PROBLEM

Start with the need.

A fluent answer is not necessarily a supported one. I explored how a question-answering system could make its evidence inspectable and refuse questions the source material could not support.

My contribution

I built and evaluated the retrieval, confidence gates, citation checks and audit workbench as my University of Leeds dissertation. The implementation combines stable paragraph identifiers, retrieval and reranking with claim-level checks and an extractive fallback.

How it works

STEP 01Policy documents
STEP 02Retrieve & rerank
STEP 03Verify evidence
STEP 04Answer or abstain

Conceptual flow. Implementation details and reproducibility instructions are in the repository.

A closer look at the work.

Repository workbench figure showing a remote-work policy question, a supported answer and paragraph references.
Workbench figure from the public dissertation repository. View original figure

The decisions behind the code.

Make evidence traceable

Stable paragraph identifiers connect each surfaced answer to its source and audit record.

Treat abstention as a design choice

Confidence gates and citation checks reduce unsupported output, but a system that refuses too often is less useful.

Keep an offline path

BM25 and extractive mode allow the core workflow to run without a paid language-model API.

What the evidence says.

The repository documents a 63-question synthetic benchmark and a 292-test core suite. It reports that the stricter configuration improved grounding while reducing answer coverage. These are documented project results, not tests rerun by this website.

Public repository README, reviewed 17 November 2026.

Read the source documentation

Where the boundaries are.

  • A small synthetic benchmark does not establish performance on every organisation's policies.
  • Dense retrieval was unavailable in one reproduction environment, which used the documented BM25 fallback.
  • Some generative adversarial evaluation was limited by API quota. Citation heuristics and confidence gates do not guarantee correctness.

What I learned

Reliability and usefulness have to be measured together. Better grounding is valuable only if people can still get enough useful answers.

UP NEXT

EventHub

Read the next case study
OPEN TO ROLES & PROJECT CONVERSATIONS

Have a role or
project in mind?

Get in touch