02 / 03 · FINAL-YEAR COMPUTER-SCIENCE PROJECT

AI AGENT SUPPORT CLASSIFIER

A support-triage system combining deterministic checks, structured AI analysis and explicit human-approval gates.

WATCH THE CLASSIFIER DEMO

HUMAN-APPROVAL CONSOLE · SANDBOX WORKSPACE · FIG. 7

STATUS: Completed · working artefact + 67-page dissertation
ROLE: Sole author · system, evaluation, dissertation
CONTEXT: Final-year project (COMP390) · University of Liverpool
TIMELINE: 2025 / 26
PLATFORM: Python · Zendesk

01 · AT A GLANCE

01
Sole author of the full system: agent, policy engine, operator console, test suite and dissertation.
02
Deterministic-first design: the LLM is consulted only on seven explicit ambiguity triggers, and every write into Zendesk is verified by re-reading it.
03
Evaluated honestly: 100% route accuracy on the eight-case labelled sandbox suite (n = 8), with 75% of cases routed to human review by design.

STATUS: Completed final-year project
ROLE: Sole author · system, evaluation, dissertation

WATCH THE DEMO ↓

02 · SYSTEM DEMONSTRATION

Both operator console screens of the triage system on a dark frame — Recorded demonstration: the system reading a live Zendesk view, classifying, and writing verified updates back, with the operator console approving low-confidence work. · LOADS YOUTUBE ON CLICKWATCH ON YOUTUBE ↗

03 · THE PROBLEM

Support triage decides which queue, owner and priority every ticket gets, and mis-routing has asymmetric cost: an outage misfiled to billing compounds in a way the reverse does not. Automating that decision is only acceptable if every risky path fails toward a human.

04 · THE SYSTEM

My final-year project operates a real Zendesk workspace through a Playwright-driven browser. It reads each ticket, scores it against seven intent profiles with a transparent confidence formula, and consults a structured-output LLM only when one of seven ambiguity triggers fires. Anything below the automation threshold waits in a human approval console with a per-reason explanation. I am the sole author of the system, the evaluation and the 67-page dissertation.

Component diagram: polling tracker triggers the workflow orchestrator, which coordinates persistence, the hybrid classifier with policy engine, and the Zendesk browser adapter — The one primary figure: deterministic orchestration around the classifier and policy engine (dissertation Fig. 1).

Ticket state machine with re-entry into pending approval when write-back verification fails — Secondary: the ticket state machine. A failed write-back re-enters approval instead of counting as triaged (Fig. 5).OPEN FULL SIZE +

Operations dashboard with polling toggle, live activity and approval queues — Operator console: live activity and both queues (Fig. 6).

Ticket review screen with routing decision, confidence breakdown and editable reply — Human approval: per-reason explanation, editable before applying (Fig. 7).

05 · KEY DECISIONS

01
Deterministic first
The failure modes of a pure LLM router are hard to audit, so the LLM is consulted only when one of seven explicit ambiguity triggers fires.
02
One shared contract
A 36-line route-critical contract is shared by the classifier and the policy engine, so safety logic cannot drift between stages.
03
Verified write-back
Every write into Zendesk is re-read before a ticket counts as triaged, and low-confidence work waits in a human approval console with a per-reason explanation.

06 · VERIFICATION AND OUTCOME

Measured on the eight-case labelled sandbox suite (n = 8) · stub-mode latency.
Route accuracy	100%
Urgent precision / recall	100% / 100%
Execution reliability	100%
Median latency	0.139s

75% of cases were routed to human review. That is a safety decision, not a failure: six of the eight cases were built to require it.

Formula: conf = 0.40·intent + 0.20·urgency + 0.30·routing + 0.10·execution
Thresholds: Auto-apply ≥ 0.86 · consequential ≥ 0.91 · LLM review < 0.72 · high-impact < 0.82
Degraded mode: If the LLM is unavailable, confidence caps at 0.74, below every auto-apply threshold, so every fallback ticket waits for a human.
Latency: Heuristic path 20 to 35ms · LLM round-trip 700 to 1,500ms · browser write + verify 2 to 8s
Scale: 5,561 lines of Python · 47 tests across six modules · 36-line shared routing contract

CONSTRAINTS, EVALUATION & LIMITS

The evaluation proves the safety gates refuse correctly, but n = 8 cannot prove generalisation.
Latency was measured in stub mode. Live LLM round-trips add 700 to 1,500ms per consultation.
Sandbox Zendesk workspace, not production traffic, volume or adversarial input.
English-language tickets only.

WHAT I WOULD IMPROVE NEXT

What carried forward: verification infrastructure (the write-back re-read, the shared contract, the stubbed behavioural harness) was worth more than model sophistication. The system is trustworthy because every risky path fails toward a human.

01A labelled evaluation corpus at realistic scale.
02Calibration validation for the confidence formula.
03Live-latency benchmarks to replace the stub-mode numbers.
04A broader intent set beyond the seven profiles.