The problem
A classifier can return a high-probability label and still be wrong. In a high-stakes workflow, a single confident mistake can be more expensive than an obvious failure.
The system needed a way to separate “the model is sure” from “the model is probably right”, apply business risk rules, and route the uncertain cases to a human.
What I built
- A FastAPI service that combines a DistilBERT classifier with a rule-based risk engine.
- Confidence and risk scoring that decides between auto-approval, review, and rejection.
- A human review path with an audit trail for decisions that need context or accountability.
- An explainable response shape that shows the signals behind a recommendation instead of only a label.
System shape
Decisions and tradeoffs
Confidence is not correctness
The model output is one input to the decision, not the decision itself.
Escalation is a product feature
Low-confidence cases are routed to a human instead of being forced through the same path as easy cases.
Make the decision legible
A label without reasons is hard to audit. The response exposes the signals and the threshold that produced it.
Outcome
A 15-star AI safety project and an AI Pick Japan feature, built around a simple principle: the system should know when it does not know.
The most useful AI system is sometimes the one that says “I’m not sure — ask a human.”