Provenance Guard
Company:
CodePath
Description
Provenance Guard is a content attribution service for a creative-writing platform. It scores submitted text on the likelihood of AI authorship using two independent signals, a semantic judgment from a language model and a set of stylometric heuristics computed in pure Python, then combines them into a single confidence score and returns one of three labels.
Every decision is logged with both raw signal scores so a reviewer can see the working, and creators can appeal a classification without the original decision being overwritten. The system is designed around one principle, which is that falsely labelling a person's writing as machine-generated is worse than missing some AI, so every threshold is tuned to be cautious about accusing.
Contributions
I specified and directed the build of Provenance Guard, a Flask API in Python. I wrote the design document first, pinning down the scoring weights, threshold bands, and exact label text before any code existed, then used that spec to hold the implementation to what I had designed.
I chose to run two independent signals rather than one. The language model reads for tone, hedging, and evenness of argument, while the stylometric heuristics measure sentence-length burstiness, type-token ratio, and punctuation variety. Because the stylometric signal is the one prone to flagging formal human prose, I weighted the combined score toward the language model at 0.65 to 0.35.
The threshold bands are deliberately asymmetric. The AI zone starts high at 0.75 and the uncertain band runs wide from 0.45, so borderline content resolves to uncertain rather than to an accusation. I added a disagreement guard that clamps the result into the uncertain band when the two signals differ by more than 0.50, since a split decision should never be presented confidently. Text under three sentences returns a neutral score, as stylometrics is statistically meaningless at that length.
Every decision is written to a SQLite audit log with both raw signal scores, so a reviewer can reconstruct how a label was reached. Creators can appeal through a separate endpoint that flips the status to under review while preserving the original attribution, confidence, and both scores. Nothing is overwritten and no automated re-classification happens, because an appeals record is only useful if the original decision survives it.
Calibration did not work on the first pass. A casual human review was scoring 0.53 on stylometrics against 0.10 from the language model, and the 0.40 disagreement threshold I had planned pushed it to uncertain. The first fix attempted was to reweight the stylometric sub-scores, but that dropped a clearly AI input to uncertain as well, so I reverted it and widened the guard to 0.50 instead, then re-ran all four test inputs to confirm both cases passed.
Implementation was done with an AI coding assistant working from my specification. The architecture, scoring model, thresholds, and the calibration fix above were my decisions, and I reviewed the generated code against the spec at each step.