Try the thresholds
Reliability
Each bar is the share of true yes among commands whose Jev probability fell in that tenth; the line is the mean probability. A bar at its line means calibrated.
Act
Review
Escalate
120 synthetic shell commands, lowest certainty first. A highlighted line is a wrong call. A line that just changed band flashes.
What Jev is
Jev is a hosted decision model from TypeSafe that answers a fixed question with a calibrated number and no prose. Ask it whether a command is destructive and it returns a probability of yes.
The number is what makes it programmable. Your code owns the policy: act on it above one cut, send it to a person between two, escalate below. Changing a cut never re-runs the model, so you can tune thresholds per question on your own labeled data.
The band rule
An item acts at or above the act cut, goes to review at or above the review cut and below act, and escalates below review. What gets compared depends on the question type.
| Question type | Banded number |
|---|---|
noul (yes or no) | certainty, max(p, 1 - p), so a confident no acts as readily as a confident yes |
choice (pick one) | the returned confidence |
score (1 to 5) | the returned confidence |
For a yes/no question certainty is never below 0.5, so a review cut of 0.50 means nothing escalates. The sliders stop at 0.51 for that reason.
The limits
- Every number on this page comes from 120 synthetic commands labeled by the rules that generated them (n=120, fit run 2026-10-18, model jev-1.13.0). Thresholds do not transfer; refit on your data.
- Jev gives numbers, not reasons, so the lab shows where it is wrong, not why.
- Calibration is a population property: one answer at 0.90 is not 90% likely to be right.
- Real traffic warning: a gate question scored macro F1 0.94 on a synthetic bench (n=60, 2026-09-22) and found zero true positives at 0.50 on real commands (n=54 hand-labeled, 2026-09-22). The model id was not recorded for either run.