Agent workflows · 2026-09-25
Designing human approval with a Jev confidence threshold
How the confidence value in Jev's Choice and Score answers is computed, what it says and does not say, and how to separate automation, human approval and stopping according to the risk of an action.
Short answer
Jev's Choice and Score answers carry a confidence value alongside a probability distribution. confidence summarises, between 0 and 1, how concentrated that distribution is on a single outcome: 1 when all the probability sits on one option, close to 0 when it is spread evenly. Noul answers have no confidence; there the probability of yes itself is read.
If only the most likely option will be used, no threshold is needed; reading the choice field is enough. A threshold is useful where a decision is split between applying it automatically and handing it to a person, and where to draw it depends on the risk of the action.
How confidence is computed
TypeSafe defines confidence as a statistic computed from the probability distribution the answer already contains. For a question with three options, the formula is three times the highest probability, minus one, divided by two. If the probabilities are 0.90 for billing, 0.06 for technical and 0.04 for sales, confidence is 0.85. If they are 0.50 for technical, 0.40 for billing and 0.10 for sales, the model still chooses technical, but confidence drops to 0.25.
Read on its own, the choice field makes the two cases look the same. confidence makes the difference visible. TypeSafe also notes that this definition is a convenience: the full probabilities are in the answer, so if a different measure is needed you can compute your own.
What confidence does not say
confidence shows how sure the model is, not the probability that the decision is correct. Decisions made at 0.85 confidence are not 85 percent correct by definition; that rate depends on how the question is written, on the option descriptions and on the kind of messages coming in. Low confidence is not an error either; it shows the message is ambiguous and that more information or a person's view may be needed.
So the meaning of a threshold only appears once it is measured on your own data. The same threshold can make almost no mistakes on one question and frequent mistakes on another whose descriptions are vaguer.
Choosing a threshold by the risk of the action
TypeSafe suggests a three-zone arrangement: act automatically at high confidence, proceed with caution in the middle, and do not act at low confidence. Where the boundaries go depends on the cost of a wrong decision, and different actions in the same system can be gated at different levels.
Adding a tag to a ticket is reversible; a wrong tag is fixed with one click. Starting a refund is hard to reverse. The same confidence value can be enough for the first and not enough for the second. The example below gives each action its own thresholds and handles Noul answers separately; the values are examples.
Per-action thresholds and a two-sided band for Noul
# Thresholds are examples; set each action's values from your own data.
THRESHOLDS = {
"add_tag": {"auto": 0.60, "review": 0.30}, # reversible
"start_refund": {"auto": 0.90, "review": 0.60}, # hard to reverse
}
NOUL_BANDS = {"yes": 0.90, "no": 0.10}
def decide(action: str, answer: dict) -> str:
if answer["type"] == "noul":
p = answer["noul"]
if p >= NOUL_BANDS["yes"]:
return "yes"
if p <= NOUL_BANDS["no"]:
return "no"
return "human_approval"
c = answer["confidence"]
t = THRESHOLDS[action]
if c >= t["auto"]:
return "automatic"
if c >= t["review"]:
return "human_approval"
return "stop"A threshold for Noul
Because Noul has no confidence, the threshold is placed directly on the probability of yes. A value near 0.5 means the model is unsure, so a two-sided band makes more sense than a single threshold: yes if high enough, no if low enough, human approval in between. A single 0.5 threshold would apply even the most uncertain decisions automatically.
The human approval queue
A decision that falls to human approval should not reach the reviewer empty-handed. When the model's proposed option, the probability distribution and the evaluated text are shown together, the reviewer usually spends a few seconds confirming or changing the proposal. Showing the second option in the distribution as well tells them which two options the model was torn between.
When the reviewer's final decision is recorded, the queue also becomes a measuring tool: the labelled examples needed to tune the threshold accumulate on their own. A queue that keeps growing shows that the threshold is too strict or the question is written too vaguely.
Tuning the threshold on your own data
The way to tune a threshold is to take a sample of past messages, have a person mark the correct decisions, and look at two numbers for each candidate threshold: how many decisions are applied automatically, and how many of those automatic decisions are wrong. The lowest threshold at which the error cost is acceptable is usually the right operating point. TypeSafe also recommends starting conservatively and testing on your own data.
When a question's text or its option descriptions change, the distribution changes too and the old threshold may no longer hold. Keeping thresholds and question texts in one file in the code makes it easy to see which one changed and to repeat the measurement.
Frequently asked questions
Is 0.85 the right threshold for everyone?
No. Neither TypeSafe nor LLMTR publishes a default threshold. The right value depends on the risk of the action and the error rate measured on your own data.
Does low confidence mean the decision is wrong?
No. It shows that probability is spread across options, in other words that the message is ambiguous. Such decisions are good candidates for human approval.
How is a threshold set for Noul?
Noul returns no confidence; the threshold goes on the probability of yes. Setting limits for the high and low ends and sending values in between to human approval keeps uncertain decisions from being applied automatically.
Does a threshold still hold after the question text changes?
It may not. The instructions and option descriptions affect the probability distribution; when the text changes, the threshold needs to be measured again.
What does confidence show in a Score answer?
It shows how concentrated the probability is across the levels. Low confidence means the model is split between more than one level and, as with Choice, it is a signal for human approval.