Token-level confidence

Most chat models can tell you a pump is failing. InertialAI-0.1 also tells you how sure it is — not by claiming a confidence in prose, but through the probability of the answer tokens themselves. On live graded runs of held-out tasks, right answers average 0.97 token confidence and wrong ones 0.71, so a single threshold at 0.90 keeps 92% of the right answers while rejecting more than half of the wrong ones. That turns a chat completion into a decision-grade signal for alerting, escalation, and human-in-the-loop review of real-world data.

How it works

Add two OpenAI-standard fields to any request:

{ "logprobs": true, "top_logprobs": 5 }

Every generated token then carries its log-probability, plus up to 5 alternative tokens with theirs, in the standard OpenAI logprobs response shape. There is no surcharge for logprobs.

exp(logprob) of the answer token is the model's probability for that answer. For a one-word classification, that single number is your confidence score.

The pattern: classify, then threshold

Constrain the answer to a known label set, then read the first answer token's probability:

import math
from openai import OpenAI

client = OpenAI(base_url="https://inertialai.com/api/v1", api_key="iai_...")

resp = client.chat.completions.create(
    model="inertialai-0.1",
    temperature=0,
    max_tokens=2,
    logprobs=True,
    top_logprobs=5,
    messages=[
        {
            "role": "user",
            "content": (
                "Vibration RMS (mm/s): 2.1, 2.0, 2.2, 4.8, 5.1, 5.0.\n"
                "Answer with exactly one word: normal or anomaly."
            ),
        }
    ],
)

tok = resp.choices[0].logprobs.content[0]
label = tok.token.strip()
confidence = math.exp(tok.logprob)

if confidence >= 0.90:
    page_oncall(label)          # act automatically
elif confidence >= 0.70:
    queue_for_review(label)     # human-in-the-loop
else:
    log_only(label)             # too uncertain to act on

The top_logprobs alternatives give you the full distribution over your label set — useful for ROC curves, picking thresholds on your own holdout data, and monitoring drift in production.

Why this works

Raw logprobs from general-purpose LLMs are notoriously overconfident. InertialAI-0.1 is trained against a proper scoring rule on physical data, so its answer-token probabilities separate right answers from wrong ones — which is what makes the thresholding pattern above safe to ship:

• Alert routing — page on high-confidence anomalies, queue the rest.
• Cost control — auto-accept high-confidence extractions; only pay humans to review the uncertain tail.
• Auditability — every automated decision has an attached probability you can log and defend.

Tips

• Set temperature: 0 for classification — you want the mode of the distribution, and its probability, not a sample.
• Keep label sets to single tokens where possible (normal / anomaly rather than multi-word phrases); if a label spans several tokens, use the first token's probability or sum the sequence logprobs.
• Sum logprobs across tokens (or average per-token) to score longer extractions such as numeric readings.
• Calibration holds best on in-domain, real-world data; for your own domain's labels, a Chronicle finetune tightens it further and is evaluated against the base automatically.