Reinforcement Learning for LLM Post-Training · Module 25
The Name RLCD: What TypeSafe AI Claims, What Is Public, and the OpenJev Reconstruction
TypeSafe AI says it trains its Jev model with "Reinforcement Learning for Calibrated Decisions" and has published no paper, code, reward function, algorithm or dataset for it. This module is a ledger: what the company, its founder and the press say, what independent teams measured on the API, what no source discloses, and what one independent reconstruction shows about a method that carries the same name. Afterwards you can say, for any statement about RLCD, who made it and with what evidence, and which public calibration literature it sits closest to.
1. Which RLCD, and what exists
1.1 The statement and its date
On 15 Sep 2026 Diogo Almeida, founder of TypeSafe AI, announced the company's first "System One Model", Jev, in early access1. One sentence in that post names the training method: "We built a new stack entirely focused on automation: with a new model architecture, parallel sampler for maximum efficiency, and training method we call Reinforcement Learning for Calibrated Decisions (RLCD)"1 company. A comparison table in the same post says what each regime optimizes: RLHF and RLVR optimize "human preference" and "verifiable rewards"; RLCD optimizes "calibrated decisions: answers with epistemically honest probabilities on System One tasks"1. The documentation's AI primer places RLCD as a third post-training route ("Pretrained language models have been adapted in two major ways. TypeSafe adds a third") and defines its output: it "trains TypeSafe to return decisions and calibrated probabilities instead of generated text"2 company.
TechCrunch's article of 18 Sep 2026 says the company released the model "this week", which fits the post's date4 press. Jev 1.14.0 followed on 14 Oct 20265. Every independent measurement in Section 5 used an earlier build (jev-1.13.0 or the alias that pointed to it).
The same post lists FAQ entries whose answers sit in collapsed panels. "Why was a new training algorithm needed?" is answered by changing the question: every lab optimizes RLHF's task, "produce the text that a human rater prefers", which is "the wrong task for automation"; RLVR "is great for tasks with simple programmatic verification, but most real-world judgement tasks don't fit into that shape"1. The answer names an objective, not an algorithm. "Where does our training data come from?" is answered "We make all the data ourselves", followed by "We do some pretty sophisticated stuff, but if you want to find out more, we'd have to hire you"1 company.
Publication status is stated by two parties. Latent Space's show notes call RLCD "a novel, unpublished technique"3. In the interview the host says "You've never actually published a paper or anything like that on it. No, right?" and the founder answers "No, not yet"3 interview (transcript timestamps 00:24:59 to 00:25:05, checked against the full transcript text). The docs index lists the AI primer as the only page about training6. Three independent preprints say the same thing in their own words (Section 4).
1.2 The name in third-party use
Module 8 (Table 9) already separates the 2023 method that shares the letters (Yang et al., Reinforcement Learning from Contrastive Distillation) from TypeSafe's, and places RLCR and RLHV beside them; that map is not repeated here. What Module 8 does not show is how the 2026 name is now used by people who are not TypeSafe. Table 1 lists each use and what it implies about the method.
| Source | Type | What it calls RLCD | Training disclosed? |
|---|---|---|---|
| TechCrunch, 18 Sep 20264 | press | "reinforcement learning from calibrated decisions": preposition "from", acronym not used; reports the founder's term | no |
| Gao and Wang, arXiv 2609.388507 | independent | its own method: a strictly proper score on the answer distribution after a sampled rationale | yes, its own (Section 7) |
| Guo et al., arXiv 2609.294298 | independent | "a model trained with reinforcement learning for calibrated decisions", meaning Jev; footnote separates it from Yang et al. | no; takes TypeSafe's statement as given |
| dos Santos (arXiv metadata: dos Santos Barbosa), arXiv 2609.289409 | independent | "trains against strictly proper scoring rules", Brier as the canonical example; calls Jev's methodology "undisclosed" | no |
| Laya README (github.com/NandhaKishorM/laya)10 | independent, project page | describes its own open model as "trained with reinforcement learning against strictly proper scoring rules (RLCD)" | only as that one-line description in the README text read |
| Yang et al., ICLR 202411 | paper | a different method: preference pairs from a positive and a negative prompt, then a preference model and PPO | yes; unrelated to TypeSafe |
The label has already drifted: for the Laya project and dos Santos it names a family, training against strictly proper scoring rules; for TypeSafe it names a specific regime whose content is withheld. None of these papers is evidence about TypeSafe's training: they build their own method (OpenJev), test the released API (Just Ask Jev), or take the label as given (dos Santos).
2. The claims ledger
2.1 Every claim, who made it, and whether anyone checked
Table 2 lists the claims about RLCD and about the model it supposedly produced. "Type" uses four labels: company (TypeSafe's own pages), interview (the founder speaking), press (a reporter's text, including quoted users) and independent (a party with no stated TypeSafe affiliation measuring or reconstructing). The last column says what an outside party has tested. Claims about training cannot be tested at all, because no mechanism is public (Section 4).
| Claim | Said by, where | Type | Outside check |
|---|---|---|---|
| Method and training | |||
| RLCD is a third post-training route beside RLHF and RLVR; it trains the model to return decisions and calibrated probabilities instead of text | docs primer; launch post21 | company | None possible: no mechanism disclosed |
| RLHF can reward sycophancy and confident hallucination and causes "mode dropping" | primer; founder at 00:06:0323 | company, interview | Not tested on Jev (Section 6 gives the public literature) |
| All training data is synthetic and made in-house; no customer data | launch FAQ; founder at 00:22:04 ("Yep"); TechCrunch; Models page1345 | company, interview, press | None; generator, size and labels undisclosed |
| Jev is "neither small nor an LLM" | launch FAQ1 | company | Not corroborated: the press reports an unconfirmed suspicion of an open-weight base |
| RLCD has not been published | founder at 00:25:05; show notes3 | interview | Consistent with every search; three preprints call the method undisclosed |
| Output contract | |||
| No text is generated; answers are typed, with probabilities; Choice and Score also return a confidence | docs Introduction, Confidence1213 | company | Yes: three outside groups consume these fields |
| Jev "can't hallucinate"; "never makes type errors" | launch post; repeated by TechCrunch14 | company, press | True by construction for schema validity (the company calls its figure "not empirical"); silent on wrong decisions |
| Questions are evaluated "in parallel and in isolation" | docs Introduction; cookbook1214 | company | Not tested by the outside work read |
| Calibration, speed, cost, accuracy | |||
| Calibrated: "higher confidence means higher accuracy"; calibration describes groups, not single answers | launch table; primer; System One page1215 | company | Mixed: good ranking, but median per-benchmark ECE 0.168 against a null of 0.0748 |
| 70 to 500 ms end to end against 3 to 329 s for frontier LLMs; "40x-200x faster" | launch table1 | company | Latency confirmed (p50 264 to 316 ms in DMB16; 0.15 s median in the judge paper17, which itself claims about 13x faster than GPT-6 at the median). The 40x-200x ratio is not borne out: my arithmetic on DMB's printed ranges gives 2.6x to 7.6x in its pilot and 1.1x to 21x in its historical table (Section 5.2); the company's own cookbook prints 7.2x to 113.7x against its own 114 ms baseline22 |
| $0.042 per million input tokens; output free | launch table; Models page15 | company | Used as list price by two outside studies, not invoice-verified; the company "can't prove it isn't subsidized"1716 |
| 193.6x faster and 444.6x cheaper | launch post, workflow evals1 | company | None; the reference is not ground truth (Section 2.3) |
| Similar intelligence to existing LLMs on System One tasks | launch post1 | company | Mixed: within three points of GPT-6 where a verdict can be read off the text, 7 to 28 points behind where it must be derived17 |
| A Vercel classifier swap gave results "five to 18 times more quickly and with greater accuracy" | Vercel engineer, in TechCrunch4 | press (user report) | None |
| No standard benchmark tables; only dated one-off internal evals | launch FAQ; "Lies, Damned Lies, and Benchmarks"; founder at 01:15:191183 | company, interview | Moot: outside groups benchmarked the API within two weeks |
The founder is not an outsider to the field he criticizes: Almeida is a starred author of the InstructGPT paper19, and the primer says he "co-invented" RLHF2. That makes the critique informed. It is still a position, and Table 2 shows that the replacement's training has no outside test.
2.2 What the founder says RLCD is: a task, not an algorithm
The interview contains the clearest statement of what the name denotes. The founder first says what he means by RLHF: "the task of instruction following", and "It's not about the PPO. That part doesn't matter. It's about ... setting a North Star"3 (00:27:18). Then he applies the same reading to RLCD: "for us, RLCD is this new task", "here's another North Star", in the way that DPO and its descendants "also do RLHF" without using PPO3 (00:27:34) interview. The line "It's not about the PPO" is therefore a statement about what RLHF means. It is not a statement about the algorithm behind RLCD, and it should not be quoted as one.
The one-line version he gives later: "RLHF is please humans ... RLVR is optimize benchmarks ... And RLCD is make it reliable for, programmatic use"3 (01:20:21). The company's essay of 10 Sep 2026 supplies the ranking behind it: "the bitterest lesson in ML is that doing the right task > data > compute > algorithms"20 company. Read together with the launch FAQ (Section 1.1), the company's own answer to "Why was a new training algorithm needed?" is that the algorithm is not the point. This matters for the disagreement below: if "RLCD" names an objective and an output contract, then asking whether it is a new algorithm asks the wrong question of the company's own claim, and the question that remains is whether the objective, data and interface are new.
2.3 The company's own caveats
The company's pages hedge more than the headline numbers suggest.
- The 193.6x and 444.6x figures come from four "workflow evals" the company designed. Accuracy is scored not against ground truth but against "the predictions of the largest, smartest, and most expensive external models": "the average of GPT-6 Astra and Fable 5.1", which, the post says, "biases answers towards OpenAI and Anthropic's models". The workflows "were made by individuals on our model capabilities team, so some bias could exist", and the company expects the speedups to be "on the higher end of real world gains"1.
- The zero-hallucination figure: "Our number is not empirical. Schema matching is guaranteed, thus we can confidently add 0% into the plots"1.
- Known failure modes. The page for
jev-1.13(reviewed 2026-10-02) lists literal reading, weak arithmetic, counting, dates, indirection, accuracy falling as irrelevant state grows, and adversarial content. It also says a Choice's option order "can affect the answer, and jev-1.13 leans toward the option that comes first", and that "jev-1.13's score levels are weak in numerical calibration"21. Presentation-order sensitivity in the probabilities is a calibration problem in its own right. - Repeatability is not calibration. The self-consistency cookbook says its percentages "measure repeatability only" and that its 0.60 threshold "is an illustrative application policy, not a calibrated guarantee"22.
3. The output contract and the confidence arithmetic
The one fully specified part of RLCD is the interface it is said to produce: public, checkable with an API key, and testable with arithmetic.
3.1 Typed answers instead of a string
A call carries a state (text or JSON) and typed questions; the response carries one typed answer per question. The docs define three primitives (Table 3), and the launch post adds that a Choice supports up to 255 options, with a two-stage score-then-choose system above that121 company. The Models page states the processing model: Jev "ingests the state once and evaluates every question against it in parallel", under limits of 64k tokens per request and 32k for the state plus any one question5.
| Primitive | Question | Returns | Confidence statistic |
|---|---|---|---|
| Choice | pick one of \(n\) unordered options | choice, probabilities, confidence | \(\dfrac{p_{\max}-1/n}{1-1/n}\) |
| Score | rate against \(n\) ordered levels | score, probabilities, confidence | \(\max\!\big(0,\,1-\sum_i p_i|i-m|/\mathrm{MAD}_{\text{unif}}\big)\) |
| Noul | is this statement true? | one probability in [0, 1], no separate confidence | offered as a convenience: \(|2p-1|\) |
3.2 What the confidence number does and does not carry
The docs state that confidence "is a statistic computed from the probability distribution the answer already gives you"13. It is a deterministic function of the returned vector, so it adds no information that the probabilities lack. For a Choice it depends on the top probability alone, which the page demonstrates: (0.6, 0.3, 0.1) and (0.6, 0.2, 0.2) both give 0.413. The Score statistic does see the shape: with three levels, (0, 0.5, 0.5) gives 0.25 and (0.5, 0, 0.5) gives 0, because probability on a distant level counts more (checked against the formula; the Choice formula would give both 0.25)13.
Two consequences are easy to miss. First, calibration is a claim about the probabilities, not about the confidence field: the primer says outcomes given probability 0.8 "should occur about 80% of the time" and that "these rates describe groups of predictions, not a guarantee about any single answer"2. Second, a confidence value is not an accuracy. Inverting the Choice formula gives \(p_{\max}=\tfrac1n+c\,(1-\tfrac1n)\), so if the probabilities are calibrated, the accuracy implied by a given confidence depends on \(n\) (Table 4).
| Options \(n\) | \(c=0.4\) implies accuracy | \(c=0.9\) implies accuracy |
|---|---|---|
| 2 | 0.700 | 0.950 |
| 3 | 0.600 | 0.933 |
| 5 | 0.520 | 0.920 |
| 10 | 0.460 | 0.910 |
3.3 What the contract says about training, and what it does not
Three company statements bear on the method. The System One page says these models "are trained for calibrated decisions: their probabilities are optimized against outcomes to reflect uncertainty" and "do not write replies, produce code, or generate explanations of their reasoning"15 company. The launch post says Jev "outputs all probabilities in parallel instead of autoregressively generating by token"1. The Models page says the weights are not tuned per customer: Jev "is trained with RLCD to return calibrated decisions, and the same weights serve every account"5. None says what "outcomes" are, what is sampled, or what is rewarded.
Two architecture claims come from elsewhere. TechCrunch calls Jev a "transformer-based model" that "is not a large language model"4; dos Santos says a System One model "is non-autoregressive" and "evaluates all questions in a single forward pass"9. The launch post supports only the narrow version (outputs are not generated token by token); the stronger attribution, a non-autoregressive model that evaluates everything in a single forward pass, is the preprint's and not a TypeSafe statement, and "transformer-based" is the reporter's phrase.
4. What no source discloses
No source read gives TypeSafe's reward function, loss, scoring rule, algorithm, dataset, base model or training run. That is a finding about the sources; Table 5 shows where each component stands. The search covered the launch post and its four sibling posts (none of the siblings contains "RLCD"), the docs index and the pages cited here, the full Latent Space transcript and the TechCrunch article. It did not cover two embedded videos on the company blog (a 31 Mar 2026 video, "Diogo Almeida - Founders You Should Know", and a 19 Jun 2026 talk, "AI: too good to be true, too bad to be useful"). The Latent Space notes point to a founder talk as the best explanation of why RLCD was needed3; if that talk describes a method, this module has not seen it.
| Component | Company pages | Interview | Press | Independent | Net status |
|---|---|---|---|---|---|
| Reward function, scoring rule | only "probabilities are optimized against outcomes" | – | – | "undisclosed"; "training recipe ... not public" | not disclosed |
| Loss | – | – | – | – | not disclosed |
| Algorithm | calls RLCD a "training method"; the "new training algorithm" FAQ is answered with a task | RLCD "is this new task", "another North Star" | – | "without disclosing a training algorithm" | not disclosed |
| Training data | "We make all the data ourselves" | synthetic ("Yep") | "trained exclusively on synthetic data" | "data are not public" | partly: synthetic; generator, labels, size unknown |
| Human labels or verifiers | – | – | – | – | not disclosed |
| Base model, size | "neither small nor an LLM" | – | architecture undisclosed; observers suspect an open-weight base | "size ... not public" | not disclosed |
| Architecture | "new model architecture, parallel sampler"; questions evaluated "in parallel" | – | "transformer-based" | "non-autoregressive", one forward pass (dos Santos) | named, not described |
| Calibration metric | population-level definition only | – | – | chosen by each outside study (ECE, Brier) | not disclosed |
| Calibration measurements | none on any page read | "it will get many things wrong" | – | three outside studies (Section 5) | independent only |
| Training compute, run length | – | – | – | – | not disclosed |
Three independent preprints say the same in their own words. The reconstruction paper: Jev "markets exactly this capability under the name reinforcement learning for calibrated decisions (RLCD), without disclosing a training algorithm", and the authors "claim nothing about Jev's internal algorithm"7 (§1, §7). The CMU judge paper: "its size, training recipe, and data are not public"17 (Limitations). dos Santos: "an undisclosed RLCD methodology"9 (§II-B). Asked directly what RLCD means, the founder gives the task framing of Section 2.2.
5. Outside evidence on the outputs
Training cannot be audited from outside, but the released model can. Within two weeks of launch, three independent efforts tested the API; their results bear on the output claims (typed answers, speed, calibration) and not at all on how the model was trained. Keeping those two questions apart is the point of this section.
5.1 The press account
TechCrunch's reporter spoke to the founder and three outsiders4 press. A Vercel engineer said Vercel swapped a safety classifier built on an OpenAI model for Jev and got "results five to 18 times more quickly and with greater accuracy". The CTO of Bryo AI found Gemini "slightly more accurate, but 10 to 20 times more expensive" on business-email classification, and valued that Jev "hands back a real probability". Armin Ronacher, CTO of Earendil, gave the sharpest reading of "cannot hallucinate": "At the end of the day, it delegates the hallucination problem a little bit to the user," he explained, and "The user has to say, okay, if this only comes back with 50% probability, maybe this is a coin toss, and I disregard it." The article also reports that the founder is "tight-lipped about the model's architecture, which outside observers suspect is built on top of an open-weight LLM": an unconfirmed suspicion, with no named source.
Ronacher's point is the practical content of "calibrated": the model outputs a number and software must decide what to do with it. The three outside studies test whether the number can be trusted.
5.2 Three outside measurements
| Source | What was run | Headline results | Limits the authors state |
|---|---|---|---|
| DMB, community benchmark repository (nibzard)16 | jev-latest on full test sets, 29 Sep 2026: Banking77, CLINC150 (1,000 out-of-scope), NLU++; 40,226 valid decisions. A 26 Sep pilot against six LLMs | Banking77 79.2%, CLINC150 88.6%, NLU++ micro intent F1 48.3% (complete-message accuracy 4.3%); p50 267 to 316 ms. A threshold frozen at 5% validation error gave 3.68% test error on Banking77 (coverage 51.2%) and 6.65% on CLINC150 (coverage 88.3%); NLU++ had none. Pilot: Jev 80.5% on banking, Gemini 3.8 Flash 88.3% | "A validation error target does not guarantee test error"; confidence is "a provider-defined score, not a calibrated probability of correctness" |
| Guo et al., "Just Ask Jev", arXiv 2609.29429 (24 Sep 2026)8 | Jev as a detector of ten alignment-failure types: 44 benchmarks, 7,193 instances | Median AUROC 0.886 over 31 benchmarks; pooled ECE 0.047; median per-benchmark ECE 0.168 against a null of 0.074, 24 of 31 above the null's 95th percentile; median F1 0.706 at threshold 0.5, 0.822 with a cross-validated threshold, 0.793 with 10 labels | One model version, English, 2B to 7B targets (§6) |
| Li et al., "JEV-as-a-Judge", arXiv 2609.26550 v5 (CMU)17 | Jev against sixteen judges, blinded human adjudication; a confidence-gated cascade to GPT-6 | Within three points of GPT-6 where a verdict can be read off the text, at 0.36% of its fee and 0.15 s median latency (its own claim: about 13 times faster than GPT-6, p95 0.24 s against 3.86 s); 7 to 28 points behind where it must be derived; cascade 0.9 points above GPT-6 on 1,610 held-out pairs at 41% of its fee; temperature scaling helps on one task and hurts on two | Training recipe and data not public; benchmark items may have been seen in training (Limitations) |
Where the speed ratios come from. None of the ratios below is printed by DMB or the judge paper (which reports only Jev's own 0.15 s median); they are my arithmetic on DMB's printed p50 ranges16, each dividing one model's slowest per-suite median by Jev's fastest or its fastest by Jev's slowest. DMB's 26 Sep pilot used the same frozen items for every model, with Jev at 0.26 to 0.28 s. The five LLMs that completed ran at 0.74 to 1.98 s, so the ratio runs from 0.74 / 0.28 = 2.6x (Gemini 3.5 Flash-Lite) to 1.98 / 0.26 = 7.6x (Gemini 3.8 Flash). Gemini 3.1 Pro (3.07 to 3.12 s) gives 11x to 12x, but DMB says its rerun included pacing and is not directly comparable. DMB's historical table, built from mixed protocol versions, has Jev at 264 to 276 ms. Cerebras gpt-oss-120b at 303 to 346 ms gives 303 / 276 = 1.1x to 346 / 264 = 1.3x, and glm-5.3 at 2.4 to 5.6 s gives 2.4 / 0.276 = 8.7x to 5.6 / 0.264 = 21x. The 21x pairs the slowest glm-5.3 suite median with the fastest Jev one, so it is an upper envelope, not an observed single-suite ratio. The company's own cookbook prints larger multipliers, 7.2x (gpt-5.4-mini single-pick, 826 ms) to 113.7x (gpt-5.5 with reasoning, 12,978 ms), against a 114 ms mean for Jev in a run the company designed22. That is company data, not an outside measurement, and its 114 ms for Jev is 2.3x to 2.8x below DMB's p50 of 264 to 316 ms (264 / 114 and 316 / 114); the setups differ, since the cookbook draws Jev's samples sequentially and DMB ran four workers per run. DMB's own conclusion for the historical comparison is that its measurements "do not support a general 40 to 200 times speed claim"16.
The judge paper's own speed claims. These are the authors' claims, not my arithmetic. In its "Cost and latency" paragraph (§5, with Figure 3 and Table 1) the paper reports, on a frozen timing panel of 120 judgments per hosted configuration run one model group at a time with eight workers (Appendix, "Timing panel"), that "JEV's median latency is 0.15 seconds and its fee $0.044 per 1,000 judgments, against 0.55 seconds and $0.390 for GPT-4.1 mini and 1.89 seconds and $12.182 for GPT-6", and that this is "about 13 times faster than GPT-6". It adds: "Tails differ even more: JEV's p95 latency is 0.24 seconds, against 3.86 for GPT-6 and up to 23.2 for Qwen3.6, whose tail includes rate-limit retries"17. The named comparator for the 3.86 s p95 is GPT-6, and hosted reasoning models ran at low effort except Qwen3.6, which ran at its default (Table 1 caption). Checking the paper's own inputs, 1.89 / 0.15 = 12.6 (the "about 13") and 3.86 / 0.24 = 16.1 at the p95. How these relate to the DMB arithmetic above: they are a different comparison, so they neither confirm nor contradict it. The judge paper compares Jev with one reasoning model on pairwise judging; DMB compares it with fast, mostly non-reasoning models on classification. The paper's 12.6x lies inside the 1.1x to 21x envelope I derived from DMB's historical table and above the 2.6x to 7.6x of DMB's pilot, but a number falling inside a range built from other comparators is not corroboration. Jev's own latency also differs by study: 114 ms mean in the company cookbook22, 0.15 s median in the judge paper, and 264 to 316 ms p50 in DMB16. The sources do not explain the spread, but all three put Jev well under a second. Against reasoning models the ratios are about 13x at the median and 16x at the p95 in the judge paper (GPT-6 at low effort) and 90.9x to 113.7x in the company cookbook (claude-opus-4-8 and gpt-5.5 with reasoning, 114 ms baseline)22. The only outside figure above 40x is derived from the paper's Qwen3.6 tail, 23.2 / 0.24 = 97x, and the paper attributes that tail to rate-limit retries. The launch post states 40x to 200x for "the same levels of frontier intelligence"1, and no clean outside comparison in these sources reproduces it.
5.3 What the numbers say about calibration
The three studies agree on three points, and each is more informative than a headline ECE.
- The ranking is good. Just Ask Jev finds a median AUROC of 0.886 with one generic question8 (abstract; §4.4 gives 0.905 within benchmark files); the judge paper finds error-detection AUROC close to GPT-6's on two of three tasks17 (§7.2). A confidence gate works because high-probability answers are right more often.
- The absolute level is not portable across tasks. Pooled over benchmarks the reliability curve is "close to the diagonal (ECE 0.047)", yet the median per-benchmark ECE is 0.168 against 0.074 under perfect calibration, because Jev's mean probability misses each benchmark's base rate by a median of 0.1258 (§4.4). A pooled curve can sit on the diagonal while its parts lie on both sides of it, and the company's definition is relative to a population ("groups of predictions")2.
- Thresholds need local labels. Just Ask Jev recommends a threshold fitted on ten labeled items8. The judge paper finds that temperature scaling fitted on a pilot set "helps on one task and hurts on two", with fitted temperatures of 0.65, 2.15 and 4.4517 (§7.2). DMB's frozen 5%-error threshold held on Banking77 and failed on CLINC15016.
None of this contradicts the company's population-level definition, and its docs say thresholds are an application decision1322. It does contradict reading the launch table's "higher confidence means higher accuracy" as a property to rely on without local validation: the ranking property is well supported, and the absolute-probability property is not supported across tasks.
5.4 Where the figure "ECE 0.246" comes from
dos Santos writes that "TypeSafe reports ECE of 0.246 for Jev on their own evaluation set", citing TypeSafe's documentation, and lists it in Table IX against 0.081 for the open Laya model9 (§XII-B). No TypeSafe page read carries that number: the attribution "TypeSafe reports ECE of 0.246" is unverified. The number appears elsewhere with a different provenance. The Laya README's comparison table lists "ECE 0.246" in its "Jev 1.13.0" column and says "Jev figures are third-party published, never measured here (no TypeSafe API access)"10; the Jev column of dos Santos's Table IX matches it on every row compared (0.727, 0.910, 0.870, 236 to 276 ms, 0.246). DMB says its reported "S5 expected calibration error (ECE) of 0.246 concerns the underdetermined half, whose reference answer was randomly planted"16, one deliberately underdetermined suite, not a general calibration test. So 0.246 is a community benchmark's number for that suite, relayed through another project's README into a preprint where it is attributed to the vendor. Do not cite it as TypeSafe's measured calibration.
6. The public machinery the claims resemble
Each company claim in Table 2 has a public counterpart in calibration-aware training. The counterparts are public and measured, and they are other people's methods. Table 7 lines them up as a resemblance, one claim at a time. Module 24 derives the Brier-augmented reward and its theorem (its Section 3), reads Behaviorally Calibrated RL in full (Section 6) and covers Kalai et al. (Section 7); the table below uses only what the papers state, and BCRL only at abstract level.
| TypeSafe says | Public counterpart | What the counterpart shows | Where it differs |
|---|---|---|---|
| RLHF "can also reward sycophancy and confident-sounding hallucinations"2 | Kalai et al.: models hallucinate "because the training and evaluation procedures reward guessing over acknowledging uncertainty" | Under 0-1 grading, "abstaining is strictly sub-optimal" and a best guess is optimal | The argument is about binary grading of answers, not about human-preference optimization |
| RLVR suits tasks with "simple programmatic verification" and yields "spikey / non-robust intelligence"1 | RLCR and BCRL: binary rewards "do not penalize guessing"; RLVR "encourag[es] guessing whenever correctness probability exceeds zero" | Independent support for the diagnosis that binary correctness rewards degrade calibration; RLCR measures it, including out of domain | These papers target RLVR specifically; TypeSafe's primer aims its calibration critique at RLHF |
| RLCD optimizes "answers with epistemically honest probabilities"1 | RLCR Eq. 1 adds the Brier score of a stated confidence \(q\) to a correctness reward; Theorem 1: the expected reward is maximized when \(q\) equals the true success probability. BCRL optimizes "strictly proper scoring rules" | A proper scoring rule makes honest probability reporting the optimal policy, and RLCR reports ECE 0.37 to 0.03 on HotpotQA and 0.26 to 0.10 on math at matched accuracy | In RLCR \(q\) is a number the model writes as text; TypeSafe says its model writes no text |
| Calibration is a property of groups: outcomes given probability 0.8 occur about 80% of the time2 | RLCR's expected calibration error; BCRL's calibrated "probability of correctness" | The same measurement family: stated probability against observed frequency | TypeSafe publishes no calibration measurement (Section 5) |
| Confidence is "a statistic computed from the probability distribution"13 | none: in RLCR the confidence is itself a trained output | – | A structural difference, not a resemblance |
| Probabilities "are optimized against outcomes"15 | RLCR: outcomes are correctness against a reference answer | Consistent with a proper-score objective on observed outcomes | Equally consistent with plain supervised training on outcomes; the sentence cannot tell the two apart |
The resemblance supports one conclusion and blocks two. It supports: the objective (report probabilities that match frequencies, scored by a strictly proper rule) is public, has a clean incentive argument (RLCR's Theorem 1) and has measured effects in text-generating policies. It does not support the inference that TypeSafe's reward is a Brier term or any other named rule, because no source says so. Nor does it support the inference that the training is reinforcement learning in the policy-gradient sense. The OpenJev authors note that "training with proper scores [...] is standard for classifiers"7 (§6), and their analysis shows that for a model with no sampled rationale the RLCD gradient equals the Brier-loss gradient in expectation (Section 7.6). A typed probability vector can be trained with a proper score by ordinary supervised learning, so whether the "RL" in RLCD is load-bearing is a question about the unpublished method.
7. The OpenJev-RLCD reconstruction
7.1 Provenance
"OpenJev-RLCD: A Working RLCD Implementation" (Gao and Wang, arXiv 2609.38850 v1, 30 Sep 2026) lists personal email addresses and no institution7 independent. It starts from the observation that open reproductions of Jev's capability "are trained by supervised fine-tuning and repaired with temperature scaling", while standard RLVR "makes reasoning models sharply overconfident", and derives an RL objective for reasoning models whose output is a probability distribution. Code, run logs and an API-compatible server are released; they were not inspected here.
7.2 The objective
A question \(x\) has \(K\) answer options and an outcome \(Y\sim q(\cdot\mid x)\), where \(q\) is one-hot when there is a right answer and spread when humans disagree. A reasoning model samples a rationale \(r\sim\pi_\theta(\cdot\mid x)\) that ends in the marker Answer:. Instead of sampling an answer, the method reads the model's next-token distribution over the \(K\) option tokens, \(u_\theta(x,r)=\mathrm{softmax}(\text{logits}[\text{option tokens}])\), and scores it with a strictly proper rule. The Brier form is
whose expectation is uniquely maximized at \(u=q\); the paper reports that a log score behaves the same way7 (§2). The gradient of the expected score splits into a pathwise part through the readout and a score-function part for the rationale tokens, with a leave-one-out baseline over \(M\) rationales; the authors check it against exact enumeration on tabular models (maximum error below \(10^{-15}\)) (§3.1).
- Sample rationales. For each question, \(M=4\) rationales \(r_i\sim\pi_\theta(\cdot\mid x)\) are drawn until the marker
Answer:. - Read the distribution. At the marker, a softmax over the \(K\) option-token logits gives \(u_i=u_\theta(x,r_i)\). The answer is not sampled.
- Score it. \(R_i=J(u_i,Y)\) with the Brier form above, against the one outcome \(Y\) observed for the question. A log score is also tested.
- Stage 1: calibrate (400 steps). The loss is \(-\tfrac1M\sum_i R_i\). Gradient flows through the readout only; rationales are sampled on-policy but not reinforced.
- Stage 2: reinforce (400 more steps). From the stage-1 model, add a score-function term on the rationale tokens with a leave-one-out baseline (weight \(c=0.3\)) and a KL penalty (\(\beta=0.04\)) anchoring to the stage-1 model.
Figure 4. The pipeline of Algorithm 1 in Gao and Wang, stepped through. Source: §2, §3 and Algorithm 17. Reconstruction, not TypeSafe's method.
7.3 The variance identity, with numbers
The paper's central lemma explains why scoring a group of samples goes wrong. Let \(u(r)\) be the readout after rationale \(r\) and \(p=\mathbb{E}_r u(r)\) the mixture. Then, for any distribution over rationales,
$$J(p,q)=\mathbb{E}_r\,J\big(u(r),q\big)+\mathbb{E}_r\,\lVert u(r)-p\rVert^2 .$$The proof is one line: \(J\) is affine in \(u\) apart from the \(-\lVert u\rVert^2\) term, so \(J(p,q)-\mathbb{E}_rJ(u(r),q)=\mathbb{E}_r\lVert u(r)\rVert^2-\lVert p\rVert^2=\mathbb{E}_r\lVert u(r)-p\rVert^2\)7 (Prop. 1). Scoring the mixture therefore pays a bonus for rationales that disagree with each other, "whether or not the disagreement is grounded". If the answer is sampled from the readout (\(u=e_A\), \(A\sim p\)), the per-sample objective is \(2\Pr(A=Y)-1\), which is the RLVR objective, so RLVR is "the mixture objective without its diversity term" (Corollary 1). Table 8 works the identity on a two-option question where the world is 80/20, with three policies.
| Policy | Readout after each rationale | RLVR value \(2\Pr(A{=}Y)-1\) | Per-rationale Brier | Variance bonus | Mixture Brier |
|---|---|---|---|---|---|
| A: calibrated hedge | \((0.8,0.2)\) every time | 0.36 | 0.68 | 0 | 0.68 |
| B: decisive, split matched to the world | \((1,0)\) w.p. 0.8, \((0,1)\) w.p. 0.2 | 0.36 | 0.36 | 0.32 | 0.68 |
| C: always the mode | \((1,0)\) every time | 0.60 | 0.60 | 0 | 0.60 |
Read the columns. RLVR gives A and B the same value (0.36), because sampling an answer from a hedged readout is randomizing, and it prefers C: with a one-hot reward the best policy is the mode. The mixture objective also gives A and B the same value (0.68), because B's confident wrong answers are paid back by a 0.32 bonus; it cannot tell hedging from randomizing. Only the per-rationale objective ranks A above C above B, and A is the calibrated optimum (Prop. 2: \(u(x,r)=q(x)\) for every rationale), which is why the paper scores each rationale's own distribution.
7.4 Two failure modes and the recipe they force
The per-rationale objective has a calibrated optimum but, trained end to end, fails in two ways the authors measure7 (§3.2).
- "System-One" collapse. Reasoning models end rationales decisively, so \(u\) after a rationale is nearly one-hot and the Brier score punishes every confident mistake. Hedging after an empty rationale is learned faster, so the policy gradient moves all probability to the empty rationale: on MMLU-Pro all four rationales become the bare marker within about 100 steps. A KL anchor (\(\beta=0.04\)) prevents it.
- Gradient dominance. Adding the score-function term to training from scratch makes every metric worse, monotonically in its weight: on MMLU-Pro (one seed) accuracy falls from 46.3 at \(c=0\) to 43.7 at \(c=1\) and Brier rises from 0.676 to 0.703. The term's gradient norm is 7 to 12 times the pathwise term's, so it dominates Adam's second-moment estimate.
The recipe is "calibrate, then reinforce": stage 1 trains the readout on the model's own rationales with the proper score, and stage 2 reinforces the reasoning with the same proper score as reward, anchored at the calibrated model. Without the stage-2 anchor the reasoning degenerates after about 330 steps, and a 200-step warm-up followed by 200 steps of reinforcement from scratch only ties readout-only training7 (§5.4).
7.5 What the experiments show
All methods start from Qwen3-1.7B in non-thinking mode (AdamW, learning rate \(2\times10^{-6}\), 8 questions and \(M=4\) rationales per step, three seeds, paired tests on every test item). Tasks: MMLU-Pro (10 options; the first 1,000 test questions), GSM8K-Verify (a yes/no task the authors built, "is this proposed final answer correct?", with proposals sampled from Qwen3-0.6B, 50% correct on test, 1,260 test problems) and ChaosNLI (100 annotator labels per item). Every baseline gets temperature scaling fitted on a dev split: SFT, rejection-sampling fine-tuning (RFT/STaR, with and without a KL anchor) and GRPO with a correctness reward7 (§4). The separating metrics are AURC (area under the risk-coverage curve; lower is better) and Cov@\(\epsilon\), "the largest fraction of items, ranked by confidence, whose error rate is at most \(\epsilon\)".
| Method | GSM8K-V accuracy | Brier | AURC | \(T\) | MMLU-Pro accuracy | Brier |
|---|---|---|---|---|---|---|
| base, zero-shot reasoning | 84.1 | 0.211 | 0.073 | 9.1 | 38.9 | 0.786 |
| SFT + TS | 70.4 | 0.382 | 0.174 | 2.1 | 42.3 | 0.711 |
| RFT/STaR + KL + TS | 91.4 | 0.160 | 0.075 | 6.6 | 40.9 | 0.761 |
| GRPO + KL + TS | 91.8 | 0.150 | 0.061 | 7.2 | 42.4 | 0.753 |
| RLCD stage 1, Brier readout | 90.4 | 0.164 | 0.047 | 1.6 | 47.0 | 0.667 |
| RLCD two-stage | 92.2 | 0.135 | 0.034 | 1.4 | 47.7 | 0.658 |
Three readings matter more than the headline 81% versus 19%. First, accuracy is a tie on the reasoning-essential task: after temperature scaling GRPO reaches 91.8% and two-stage RLCD 92.2%, not significantly different, and the Brier gap (−0.015) has \(p=0.09\). What differs is the ranking of confidence (AURC 0.061 against 0.034, \(p<10^{-3}\)) and the temperature each needs (GRPO and RFT \(T\approx7\), RLCD 1.4); temperature scaling rescales confidence but cannot repair a ranking in which right and wrong answers score alike7 (§5.1). Second, on the knowledge-heavy task the reasoning baselines fail: RFT without the anchor collapses to empty rationales in all three seeds (32.7% accuracy, below the base model's 38.9%), and no reasoning baseline decides any item at a 20% error budget. Third, the reward must be proper: in forked stage-2 continuations (per seed), GRPO's correctness reward matched RLCD-RL on GSM8K-Verify (both +1.8 accuracy points) but on MMLU-Pro re-inflated the calibrated readout's confidence (\(T\) from 1.4 to 3.8) and made Brier and AURC significantly worse, consistent with the RLCR and BCRL diagnosis in a small setting (§5.2).
The paper also reports where its own objective cannot help. On ChaosNLI, where the uncertainty is annotators disagreeing, reasoning cannot reduce it; the per-rationale objective "discovers that reasoning is useless here and switches it off", matching cross-entropy on direct answers (Jensen-Shannon divergence to the human label distribution 0.075 against 0.074), while the mixture objective makes rationales random and runs them to the length limit (§5.3).
7.6 What the reconstruction does and does not say about TypeSafe
It establishes that a concrete, internally consistent objective with a calibrated optimum exists, that its natural failure modes can be named and fixed, and that in one small controlled setting a proper-score readout ranks its own confidence far better than GRPO plus temperature scaling. It does not say what TypeSafe did, and its setting differs from TypeSafe's documented interface in ways that matter.
- A rationale versus no explanation. The recipe needs a sampled text rationale before the readout. TypeSafe's System One page says its models do not "generate explanations of their reasoning"15. Whether TypeSafe's training has any analogue of a rationale is not stated.
- The no-rationale case collapses to supervised learning. For "a model without rationales (a direct scorer), RLCD with an explicit distribution has the same expected gradient as the Brier loss; on CLINC150 and ChaosNLI it never beat cross-entropy (CLINC150 accuracy 94.6% vs. 96.3%), and RLVR on the same models collapsed to the mode"7 (§5.4). That is a result about the reconstruction's own objective. If TypeSafe's model is a direct typed scorer, it suggests a proper-score objective would not need policy-gradient reinforcement; if the model has an internal stochastic computation that is reinforced, it would not apply. The sources do not say which.
- Scale and task. One 1.7B model, 400 to 800 update steps, three seeds and a verification task the authors built. They write that "larger models and longer training may change the balance between the two stages"7 (§7).
Where experts disagree
The live question is whether TypeSafe's RLCD is a new post-training regime or known calibration-aware reinforcement learning under a new name. The sources do not stage it as a debate: TypeSafe asserts the first reading, no source argues the second in those words, and the outside evaluators test the released model instead. The three positions below are what the sources, taken together, support; the second is an implication of how the independent papers use the name. The related axis, train calibration in or fix how benchmarks grade (RLCR and BCRL against Kalai et al.), is covered in Module 24; the verdict places TypeSafe on it.
A new regime: a different task and a different interface
TypeSafe: the launch post, the AI primer and the founder123. Company and interview claims.
RLHF optimizes what raters prefer and RLVR what a program can verify; both are the wrong target for software that needs a trustworthy probability. RLCD changes the task to calibrated typed decisions and treats the algorithm as secondary: RLCD is "this new task", "another North Star", as DPO and its descendants also "do RLHF" without PPO, and the ranking is "right task > data > compute > algorithms"320.
- Coherent at the level of the output contract, which is public, specified to the formula, and consumed by three outside groups13817.
- The motive is independently supported: binary rewards and 0-1 grading reward guessing and degrade calibration232425.
- Naming a regime by its task is how RLHF itself is named.
- Novelty is asserted, not shown: no mechanism, data, ablation or calibration table is public (Section 4).
- The objective, calibrated probabilities scored by a strictly proper rule, predates it (Section 6).
- Outside calibration evidence is partial: good ranking, base-rate mismatch within a benchmark8.
Known calibration-aware training, plus a product interface
Implied by the independent papers' usage, not stated by them: dos Santos defines RLCD as a method that "trains against strictly proper scoring rules"9; the Laya project applies the label to its own open model10; Gao and Wang build theirs from standard parts and note that proper-score training "is standard for classifiers"7.
If the objective is a proper score of the model's reported distribution, the closest public methods are RLCR's Brier reward and BCRL's proper-scoring interventions2324, and for a model that emits a distribution directly the reconstruction's own analysis shows the RL objective equals Brier-loss training in expectation7. On this reading the new thing is the interface and packaging, not the training.
- Parsimonious: it needs no hidden method beyond what the literature already shows has the claimed effect.
- Open reproductions trained by SFT and repaired with temperature scaling exist7, and the Laya README reports mean ECE falling from 0.466 to 0.081 after temperatures are refit10.
- It assumes the undisclosed method is no more than the public ones, which no source can show.
- The public methods act on text policies with verbalized confidence; none produces typed parallel output, and synthetic data at scale may be the real contribution.
- dos Santos and Laya infer from the name; neither has seen the training.
Judge the released model, not the label
The outside evaluators: Li et al. (CMU), Guo et al., the DMB repository, and Ronacher in the press178164.
Each says the training is not public, then tests behavior. The consistent finding: the ranking by confidence is useful, the absolute probability is not portable across tasks, and the user must set and validate thresholds locally ("it delegates the hallucination problem a little bit to the user")4. Whether the regime is new is not their question.
- Testable today without disclosure, and it matches how an engineer decides.
- Produced the only calibration and latency numbers that were not computed by the vendor.
- Cannot distinguish a new training regime from a good SFT-plus-temperature-scaling model: both give a useful ranking.
- Small, single-version, self-reported work: two preprints (one under review) and a community repository.
Split the novelty claim into four parts and it resolves unevenly. The objective (calibrated probabilities under a strictly proper score) is not new: RLCR was posted in July 202523 and the Brier score dates to 19507. The interface (typed, parallel, no generated text, a probability vector per question) is real, documented and behaviorally confirmed by three outside groups. The data (all synthetic, generated in-house) is claimed and unseen, and may be where the advance lies. The algorithm is unknown. So "new regime" is supported at the interface and unsupported at the training; "rebranding" is unsupported as a statement about TypeSafe's method, because there is no method to compare, though it describes the name's drift in third-party use. On the secondary axis, TypeSafe trains calibration in, as RLCR and BCRL do, changes the interface so calibration is inspectable by software, and opts out of public grading rather than reforming it as Kalai et al. propose.
Position 1 is right if the undisclosed training cannot be reduced to a proper-score loss on observed outcomes, for instance a reinforcement loop over sampled computations in synthetic environments, and if that yields a measurable advantage. Position 2 is right if it reduces to Brier or log loss on synthetic labeled outcomes, possibly with a temperature fit, in which case "RL" is a label. Four things would settle it: (1) disclosure of the reward or loss, the data generator, and per-task calibration tables with reliability diagrams; (2) an ablation only TypeSafe can run, same data and size, plain proper-score loss with temperature scaling against the RL method; (3) a third-party, pre-registered per-task calibration study of a later Jev version, using Just Ask Jev's split-half threshold protocol as a template8; (4) a common held-out suite run by a third party against Jev and an open SFT-plus-temperature-scaling model. Until one exists, treat "RLCD" as the name of a regime whose output contract you can test and whose training you cannot see, and use the public machinery in Module 24 for anything you need to implement.
What is changing
As of 2026-10-27 the situation is six weeks old and moving. TypeSafe shipped Jev 1.14.0 on 14 Oct 20265, with opt-in "danger zone" features (images, a bounding-box type, longer context), and it now shares some internal measurements, only for open-weight models26; its stated policy remains no standard benchmark tables and dated one-off evaluations18. The founder says he wants to write about mode dropping and calls RLCD unpublished "not yet"3, so a paper or a post that finally states the reward would supersede Sections 4 and 7.6 at once. Independent work is arriving faster than company disclosure: the three evaluations in Section 5 appeared between 22 and 29 Sep, the reconstruction on 30 Sep, and the preprints that use the name are still unreviewed (Just Ask Jev is marked under review for ICLR 2027)8. All outside numbers in this module concern jev-1.13.0; the company says 1.14.0 is "small, subtle improvements to everything"26, so none of them should be assumed to carry over.
References
- officialAlmeida, D. (TypeSafe AI). Introducing System One Models & Jev. Company blog, 15 Sep 2026. https://typesafe.ai/blog/introducing-system-one-models-and-jev. Accessed 2026-10-26. Used for: the sentence naming RLCD, the comparison table, workflow-eval caveats, hallucination caveat, FAQ answers (training data, "new training algorithm", benchmarks); company claims.
- officialTypeSafe AI. AI primer (docs.typesafe.ai). Live documentation, no version stamp. https://docs.typesafe.ai/introduction/machine-learning-primer.md. Accessed 2026-10-26. Used for: RLCD as a third route, the output contract, population-level calibration, the critique of RLHF; company claims.
- talkLatent Space (swyx) with Almeida, D. Jev: System One models for Prod, not God. Podcast episode and transcript, 2026 (publication date not captured). https://latent.space/p/jev/. Accessed 2026-10-27. Used for: unpublished status (00:24:59 to 00:25:05 and show notes), RLCD as a task (00:27:18, 00:27:34, 01:20:21), synthetic data (00:22:04), "get many things wrong" (01:08:14); full transcript text read.
- analysisFernholz, T. A new kind of AI model from a ChatGPT inventor is thrilling developers. TechCrunch, 18 Sep 2026. https://techcrunch.com/2026/09/18/a-new-kind-of-ai-model-from-a-chatgpt-inventor-is-thrilling-developers/. Accessed 2026-10-27. Used for: press account, Vercel and Bryo user reports, Ronacher's critique, "transformer-based", unconfirmed open-weight suspicion; news, not a primary source.
- officialTypeSafe AI. Models (docs.typesafe.ai). Live documentation. https://docs.typesafe.ai/models.md. Accessed 2026-10-27. Used for: Jev 1.14.0 date (14 Oct 2026), price, 64k and 32k context limits, "ingests the state once", "trained with RLCD ... same weights serve every account", data handling.
- officialTypeSafe AI. Documentation index (llms.txt). https://docs.typesafe.ai/llms.txt. Accessed 2026-10-27. Used for: the AI primer is the only page on training.
- paperGao, Z., Wang, P. OpenJev-RLCD: A Working RLCD Implementation. arXiv 2609.38850 v1, 30 Sep 2026. https://arxiv.org/pdf/2609.38850v1. Accessed 2026-10-26. Used for: the reconstruction (§2, §3, Algorithm 1, Tables 1, 2, 4), Prop. 1 and Corollary 1, ablations (§5.4), limitations (§7), statement that TypeSafe discloses no algorithm (§1); independent reconstruction, not TypeSafe's method.
- paperGuo, R., Liu, Y., Deng, G., Li, Y., Zhao, L., Wu, Y., Chen, S., Zhang, Y., Zhang, L. Y. Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures. arXiv 2609.29429 v1, 24 Sep 2026 (marked under review for ICLR 2027). https://arxiv.org/pdf/2609.29429v1. Accessed 2026-10-27. Used for: use of the name RLCD (§1, footnote 1), pooled and per-benchmark ECE, AUROC and threshold results (abstract, §4.4), limitations (§6).
- paperdos Santos, J. A. (arXiv metadata: dos Santos Barbosa). Calibrated Decision Models for Autonomous Penetration-Testing Harnesses: JEV and Laya as System One Decision Layers for LLM-Driven Pentest Agents. arXiv 2609.28940 v1, 24 Sep 2026. https://arxiv.org/pdf/2609.28940. Accessed 2026-10-26. Used for: the definition of RLCD (§II-B), the "non-autoregressive" and "single forward pass" attribution (§III), the ECE 0.246 attribution (§XII-B, Table IX); both attributions are unverified against TypeSafe's pages.
- repoLaya project (github.com/NandhaKishorM/laya). README. https://github.com/NandhaKishorM/laya. Accessed 2026-10-27. Used for: the open model's description as trained with "reinforcement learning against strictly proper scoring rules (RLCD)", the "Laya (with routing) vs Jev" table whose Jev column is "third-party published, never measured here", ECE 0.466 to 0.081 after temperature refit; another project's page, whose open model positions itself against Jev; labeled as such.
- paperYang, K., Klein, D., Celikyilmaz, A., Peng, N., Tian, Y. RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment. ICLR 2024; arXiv 2307.12950 v3, 16 Mar 2024. https://arxiv.org/pdf/2307.12950v3. Accessed 2026-10-26. Used for: the unrelated 2023 method that shares the acronym (abstract, §1); treated in Module 8.
- officialTypeSafe AI. Introduction (docs.typesafe.ai). Live documentation. https://docs.typesafe.ai/introduction.md. Accessed 2026-10-26. Used for: the three primitives and what they return, "evaluated in parallel and in isolation".
- officialTypeSafe AI. Confidence (docs.typesafe.ai). Live documentation. https://docs.typesafe.ai/confidence.md. Accessed 2026-10-26. Used for: the Noul, Choice and Score confidence formulas and worked examples, threshold guidance.
- officialTypeSafe AI. Parallel questions (cookbook, docs.typesafe.ai). https://docs.typesafe.ai/cookbooks/parallel_questions.md. Accessed 2026-10-27. Used for: the company's statement that each question is scored on its own so an answer does not depend on the other questions in the request; company-run.
- officialTypeSafe AI. System One (concept page, docs.typesafe.ai). Live documentation. https://docs.typesafe.ai/concepts/system-one.md. Accessed 2026-10-27. Used for: "probabilities are optimized against outcomes", "do not ... generate explanations of their reasoning", calibration across groups; company claims.
- reponibzard. DMB: decision-model benchmark (README and linked reports). Run of 29 Sep 2026, pilot of 26 Sep 2026. https://github.com/nibzard/decision-model-benchmark. Accessed 2026-10-27. Used for: Banking77, CLINC150 and NLU++ results, frozen-threshold results, latency table, the 40-to-200x statement, the provenance of ECE 0.246 (S5 suite); a community benchmark, not peer reviewed; only the README was read.
- paperLi, Y., Miao, Y., Krishnan, R., Padman, R. (Carnegie Mellon University). JEV-as-a-Judge: Accept When Confident, Escalate When Unsure. arXiv 2609.26550 v5, 9 Oct 2026 (v1 22 Sep 2026). https://arxiv.org/pdf/2609.26550v5. Accessed 2026-10-27. Used for: accuracy against sixteen judges, cascade result, §7.2 probability quality and temperature-scaling transfer, Limitations ("Black box").
- blogTypeSafe AI. Lies, Damned Lies, and Benchmarks. Company blog, 11 Sep 2026. https://typesafe.ai/blog/antibenchmaxxing. Accessed 2026-10-27. Used for: "no standard benchmark table in our model releases", dated snapshots; company policy.
- paperOuyang, L., Wu, J., Jiang, X., Almeida, D., et al. Training language models to follow instructions with human feedback. arXiv 2203.02155 v1, 2022. https://arxiv.org/pdf/2203.02155v1. Accessed 2026-10-26. Used for: Almeida as a starred author (author list, first page).
- blogTypeSafe AI. The Bitterest Lesson. Company blog, 10 Sep 2026. https://typesafe.ai/blog/bitterest-lesson. Accessed 2026-10-27. Used for: "doing the right task > data > compute > algorithms"; company position.
- officialTypeSafe AI. Jev 1.13 jaggedness (docs.typesafe.ai). Last reviewed 2026-10-02. https://docs.typesafe.ai/model-jaggedness/jev-1.13.md. Accessed 2026-10-27. Used for: the company's list of known failure modes, option-order effect, weak numerical calibration of score levels.
- officialTypeSafe AI. Self-consistency: choices (cookbook, docs.typesafe.ai). https://docs.typesafe.ai/cookbooks/consistency_choice_cookbook.md. Accessed 2026-10-27. Used for: "measure repeatability only" and "not a calibrated guarantee".
- paperDamani, M., Puri, I., Slocum, S., Shenfeld, I., Choshen, L., Kim, Y., Andreas, J. Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (RLCR). ICLR 2026; arXiv 2507.16806 v2, 15 May 2026 (v1 22 Jul 2025). https://arxiv.org/pdf/2507.16806v2. Accessed 2026-10-26. Used for: the diagnosis that binary rewards do not penalize guessing, Eq. 1 and Theorem 1, ECE results (abstract, §1, §3).
- paperWu, J., Liu, J., Zeng, Z., Zhan, T., et al. Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning. arXiv 2512.19920 v3, 28 Jan 2026. https://arxiv.org/abs/2512.19920. Accessed 2026-10-26. Used for: the abstract's claims about binary RLVR and proper-scoring interventions; abstract only.
- paperKalai, A. T., Nachum, O., Vempala, S. S., Zhang, E. Why Language Models Hallucinate. arXiv 2509.04664 v1, 4 Sep 2025. https://arxiv.org/pdf/2509.04664v1. Accessed 2026-10-26. Used for: the claim that training and evaluation reward guessing and that abstaining is strictly suboptimal under 0-1 grading (abstract, §4).
- blogTypeSafe AI. Jev: For the builders AND dreamers. Company blog, 14 Oct 2026. https://typesafe.ai/blog/jev-1-14. Accessed 2026-10-27. Used for: Jev 1.14.0 release, experimental images, box type and long context, internal measurement shared for open-weight models only.