A small model that does nothing but judge

Made by AISummarised by AI (Claude) from one piece of material it read (a talk or an explainer).How it's made

Jev, a judgment-only model that cannot hold a conversation or write code and instead only picks from the options it is given, assigns a score and returns the probability of being right, was taken up in several places in the same week. It spread fast enough that even the spelling wobbled between Jev and JEV. One write-up classified 100 Korean-language inquiries with the same instruction and reported that a direct call took 2.1 seconds and $0.0037 while a large model took 17.4 seconds and $0.28, which is one team's own measurement. In the same test the classifications matched on 89 of the 100 cases, and the judgment of whether a case was urgent was the same on all 100. Microsoft has also put out a judgment model of the same kind, which shows that the concern is shared in several places: calling a large model every time for a small decision, such as choosing a tool or deciding whether to retry, comes back as cost and latency.

Key points

  • A new category of model that returns only a choice among options, a score or a probability
  • Across 100 inquiries the split was 2.1 seconds and $0.0037 against 17.4 seconds and $0.28
  • Classification matched on 89 of 100 cases; urgency was the same on all 100
  • Calling a large model every time for a small decision piles up latency and cost

This is a note: a summary an AI (Claude) wrote of one piece of material it read (a talk or an explainer). It has no source link; events checked against a source are kept as records.

Same week's notes

Where you stack context is what makes the gapAdding more features made it worseJudgment up front, code behind, a person when it is unclearA spec written first, and review gatesWhy reading only the scoreboard leads you wrong