Jev is a multiple-choice model. You give it a context, a question and a list of allowed answers, and it returns a probability for each answer in a single forward pass. It never generates text. So the question for this post is: how do you get a probability over a handful of options out of a decoder LLM?

One recommendation up front. Even in the age of Claude Code, write a few lines of this yourself. Everything below runs on a free Colab CPU with a 0.6B model.

A decoder LLM in a few lines

The backbone maps tokens \(x_1, \dots, x_n\) to latent vectors \(H \in \mathbb{R}^{n \times d}\), where \(n\) is the sequence length and \(d\) the latent dimension (also called hidden size). Because of the causal mask, row \(h_i\) depends only on \(x_1, \dots, x_i\). The LM head is one matrix \(W_U \in \mathbb{R}^{V \times d}\), shared across positions, where \(V\) is the vocabulary size, the number of different tokens the model knows:

So \(h_i\) predicts token \(i+1\). For Qwen3-0.6B, the latent dimension is \(d = 1024\) and the vocabulary size is \(V = 151{,}936\).

Two words of jargon:

• Prefill is the first pass, over the whole prompt. All \(n\) prompt tokens are processed in parallel, and the pass gives the distribution of the first new token. It also leaves behind the keys and values of every layer, the KV cache.
• Decode is every pass after that. Each one produces one more token and reuses the cache.

A single prefill pass costs more than a single decode pass, because it handles \(n\) tokens instead of one. But there is only one prefill, while decoding is autoregressive: every new token needs its own pass, and the passes have to run one after the other because each token depends on the previous ones. An answer of 500 tokens means 500 passes. For anything but very short answers, decoding is where the time goes.

Everything in this post happens in the prefill. We never decode.

LLMs as classifiers

Transformers were used as classifiers long before Jev. The recipe is always the same: pick one latent vector that summarizes the whole input and put a small linear head on it. Which vector that is depends on the type of model.

• Encoder models like BERT have no causal mask, so every token attends to every other token. A special [CLS] token is added at the start of the input. It stands for no word, and its latent vector \(h_{\texttt{[CLS]}}\) is used as the summary of the sequence.
• Decoder models like GPT or Qwen are trained to predict the next token, so they must not look into the future of a sentence. The causal mask enforces this: every token only attends to the tokens before it, so \(h_i\) only knows tokens \(1, \dots, i\). The only vector that has seen the whole input is the last one, \(h_n\), where \(n\) is the length of the input sequence.

In both cases you replace the LM head \(W_U\) with a classification head \(W_C \in \mathbb{R}^{C \times d}\), one row per class, and fine-tune:

This works well when the classes are fixed, for example in a spam filter with the two classes “this is spam” and “this is not spam”.

The problem with this approach is that \(W_C\) fixes both the number of classes \(C\) and what each class means. With an approach like Jev, both change at every pass, because the possible answers are given as text in the prompt:

Question: The traffic light is red. What should the self-driving car do?
Options: speed up, stop, turn around

The next prompt may have five options that say something completely different.

There is a way around this that still uses a classification head: give it a single output, \(w \in \mathbb{R}^{1 \times d}\). Run one pass per option, each time with the question and only that option in the prompt, and read the single output as a logit, \(s_j = w\, h_n^{(j)}\). A softmax over the logits of all options gives the probabilities. This is the standard way to do multiple choice with BERT. It does generalize to new options and to any number of them, because \(w\) is not tied to a particular option. It learns to judge whether what it just read is a good answer. The downsides are that it needs one full pass per option, and that each option is scored without seeing the others.

Choosing between options given in the prompt

So what we are looking for is a readout that takes the options from the prompt and returns a probability for each of them. It should have no weights that belong to a specific option, and it should get by with the prefill alone. Below are three ways to do that. The first two need no training and only reuse the LM head. The third is the pointer head.

The running example, with the same setup for all three methods:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

name = "Qwen/Qwen3-0.6B"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.float32).eval()

question = "The traffic light is red. What should the self-driving car do?"
options = ["speed up", "stop", "turn around"]  # correct: stop

def chat(system, user):
    """Wrap a system and a user message in the chat format Qwen was trained on (still a string)."""
    messages = [{"role": "system", "content": system}, {"role": "user", "content": user}]
    return tok.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,  # appends "<|im_start|>assistant\n": your turn to speak
        enable_thinking=False,       # skip Qwen3's <think>...</think> block
    )

Method 1: label logits

Write the options into the prompt with a letter in front of each one, and tell the model to answer with a single letter. This is the full prompt:

<|im_start|>system
Answer with a single letter: A, B, or C. Do not explain.<|im_end|>
<|im_start|>user
The traffic light is red. What should the self-driving car do?

A. speed up
B. stop
C. turn around<|im_end|>
<|im_start|>assistant
<think>

</think>

The empty <think> block is how Qwen3 is told to skip its reasoning and answer directly. (We cannot use thinking here: the reasoning is generated text, so it would mean decoding, and the next token after the prompt would be the first word of the reasoning instead of the answer letter.)

If we let the model generate from here, the next token would be A, B or C. We do not generate. We take the logits \(z_n\) for that next token. \(z_n\) is a vector with \(V = 151{,}936\) entries, one for every token in the vocabulary. In Qwen’s vocabulary the token A has the id 32, so \(z_n[\text{A}]\) is entry 32 of that vector. B and C have the ids 33 and 34 (label_ids in the code below). We keep only these three entries and take a softmax over them:

labels = ["A", "B", "C"]
listing = "\n".join(f"{l}. {o}" for l, o in zip(labels, options))
prompt = chat("Answer with a single letter: A, B, or C. Do not explain.", f"{question}\n\n{listing}")

ids = tok(prompt, return_tensors="pt").input_ids   # (1, n)
with torch.no_grad():
    out = model(ids, output_hidden_states=True)    # one forward pass, no generate()

h = out.hidden_states[-1][0, -1]                   # (d,)  last latent vector
W_U = model.get_output_embeddings().weight         # (V, d)
z = W_U @ h                                        # (V,)  same as out.logits[0, -1]

label_ids = [tok.encode(l, add_special_tokens=False)[0] for l in labels]
p = z[label_ids].softmax(-1)                       # softmax over the option labels only

A (speed up): 0.399
B (stop): 0.513
C (turn around): 0.088

One pass, no training. The method has two limitations, though.

The labels must be single tokens. There are roughly 52 single-letter tokens, which caps the number of options.

The model scores a letter, not an answer. The text “stop” never gets a probability. The letter B does, and the model has to work out from the prompt that B stands for “stop”. On top of that it has its own preferences for certain letters, whatever they stand for.

You can see this by changing which option gets which letter. The question and the three options stay the same, only the order in which they are listed changes:

A. speed up   B. stop       C. turn around   ->  p(stop) = 0.513
A. stop       B. speed up   C. turn around   ->  p(stop) = 0.996

Over all six orderings, “stop” gets between 0.51 and 1.00. If the model only looked at the content of the options, we would get the same number six times.

Method 2: sequence likelihood

This time we do not look only at a letter but at the probability of the entire option. The prompt contains only the question, and each option is written out as the full answer the model could give:

The car should speed up.
The car should stop.
The car should turn around.

For each of these candidate answers we ask: how likely is it that the model produces exactly this text after the prompt? An answer is a sequence of tokens \(a = (a_1, \dots, a_m)\), so its probability is a product over tokens, or a sum in log space:

To compute it, append the answer to the prompt and run one pass over prompt + answer. Position \(i\) predicts token \(i+1\), so the logits at positions \(n, \dots, n+m-1\) contain all \(m\) terms. Nothing is generated. Do this once per answer and take a softmax over the three log-likelihoods.

prompt = chat("Answer in one short sentence.", question)
n = tok(prompt, return_tensors="pt").input_ids.shape[1]
answers = [f"The car should {o}." for o in options]

scores = []
for a in answers:
    ids = tok(prompt + a, return_tensors="pt").input_ids   # prompt + answer in one pass
    with torch.no_grad():
        logp = model(ids).logits[0].log_softmax(-1)        # (n + m, V)
    target = ids[0, n:]                                    # the m answer tokens
    # the logits at position i predict token i + 1, hence the shift by one
    logp_tokens = logp[n - 1 : -1].gather(-1, target[:, None]).squeeze(-1)
    scores.append(logp_tokens.sum())                       # log p(answer | prompt)

p = torch.stack(scores).softmax(-1)

The car should speed up.     log p = -23.76   p = 0.000
The car should stop.         log p = -14.84   p = 1.000
The car should turn around.  log p = -29.70   p = 0.000

The answers can now be any text of any length, still without training. Two things to keep in mind:

Cost. One pass per answer instead of one pass in total. The prompt is the same every time, so its KV cache can be reused and each extra pass only pays for the answer tokens.

Length bias. Every token adds a negative term to the sum, so a longer answer gets a lower score just for being longer. The usual fix is to divide the sum by \(m\). Here the answers have five or six tokens, so it does not matter.

Method 3: pointer head

The prompt contains the question followed by the options, one per line, without letters:

The traffic light is red. What should the self-driving car do?

Options:
speed up
stop
turn around

When using a pointer head, we run a forward pass over the prompt and select the latent vectors at four token positions (the number of options plus one) in the following way:

• \(h_n\), the vector at the last token of the prompt. It has seen the question and all the options. This is the query: it encodes which answer the question calls for.
• \(h_{e_j}\) for \(j = 1, 2, 3\), the vector at the last token of option \(j\), where \(e_j\) is the index of that token. For “turn around” that is the vector at “around”. Because of the causal mask it is the first vector that has seen the whole option, so it is our summary of option \(j\).

The pointer head consists of two matrices \(W_q, W_k \in \mathbb{R}^{r \times d}\). They project the query and the option summaries into a common \(r\)-dimensional space, where the score of an option is an inner product:

This is the formula of an attention head, with the attention weights as the output: the head points at one of the options in the input, hence the name. \(W_q\) and \(W_k\) are trained with cross-entropy on questions with known answers.

def latents(question, options):
    """One pass over question + options. Returns h_n and the latent vector at the last token of each option."""
    prompt = chat("Pick one of the options.", f"{question}\n\nOptions:\n" + "\n".join(options))
    enc = tok(prompt, return_tensors="pt", return_offsets_mapping=True)

    # index of the last token of each option, found via character offsets
    offsets = enc.offset_mapping[0].tolist()
    pos, start = [], prompt.index("Options:")
    for o in options:
        end = prompt.index(o, start) + len(o)
        pos.append(next(i for i, (s, e) in enumerate(offsets) if s < end <= e))
        start = end

    with torch.no_grad():
        H = model(enc.input_ids, output_hidden_states=True).hidden_states[-1][0]   # (n, d)
    return H[-1], H[pos]

h_query, h_options = latents(question, options)   # (d,) and (num_options, d)

torch.manual_seed(0)
d, r = h_query.shape[-1], 256
W_q = torch.nn.Linear(d, r, bias=False)   # trainable
W_k = torch.nn.Linear(d, r, bias=False)   # trainable

def scores(h_query, h_options):
    return W_k(h_options) @ W_q(h_query) / r**0.5   # (num_options,) one score per option

s = scores(h_query, h_options)
target = torch.tensor([1])                # index of the correct option: stop
loss = torch.nn.functional.cross_entropy(s[None], target)

p    = [0.898, 0.029, 0.073]
loss = 3.531

The matrices are untrained, so these probabilities mean nothing yet. What the code does show is how little is needed. A single forward pass scores all options at once, however many there are and however long they are. And the only new parameters are the two matrices, \(2rd = 524{,}288\) in total for \(r = 256\), next to the 0.6 billion of the backbone.

One consequence of the causal mask: \(h_{e_j}\) only sees what comes before it. So the question has to come before the options, and option \(j\) knows about options \(1, \dots, j-1\) but not the later ones. Only the query \(h_n\) sees everything.

How is this different from a classifier?

Rewrite the score as

This has the form of the classifier from the beginning, \(W_C h_n\), with \(w_j\) as the row for class \(j\). The difference is where that row comes from. In the classifier, \(w_j\) is a trained parameter, so each class is stored in the weights. Here \(w_j\) is computed by the backbone from the text of option \(j\). The trained part, \(W_q^\top W_k\), contains nothing about any particular option. It only learns how to compare a query with an option summary.

That is why the head can handle options it has never seen, and any number of them: a new option gets its \(w_j\) from the same forward pass. How well this works in practice depends on how varied the training questions are, and that is an empirical question.

People in vision will recognise the setup. A linear probe on a frozen DINO model is the classifier version: a strong backbone, fixed features, and a small trained \(W_C\) with fixed classes. The pointer head makes the same bet that the features already contain the answer and a small readout is enough. It just compares features with features instead of features with stored class vectors.

A training loop (that overfits)

To close the loop, here is the training of \(W_q\) and \(W_k\) on our single question. The backbone stays frozen, so its latent vectors are computed once and reused in every step.

opt = torch.optim.Adam([*W_q.parameters(), *W_k.parameters()], lr=1e-5)

for step in range(31):
    s = scores(h_query, h_options)
    loss = torch.nn.functional.cross_entropy(s[None], target)
    opt.zero_grad()
    loss.backward()
    opt.step()

step  0   loss = 3.531   p(stop) = 0.029
step  5   loss = 0.168   p(stop) = 0.845
step 10   loss = 0.005   p(stop) = 0.995
step 20   loss = 0.000   p(stop) = 1.000
step 30   loss = 0.000   p(stop) = 1.000

The loss goes to zero within a few steps. That is not a success. With one training example and half a million parameters, the head completely overfits to this one question. You can see it by changing the question so that “stop” is no longer correct:

green = "The traffic light is green. What should the self-driving car do?"
new_options = ["drive on", "stop", "turn around"]
p = scores(*latents(green, new_options)).softmax(-1)

drive on     0.000
stop         1.000
turn around  0.000

The head has learned “the answer is stop”, not how to match a question with an option. To learn that, it needs many different questions, with different options and with the correct answer in different places.

Summary

As far as I can tell, the pointer head is what Jev-style models use, together with LoRA on the backbone. LoRA (low-rank adaptation) freezes a weight matrix \(W\) and trains a low-rank update \(W + BA\) instead, with \(B \in \mathbb{R}^{d \times k}\), \(A \in \mathbb{R}^{k \times d}\) and \(k \ll d\). That is a small fraction of the parameters, and it lets the backbone adapt its latent vectors to the head.

In the next post we train the pointer head on a simple robotics example. After that I want to look at vision transformers for Jev-like models.

I know I should take the discussions on social media with a huge grain of salt. But they’re hard to miss and there’s probably some truth in there. The way we work is going to change a lot over the next few years. Parts of jobs, probably not entire jobs, will get automated. We already see it in software engineering. At my level of SWE, I’m barely writing any code myself anymore. This might be different for more security-sensitive parts of the architecture, or for people working on bigger, more stable codebases. But I’m pretty convinced most SWEs work the way I do now: find a bug or think of a feature, go to Claude Code planning mode, prompt back and forth a few times, then let it rip. Some testing. Done.

The loud version of this on social media gets exaggerated to “software engineering is dead, no need to hire SWEs, fire 50% of the workforce.” That’s wishful thinking from people selling a narrative. My job has changed, but I don’t have less work. My focus has shifted. I’m writing this mostly for myself, to figure out how I want to use my skills going forward and what new ones I should pick up.

What am I good at?

I consider myself an all-rounder. I did an apprenticeship as an electronics technician when I was 16: soldering, programming microprocessors, designing simple circuits, plus 10–14 weeks of mechanical shop work where we learned to drill, turn parts on a lathe, mill, and do a bit of CNC programming.

I’m not good at any of these anymore, but I’m sure I could pick them up again pretty quickly. More importantly, it was my first contact with the real business world, and it shaped how I approached university. To be frank, by the end of the apprenticeship I knew I didn’t want to stay in that profession at all, so I went back to do my high-school degree and then studied physics. Physics felt like a good bet because it teaches relevant math and a bit of programming, and I was vaguely considering finance later, where physicists are in demand.

While studying physics I started to enjoy the process of learning for its own sake, which is why I wanted to do a PhD. There’s a lot of discourse right now claiming PhDs are useless and you should just go to industry. I disagree, especially if you’re someone like me who just wants to keep learning. Being surrounded by brilliant people for more than five years was transformative. You can do that in industry too, but you’ll have much less freedom to chase rabbit holes; most of the time you’re on a tighter schedule. That has its own advantages, of course. If you have a clear career goal and a PhD is just a means to get there, you probably won’t enjoy the PhD itself. For context: I never planned to stay in academia. In Switzerland a PhD is a fairly common advanced degree.

After a one-year postdoc, I went to industry. First to a startup called Menten, where we tried to show how quantum computing could improve or accelerate protein design. The premise was cursed from the start: classical optimization on a quantum computer just doesn’t work. Menten realized this and pivoted away from quantum at the same time I did, and I had a gig lined up at Extropic where I started as a Staff Scientist. Extropic was the perfect blend of research and industry: figuring out how to build a new computing paradigm and designing algorithms for it. I left when they decided to relocate everyone to the US, which wasn’t possible for me at the time.

That brought me to Axiomatic as AI Lead, where I had a deep dive on agentic AI. The pace was insane. Late 2024 to December 2025 felt like a different decade every quarter. December 2025 in particular was when Claude Code became really good. So good that most people just started running it with --dangerously-skip-permissions. It changed the way I work.

I’m now at Noumenal and the work is extremely diverse. From general discussions about the direction of the company, to grant applications, to writing code for our robots, to testing them in the lab. We all do everything, and AI agents make that genuinely possible. I let an agent plan my meetings, give me briefs and reminders ahead of time, and help me track my own and others’ tasks. The main operational benefit is that it lowers the cost of switching tasks. I don’t yet let it do many things autonomously (like sending emails), but I use it as a second memory and as a virtual assistant that brings me up to speed before I context-switch into something.

On the development side, AI agents do a lot of the heavy lifting. Anyone who has worked with ROS knows how painful that software is, and an agent can just take care of wrapping our Python scripts into ROS nodes for me. I had never used ROS before this job, and I can take care of all of it myself. Same for quick prototypes of frontend interfaces, scripts to parse a new dataset, or one-off tools I need for an afternoon. All of that previously would have meant either a context-switch into an unfamiliar stack or asking someone else to do it.

Spare a thought for the juniors

I don’t envy the next generation. The uncertainty is big, and it’s genuinely hard to learn deep skills if you never get the chance to explore. Thinking about code, really thinking, including on an architectural level, is what teaches you to code. Watching an agent produce something that works isn’t the same thing.

It’s also why a lot of vibe-coded projects fail. People who don’t really know how software works make big architectural mistakes from the very beginning, and agents aren’t yet good enough to fix those ad hoc. Worse, a messy codebase actively makes the agent perform worse, which makes the codebase messier, which makes the agent worse. You end up in a hole you can’t get out of. Jank in, jank out.

I don’t have a clean answer to either of these. I think there will still be deeply skilled engineers in five years, and they’ll be more valuable, not less. But the path to becoming one looks much harder from where I’m standing.