What Noughts and Crosses Taught Me About Asking Jev the Right Question

Jev is a new type of AI model, and it works differently to the LLMs we’ve all got used to. There is no chat, no prompt-and-parse, and it never generates text. You send it some state and a set of typed questions - pick one of these options, how likely is this statement, where on this scale - and it sends back typed answers with probabilities your code can use directly. It does that in about a tenth of a second, for next to nothing. TypeSafe call it a “System One” model: fast judgement, no thinking out loud.

I’ve been using it in a couple of projects for things like classifying traffic sources, and I wanted a better feel for what it is and isn’t good at. The trouble with judging an AI model on real work is that you rarely know the right answer yourself. So I picked a job where I always would: noughts and crosses.

The game is solved. There are 5,478 positions that can occur, and for every one a few lines of code can tell you exactly which moves are best. That means every single answer Jev gives can be marked right or wrong, which is a luxury you don’t normally get.

TL;DR: the code, the game and all the results are on GitHub at robwent/jev-tic-tac-toe. The model never changed during any of this. What changed was the information I gave it and how I asked, and that moved it from 55% best moves to 83%. I’m not claiming any authority here - this is one model version, one game and one weekend, built with Claude Code doing most of the typing. It’s just what we found.

The setup

For each position, one request with four questions:

• move - a Choice between the empty cells. It can’t make an illegal move because illegal moves aren’t offered.
• can I win now? and is my opponent threatening to win? - two yes/no probabilities.
• how does this end? - a Score across loses / draws / wins.

A minimax solver marks the answers. I only count “decision positions” - the 3,191 where at least one legal move is genuinely worse than another. In the rest, every move is equally good (usually because the game is already lost), so they’d only flatter the numbers. A player picking at random gets about 40% of decision positions right, so that’s the floor.

Before looking at any results I set aside 15% of positions for tuning and kept the rest as a test set, split so that rotations and mirror images of the same board always land on the same side. Wording was only ever adjusted against the small set. Unless I say otherwise, the numbers below are from the test set.

The whole first benchmark was 18,080 requests. It cost $0.49 and not one request failed.

Round one: how you draw the board

I started with the most obvious question I could write - “Choose the best cell for X to play next.” - and sent the same positions described four ways:

1. grid - three strings like "X X ."
2. cells - nine named cells: top_left: "X", middle_middle: "empty" and so on
3. lines - the named cells, plus all eight lines written out, e.g. row_1: { top_left: "X", top_middle: "X", top_right: "empty" }
4. rules - the same as lines, plus three sentences explaining the rules

Board given as	Best move	Missed a win	Missed a block
grid	55.4%	22.2%	14.5%
cells	57.6%	20.1%	15.1%
lines	69.4%	11.0%	12.6%
rules	70.8%	8.5%	13.6%

Naming the cells instead of drawing a grid did very little. Writing out the lines was worth 12 points. Explaining the rules was worth about one, which is within the noise.

That makes sense once you think about what a board looks like as text. In a grid, the three cells of the right-hand column are nowhere near each other in the string. A person sees geometry; the model sees characters. Putting X, X, empty side by side turns “is there a line here?” from a spatial problem into a reading problem.

Explaining the rules didn’t help because it can’t reason its way from rules to a move in a single pass. It isn’t that kind of model.

I could beat it easily at this stage. My favourite was opening in the centre, then taking a corner: it would ignore the diagonal and I’d win on move three, every time.

Round two: one clearer sentence

Here’s the thing that surprised me most. In the same request where it failed to play a winning move, it would often answer “can I win now?” correctly. With the rules board it got that question right 95% of the time and still missed the winning move in 8.5% of positions.

The questions in a request are answered independently. None of them can see the others’ answers. So the knowledge was there, and the move question wasn’t using it.

TypeSafe’s docs are upfront that Jev reads instructions literally, and “best” is about as vague as an instruction gets. So I rewrote the move question to say what I actually meant:

Choose the cell for X to play next. When X has two marks in a line and the third cell of that line is empty, choose that empty cell. Otherwise, when O has two marks in a line and the third cell of that line is empty, choose that empty cell. Otherwise choose the cell that gives X the best chance of winning.

On the lines board that one change took it from 69.4% to 83.3%, and missed wins from 11% to 2%. It scored 83.8% on the tuning set and 83.3% on the held-out set, so it wasn’t a fluke of the positions I tuned on. My three-move win stopped working.

On the grid and cells boards the new wording did nothing at all (52-54% on the tuning set, against 55-57% before). Telling it to block a line only helps if it can see lines.

I also tried a two-step version: take its own yes/no answers, write them into the state (“X can win this turn: yes”), and ask for the move again. That more than halved the missed blocks, but when its first answer wrongly said “can’t win”, the second call believed it, and missed wins went up. Overall it was a wash on the lines board and worth about four points on the rules board. As a control I fed it the true answers instead, which got to about 88%. So perfect hints are worth four to six points, and its own hints somewhere between nothing and four.

It sees more than it acts on

To separate seeing from deciding, I asked it four plain yes/no questions about every board, with nothing about whose turn it was or what to play: has X completed a line, has O, does X have two in a line with the third cell empty, does O.

It spotted every completed line. It spotted 99% of two-in-a-rows on rows and columns and 94% on diagonals. So with the lines written out, perception is basically fine.

Now compare that with what it did when asked for a move and blocking was the only job. With the vague wording it blocked a column threat 34% of the time - a threat it sees 99% of the time when asked directly. With the clearer wording that rose to 71%, and rows went from 54% to 90%.

I find that gap the most useful thing in the whole exercise. The model had the information, in the state, and could demonstrably read it. It just wasn’t being asked a question that made it use it.

Its mistakes on the yes/no questions were revealing too. They were nearly all false alarms: it said “two in a line with the third cell empty” for 37% of boards where the opponent already held that third cell, and 40% where the line was already complete. That’s the literal reading the docs warn about, in the other direction - it latches on to “two in a line” and goes soft on the condition attached to it.

Same question, same answer

I noticed I could replay a winning line and Jev would make exactly the same moves every time. At first that looked like a weakness. I think it’s the opposite: it’s a decision component, and making the same decision from the same information is what it’s for.

It isn’t strictly deterministic - the probabilities wobble by a few hundredths between identical requests - so I asked 300 positions five times each.

When it gave its pick 60% or more, the pick was identical all five times in every one of 193 positions. It only wavered where its own probability already said “close call”. Relatedly, it turned out to be under-confident rather than over-confident: on the rules board, moves it gave 80% or more were right 99% of the time.

That’s a practical rule you can build on: trust the confident answers, and send the rest somewhere else - more context, a second question, some code, or a person.

Where it runs out

About 8% of decision positions stayed wrong whatever I did, in every version that could see the lines. They’re nearly all forks and quiet early moves - cases where the right answer depends on imagining the reply to your move. A model that answers in one pass with no scratchpad can’t do that, and no wording fixed it. The “how does this end?” question was the same story: it answered “draw” about 80% of the time and did worse than always guessing “win”.

Out of curiosity we tried handing the look-ahead to code. The code tries each move and the replies to it, but knows nothing about lines - for every board it reaches, it uses Jev’s stored yes/no answers. That got to about 94% best moves with three moves of look-ahead. I couldn’t beat it by hand.

But that isn’t really a test of Jev any more. The decisions are the code’s. And it turned out a win for X still exists from six of the nine opening moves, because a search actively hunts for the one board in a hundred that Jev misjudged. In one line it skips an obvious block because Jev gave 0.56 to “O has completed a line” on a board where O only had two. 94% per position is not 94% per game when your opponent gets to choose the positions.

What I took from it

None of this is specific to noughts and crosses, and I’d be wary of reading the exact numbers across to anything else. But the shape of it matches what I’ve seen using Jev on real data:

• The representation is the first lever. If the thing it needs to notice isn’t adjacent in the text, restructure the state until it is. Code is good at this and it’s cheap.
• Say what you mean in the question. “Best” cost 14 points. If you find yourself explaining what you meant when it gets one wrong, that explanation belonged in the instruction.
• Seeing and acting are separate. It can know something in one answer and not use it in another. Ask narrow questions, and let code join them up.
• Use the probabilities. They’re honest. Confident answers were reliable and repeatable; unsure ones were the ones that flipped.
• Don’t ask it to look ahead. That’s what code, or a reasoning model, is for.

The whole thing - every benchmark run, the reruns, the repeats, 36,725 requests in all - cost $1.09, and a request takes about 100 ms of model time plus the network. At that price you can afford to measure rather than guess, which is really the point. If you’ve got a Jev key, the repo has everything needed to rerun it or to try your own wording against the solver. I’d be interested to hear if anyone gets a single question past 84%.

Where I actually use it

The reason I wanted to understand Jev better is that it’s already doing real work in AI Data Stream, my tool for asking questions about your analytics in plain English using your own AI keys. There, Jev sorts the referral and unassigned traffic that GA4 gives up on into sensible channels - including spotting visits that came from AI assistants - and it only overrules GA4 when it’s confident. That’s the same lesson as the noughts and crosses: give it the right information, ask a narrow question, and let the probabilities decide what happens next.

If you’d like to see your own GA4, Search Console and ads data explained rather than just charted, there’s a free tier to try.