A quantitative comparison of the Open Sage evaluation engine — which powers Backgammon Sage Pro — against eXtreme Gammon's bot engine (XG).
We compare the Open Sage bot engine against eXtreme Gammon (XG) across matched evaluation levels — direct neural-network evaluation, multi-ply lookahead, and truncated rollouts. Because XG exposes no programmatic interface, we score it through its Batch Analysis feature. We use three complementary methods. The first constructs a layered set of ground-truth evaluations — trusting fast evaluations where the choice is clear and escalating close calls to full rollouts — and scores each engine as a Performance Rating (PR) against that reference, across both 500 money games (17,535 decisions) and 130 five-point matches (18,292 decisions). For the money games we then rebuild that entire comparison against XG's own tiered analysis as the reference — the exact mirror. The second method isolates the money positions on which Sage 3T and XG Roller++ genuinely disagree and adjudicates each against both engines' full rollouts. The same reference-and-score method is applied to thirteen benchmarks of backgames, containment games and the snake — the positions engines have historically played worst. Across the strength studies the two engines are close: scored against Sage's reference, Sage is ahead at the matched truncated-rollout levels most users rely on; scored against XG's, XG is ahead or level. The third method turns the question around: on 290 real tournament matches already analyzed in XG, we re-analyze each in Sage and compare the Performance Rating the two engines assign to every player. The two ratings track each other very closely: across 580 player ratings they have a 98% correlation, and the average difference between them is +0.002 PR — statistically indistinguishable from zero.
eXtreme Gammon (XG) is a standard reference for computer backgammon analysis, and its evaluation engine is widely considered one of the strongest available. Any new engine that aims to be taken seriously has to answer a simple question: how does it compare to XG? This study sets out to answer that quantitatively for Open Sage, the neural-network engine behind Backgammon Sage Pro.
Our goal was to compare Open Sage's evaluations against XG's at comparable levels of effort. The natural experiment — having the engines play many millions of games head-to-head and tallying the result — is not practical: the XG desktop app exposes no programmatic interface, so feeding positions and moves between the two engines is a manual, click-through process. Reaching a sample size large enough to resolve the small differences between two strong engines would take far too long.
Instead, we use XG's Batch Analysis feature, which can score hundreds of transcribed games in a single unattended pass. Open Sage plays both sides of many games; each game is exported in a standard text format that XG can import; XG analyzes the lot; and we parse its verdicts back out. On top of that shared foundation we apply two distinct methods, described in Section 3 and Section 4.
A third study (Section 5) sets the simulations aside altogether and works from real tournament matches analyzed in XG. Its purpose is different: rather than asking which engine is stronger, it asks whether a player who analyzes a match in Sage gets the same Performance Rating that XG would give — so that years of intuition about what a PR means in XG carry straight over to Sage.
Both engines can evaluate a position at a range of depths, trading accuracy for speed. Understanding these levels is essential to reading the results, because the comparison is always between matched levels of the two engines.
Direct evaluation (1-ply). The raw output of the neural network for a position, with no lookahead. This is the fastest setting and the foundation every deeper level is built on. (We follow XG's convention, in which 1-ply denotes the raw network evaluation.)
Multi-ply lookahead (2-, 3-, 4-ply). The engine looks several turns into the future, considering the opponent's replies and averaging over the possible dice rolls at each step, with a raw network evaluation at the end of each line. Each additional ply is more accurate and substantially more expensive.
Truncated rollouts (1T, 2T, 3T). Rather than searching to a fixed depth, the engine plays out many short simulated games, truncates them after a few turns, and evaluates the resulting position with multi-ply lookahead. Variance reduction cancels out much of the luck in the sampled dice. These are XG's "Roller" settings, and they are stronger than fixed-depth search while remaining far cheaper than a full rollout.
Full rollout. Many simulated games played all the way to completion. Run at sufficient volume with variance reduction, a full rollout is the closest thing to ground truth that a backgammon engine can produce, and we use it as the reference standard throughout this study.
| Family | Sage | XG equivalent | What it does |
|---|---|---|---|
| Direct | 1-ply | 1-ply (raw eval) | A single neural-network evaluation of the position — no lookahead. |
| Multi-ply | 2P · 3P · 4P | 2-ply · 3-ply · 4-ply | Searches the opponent's replies several turns deep, averaging over the dice. |
| Truncated rollout | 1T · 2T · 3T | Roller · Roller+ · Roller++ | Many short simulated games, truncated after 5–7 turns and evaluated with multi-ply, with variance reduction. 3T/Roller++ use 3-ply decisions; 2T/Roller+ use 2-ply; 1T/Roller use 1-ply. |
| Full rollout | Rollout | Rollout | Simulated games played to completion — the reference "truth" in this study. |
The first method scores every engine against a fixed set of decisions whose "true" best play we have established as accurately as possible. It follows the same approach used in the well-known 2012 XG study that compared XG against a number of other bots.
We began by simulating 500 money games of Sage 3P playing itself. A moderately strong level like 3P produces a realistic distribution of positions across every game plan — racing, attacking, priming, anchoring — which is exactly the variety we want to test against. We then ran the whole study a second time for 130 five-point matches, where the score on the board changes the value of every decision; results for both appear below.
Establishing the true best play for every decision by full rollout would be enormously expensive — and unnecessary, because most decisions are not close. We therefore build the reference in three escalating passes, spending rollout effort only where the choice is genuinely in doubt. The same recipe is applied to both data sets; each pass settles the decisions it can resolve confidently and hands the rest down:
Every decision is evaluated at 3-ply. Where the best move beats the next-best by more than 0.05 equity, the 3P verdict is accepted as truth and the decision is settled.
The remaining decisions are re-evaluated at 3T. Where the margin is now wider than 0.02 equity, the 3T verdict is accepted as truth.
Everything still inside 0.02 is settled by a full Sage rollout: 3P decisions for both checker play and cube actions, run in batches of 1,296 paths until the 95% confidence band on the equity falls under 0.005 — or a ceiling of 20,736 paths (16 × 1,296) is reached. The slowest backgame positions took well over an hour each.
The result is a layered reference in which every decision is resolved at exactly the depth its difficulty demands. We ran this same three-pass recipe over both data sets — the 500 money games and the 130 five-point matches — and the resulting tier compositions are shown alongside each set's results below.
With a layered set of trusted evaluations in hand, scoring any engine is straightforward: we present it with each benchmark decision, record the equity error of the move or cube action it chooses relative to the truth, and report the average error × 500 as a PR. The same total is broken out into checker-play and cube PRs, and further by game plan.
Scoring XG required the extra step described in the introduction. Because XG has
no API, we ran its Batch Analysis over the 500 money-game and 130 match transcripts
using a custom level — 3-ply decisions, upgrading to the level under test wherever it disagreed —
and parsed XG's preferred decision for each position out of the
.xg
files it produces, scoring those against the same reference equities.
Across the 500 money games, the three passes resolved 16,889 positions (17,535 decisions) as follows:
The five matched levels, scored over those 17,535 decisions. Lower PR is better; the stronger engine in each pair is highlighted.
PR for every engine and level tested, broken out by decision type and game plan. Lower is better.
| Engine | PR | Checker | Cube | Pure Race | Racing | Attacking | Priming | Anchoring |
|---|---|---|---|---|---|---|---|---|
| Sage 3T | 0.23 | 0.20 | 0.41 | 0.02 | 0.25 | 0.20 | 0.32 | 0.32 |
| XG Roller++ | 0.34 | 0.33 | 0.38 | 0.04 | 0.44 | 0.25 | 0.41 | 0.48 |
| Sage 2T | 0.27 | 0.24 | 0.43 | 0.02 | 0.32 | 0.20 | 0.40 | 0.36 |
| XG Roller+ | 0.41 | 0.41 | 0.39 | 0.05 | 0.60 | 0.31 | 0.47 | 0.54 |
| Sage 1T | 0.49 | 0.50 | 0.43 | 0.04 | 0.58 | 0.43 | 0.62 | 0.64 |
| XG Roller | 0.53 | 0.54 | 0.48 | 0.05 | 0.64 | 0.45 | 0.71 | 0.67 |
| Sage 4P | 0.43 | 0.42 | 0.52 | 0.08 | 0.54 | 0.39 | 0.45 | 0.60 |
| XG 4-ply | 0.47 | 0.46 | 0.52 | 0.06 | 0.59 | 0.40 | 0.57 | 0.59 |
| Sage 3P | 0.60 | 0.60 | 0.61 | 0.13 | 0.78 | 0.53 | 0.67 | 0.75 |
| XG 3-ply | 0.57 | 0.57 | 0.58 | 0.05 | 0.72 | 0.48 | 0.73 | 0.72 |
| Sage 2P | 1.68 | 1.44 | 2.91 | 0.40 | 1.84 | 1.86 | 1.87 | 1.77 |
| Sage 1P | 2.55 | 2.42 | 3.24 | 0.42 | 2.71 | 2.79 | 3.13 | 2.74 |
Every number above grades the engines against Sage's tiered reference. The obvious objection is home-field advantage — Sage is being measured against its own rollouts. So we built the exact mirror: the same 500 games and the same decisions, but with XG's own analysis as the truth at every tier. XG 3-ply settles the 3P-tier decisions, XG Roller++ the 3T-tier decisions, and XG's own full rollout the rolled-out decisions — XG's tier-for-tier analogue of the Sage reference. Every engine is then re-scored against it.
| Engine | PR | Checker | Cube | Pure Race | Racing | Attacking | Priming | Anchoring |
|---|---|---|---|---|---|---|---|---|
| Sage 3T | 0.28 | 0.26 | 0.41 | 0.02 | 0.40 | 0.23 | 0.31 | 0.37 |
| XG Roller++ | 0.21 | 0.20 | 0.23 | 0.01 | 0.30 | 0.18 | 0.22 | 0.25 |
| Sage 2T | 0.30 | 0.29 | 0.36 | 0.02 | 0.39 | 0.28 | 0.36 | 0.38 |
| XG Roller+ | 0.30 | 0.32 | 0.23 | 0.01 | 0.48 | 0.25 | 0.26 | 0.40 |
| Sage 1T | 0.46 | 0.47 | 0.42 | 0.03 | 0.61 | 0.42 | 0.50 | 0.57 |
| XG Roller | 0.40 | 0.41 | 0.35 | 0.03 | 0.53 | 0.34 | 0.53 | 0.45 |
| Sage 4P | 0.39 | 0.38 | 0.43 | 0.06 | 0.55 | 0.36 | 0.42 | 0.42 |
| XG 4-ply | 0.34 | 0.33 | 0.40 | 0.03 | 0.46 | 0.33 | 0.41 | 0.36 |
| Sage 3P | 0.50 | 0.49 | 0.56 | 0.08 | 0.67 | 0.47 | 0.56 | 0.56 |
| XG 3-ply | 0.41 | 0.41 | 0.40 | 0.03 | 0.54 | 0.36 | 0.52 | 0.46 |
| Sage 2P | 1.38 | 1.17 | 2.44 | 0.30 | 1.57 | 1.50 | 1.55 | 1.45 |
| Sage 1P | 2.19 | 2.07 | 2.81 | 0.36 | 2.27 | 2.44 | 2.71 | 2.30 |
The match set runs the same three passes, with one addition: each player's away-score and the Crawford flag are threaded through every evaluation, so the truth is computed in match-equity (MWC) space against the correct score. Across the 130 five-point matches, the passes resolved 17,892 positions (18,292 decisions) as follows:
The five matched levels, scored over those 18,292 decisions. Lower PR is better; the stronger engine in each pair is highlighted.
PR for every engine and level tested on the 5-point match set, broken out by decision type and game plan. Lower is better.
| Engine | PR | Checker | Cube | Pure Race | Racing | Attacking | Priming | Anchoring |
|---|---|---|---|---|---|---|---|---|
| Sage 3T | 0.23 | 0.20 | 0.46 | 0.07 | 0.22 | 0.20 | 0.28 | 0.31 |
| XG Roller++ | 0.36 | 0.35 | 0.44 | 0.08 | 0.44 | 0.32 | 0.33 | 0.49 |
| Sage 2T | 0.26 | 0.24 | 0.40 | 0.02 | 0.35 | 0.18 | 0.31 | 0.35 |
| XG Roller+ | 0.44 | 0.44 | 0.44 | 0.09 | 0.54 | 0.44 | 0.39 | 0.55 |
| Sage 1T | 0.47 | 0.47 | 0.47 | 0.06 | 0.55 | 0.45 | 0.55 | 0.56 |
| XG Roller | 0.51 | 0.51 | 0.56 | 0.09 | 0.63 | 0.49 | 0.50 | 0.65 |
| Sage 4P | 0.41 | 0.41 | 0.47 | 0.05 | 0.50 | 0.37 | 0.44 | 0.53 |
| XG 4-ply | 0.46 | 0.45 | 0.58 | 0.09 | 0.59 | 0.44 | 0.42 | 0.60 |
| Sage 3P | 0.56 | 0.54 | 0.71 | 0.05 | 0.73 | 0.55 | 0.59 | 0.63 |
| XG 3-ply | 0.54 | 0.53 | 0.62 | 0.09 | 0.68 | 0.53 | 0.52 | 0.68 |
| Sage 2P | 1.27 | 1.23 | 1.58 | 0.39 | 1.46 | 1.24 | 1.49 | 1.39 |
| Sage 1P | 2.22 | 2.21 | 2.35 | 0.38 | 2.43 | 2.41 | 2.44 | 2.47 |
Ordinary self-play games rarely produce a deep backgame, a containment game or a far-side prime holding a single straggler, so the money and match sets say little about how an engine plays them — and they are the positions where engines have historically been weakest. Thirteen position-family benchmarks measure them directly, each built like the money benchmark — real decisions, each with a rollout-grade reference — but with the decisions drawn from inside the family:
Each family starts from a small set of seed positions. Sage plays unlimited games out of those seeds against itself at 3-ply, every decision that arises while the position still belongs to the family is recorded, and every recorded decision is rolled out to produce its reference — 5,184 paths played to completion, with 3-ply checker and cube decisions for the first three half-moves and 2-ply thereafter, variance reduction on. Candidate-completion passes rolled out every move an engine under test actually chose, so each score below is against a rollout-graded candidate. A cube position where the opponent owns the cube is not a decision for the player on roll and is not scored.
| Benchmark | Decisions | 2-ply | 3-ply | 4-ply | 1T / Roller | 2T / Roller+ | 3T / Roller++ | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Sage | XG | Sage | XG | Sage | XG | Sage | XG | Sage | XG | Sage | XG | ||
| 2-1 backgame | 1,000 | 2.26 | 2.84 | 1.01 | 1.64 | 1.03 | 1.22 | 0.61 | 1.67 | 0.41 | 0.98 | 0.38 | 0.93 |
| 3-1 backgame | 1,001 | 2.22 | 2.22 | 0.88 | 0.95 | 0.67 | 0.72 | 0.66 | 0.91 | 0.34 | 0.59 | 0.32 | 0.48 |
| 3-2 backgame | 1,008 | 2.07 | 2.39 | 0.91 | 1.31 | 0.78 | 0.99 | 0.65 | 0.95 | 0.44 | 0.61 | 0.25 | 0.51 |
| 4-1 backgame | 1,001 | 1.77 | 1.83 | 0.78 | 0.82 | 0.60 | 0.53 | 0.55 | 0.69 | 0.24 | 0.47 | 0.19 | 0.45 |
| 4-2 backgame | 1,000 | 2.16 | 1.96 | 0.73 | 1.05 | 0.69 | 0.55 | 0.63 | 0.79 | 0.36 | 0.61 | 0.27 | 0.43 |
| 5-1 backgame | 1,003 | 1.32 | 1.77 | 0.66 | 0.76 | 0.46 | 0.64 | 0.49 | 0.60 | 0.26 | 0.41 | 0.23 | 0.32 |
| 5-2 backgame | 1,002 | 1.53 | 1.60 | 0.49 | 0.82 | 0.48 | 0.63 | 0.53 | 0.63 | 0.33 | 0.41 | 0.24 | 0.30 |
| 4-3 backgame | 1,002 | 1.69 | 2.13 | 0.74 | 0.80 | 0.55 | 0.58 | 0.52 | 0.76 | 0.24 | 0.55 | 0.19 | 0.44 |
| 5-3 backgame | 1,003 | 1.86 | 2.13 | 1.02 | 0.89 | 0.58 | 0.63 | 0.68 | 0.81 | 0.35 | 0.46 | 0.21 | 0.44 |
| 5-4 backgame | 1,001 | 2.45 | 2.74 | 0.93 | 1.15 | 0.73 | 0.78 | 0.76 | 0.97 | 0.47 | 0.55 | 0.30 | 0.49 |
| Containment | 3,270 | 4.73 | 14.62 | 2.54 | 11.25 | 2.05 | 10.37 | 2.29 | 7.76 | 1.30 | 7.85 | 1.05 | 6.16 |
| Snake | 978 | 21.00 | 39.92 | 21.42 | 36.64 | 23.53 | 36.17 | 11.19 | 32.99 | 8.42 | 32.38 | 5.85 | 32.24 |
| Massive backgame | 2,103 | 4.33 | 5.76 | 2.56 | 4.44 | 2.17 | 3.68 | 1.91 | 3.14 | 1.39 | 2.75 | 1.09 | 2.64 |
.xg exports (see the
Appendix).
The Rollout PR method scores both engines against a shared reference. A sharper, more direct question is this: in realistic play, where do the two engines actually disagree on the best move — and when they do, which one is right?
We answer it on the hardest money decisions from Method One — the rolled-out positions, each of which now carries both a full Sage rollout and a full XG rollout. Among them we find every decision where Sage 3T and XG Roller++ chose differently, and ask which choice each rollout preferred. Having both rollouts is the whole point: every disagreement is judged against Sage's rollout and XG's own, so the answer never rests on trusting a single engine's truth.
Across the rolled-out money positions — the hardest decisions in the study — the two engines disagree on 1,286 checker plays and 80 cube decisions. Each panel shows how often each engine matched the rollout's best decision — scored against XG's own rollout and against Sage's rollout in turn.
The first two methods ask which engine is stronger. A third question is just as important to anyone who uses an engine to study their own play: if you analyze a real match in XG, note your Performance Rating, then analyze the very same match in Sage — how close are the two ratings? A player who has spent years building intuition for what a given PR means in XG should get essentially the same number from Sage.
To test this directly, we took a large collection of real tournament matches that had already been analyzed in XG, re-analyzed every one of them from scratch in Sage, and compared the Performance Rating each engine assigned to each player.
The match files come from three 2026 tournaments, all 7-point matches, generously provided by Máté Fehér — already analyzed in eXtreme Gammon, exactly as a competitive player would study their own games.
290 matches and 580 individual player ratings in all. One further match was set aside as a corrupted transcription.
Pooling all 580 player ratings, the two engines agree almost exactly. The average difference is statistically indistinguishable from zero, and the spread of that difference is small next to the spread in PR itself.
| Per-player Performance Rating | XG | Sage | DifferenceSage − XG |
|---|---|---|---|
| Average | 4.36 | 4.36 | +0.002 |
| Standard deviation | 2.08 | 2.10 | 0.37 |
| 95% range | 1.52 – 9.36 | 1.44 – 9.67 | −0.76 – +0.74 |
Across the strength studies, Open Sage 3T and XG Roller++ are two of the strongest backgammon evaluations available, and close to each other. The Rollout PR study — run for both money play and 5-point matches — finds Sage stronger at every matched level except 3-ply, where the two engines are within noise of each other, with the clearest edge at the truncated-rollout levels. When the money-game study is rebuilt with XG's own analysis as the reference, the ranking turns around: XG is ahead at every matched level except 2T, where the two are level. The disputed-position study, which looks at positions where the two engines disagree at the strongest evaluation level and scores them against both engines' rollouts, is level on checker play — each rollout favouring the engine that ran it — with cube disagreements too rare and too split to call.
The honest summary, then, is that the two engines play at a very similar, very high level, and that each scores best against its own analysis: by Sage's reference its truncated rollouts get right decisions that XG's Roller levels miss, by XG's the reverse holds, and on the positions where the two engines disagree outright neither engine's rollout puts the other clearly behind. The family benchmarks — backgames, containment games and the snake — are where the two engines separate: measured to the same standard and at every matched level, Sage leads on all thirteen, and by a factor of three or more on containment games and the snake.
And the third study answers the question a Sage user actually faces: across 290 real tournament matches already rated by XG, Sage produces essentially the same Performance Rating — the average difference between a player's Sage and XG rating is statistically indistinguishable from zero, and the spread of that difference is small next to the spread in PR itself. Whether the test is strength against a rolled-out truth or simple agreement on how a real game was played, Open Sage and XG land in the same place.
Open Sage is released as open source, and the complete pipeline behind both
methods ships in the engine's repository. Everything below runs against the
bgsage package and a local build of its engine alone — no
external services, datasets, or infrastructure are required. The only manual
dependency is eXtreme Gammon itself, used to regenerate the XG columns, since XG
has no programmatic interface.
bgbot_cpp extension) following the repository's instructions
for your platform. Reproducing any XG column additionally requires eXtreme Gammon
(Windows) for its Batch Analysis step. Every script resolves its paths inside the
repository, so run them from the bgsage/ root.
You don't have to drive the pipeline by hand. Claude Code — Anthropic's agentic command-line coding tool — can build the engine and run the benchmarks for you. Point it at a checkout of the repository and describe what you want; it works through the same steps detailed below:
data/money_benchmark/benchmark.json.gz and print it as a table I can compare to the study."./mybot — for each decision in the shipped benchmark, evaluate it with my bot and compute its PR against the reference equities."
The benchmark — the layered set of trusted evaluations — is built and scored by a
single script, scripts/benchmark_money.py. There are two
ways in.
The repository ships the assembled benchmark as
data/money_benchmark/benchmark.json.gz (the uncompressed
JSON exceeds GitHub's file-size limit; the scripts read the gzip transparently).
Scoring any Sage level reproduces its row in the Section 3 table:
# lower PR is better python scripts/benchmark_money.py score --level 3ply # Sage 3P python scripts/benchmark_money.py score --level truncated3 # Sage 3T # --level: 1ply 2ply 3ply 4ply | truncated1/2/3 | rollout
To score a different engine entirely, apply the same rule the script uses internally: for every decision in the dataset, take your engine's choice and compare its equity to the stored reference ("rollout") equity — the mean error × 500 is its PR.
Three adaptive passes re-create the dataset. The games are seeded (500 games from seed 1), so the decisions and reference equities come out the same:
# 1. simulate 500 Sage-3P self-play games (+ XG transcripts) python scripts/benchmark_money.py build --stages pass1 --n-games 500 --workers 6 # 2. re-evaluate every decision within 0.05 at 3T python scripts/benchmark_money.py build --stages pass2 --n-threads 16 # 3. roll out every decision still within 0.02 python scripts/benchmark_money.py build --stages pass3 --n-threads 16
Pass 3 writes data/money_benchmark/benchmark.json
(and its .gz); score it exactly as in Option A.
Omitting --stages runs all three passes in order.
XG has no API, so its results come from XG's Batch Analysis. Pass 1 writes
XG-import transcripts to data/money_benchmark/xg/ — these
are not shipped, so regenerate them with pass 1 if you started from the
dataset alone.
xg/ folder with a custom level —
3-ply decisions, upgrading to the level under test on disagreements — with
Save Games after analyze checked. XG writes one
.xg per game.
python scripts/benchmark_pr_xg_levels_all.py --benchmark money,
which reads XG's top decision per position at every analysed level from the
.xg files and scores it against the same reference equities,
printing the same PR breakdown and recording XG's picks beside it.
The XG-reference money table in Section 3 — and the
disputed-position study — instead need XG's rollouts of the hardest
positions. Run XG's Batch Rollout on those positions (Save Games
checked); python scripts/xg_benchmark_report.py parses them.
The XG-reference table then re-scores every level from the picks each scorer recorded.
See Method Two below.
# XG-reference table: harvest XG Roller++ (the 3T tier), then re-score every level; # Sage's picks come from benchmark_money.py score --level <level> --model stage11 python scripts/xg_harvest_results.py --benchmark money --set rollerpp python scripts/benchmark_pr_xg_reference_all.py --sage-suffix stage11
The 5-point match study is reproduced the same way with
scripts/benchmark_match.py, the match-play twin of
benchmark_money.py — the match length and number of matches
are arguments, and the match score is threaded through every evaluation. Its ground
truth ships as data/match_benchmark/5pt/benchmark.json.gz.
# score against the shipped match dataset (no rebuild) python scripts/benchmark_match.py score --match-length 5 --level truncated3 # or rebuild: 130 seeded 5-pt matches, three adaptive passes python scripts/benchmark_match.py build --match-length 5 --n-matches 130 --stages pass1 --workers 6 python scripts/benchmark_match.py build --match-length 5 --n-matches 130 --stages pass2 --n-threads 16 python scripts/benchmark_match.py build --match-length 5 --n-matches 130 --stages pass3 --n-threads 16 # XG columns: batch-analyze data/match_benchmark/5pt/xg/ per level, then python scripts/benchmark_pr_xg_levels_all.py --benchmark match --match-length 5
The disagreement study draws from the same money benchmark built for Method One, cross-referenced against XG's own full rollout of the hardest positions. The only manual dependency is XG's Batch Rollout; a single script does the rest.
data/money_benchmark/xg/.
.xg files.
python scripts/xg_benchmark_report.py python scripts/xg_dispute_analysis.py --sage-picks data/money_benchmark/scores/sage_truncated3.picks.jsonl
The first parses XG's rollouts into
data/money_benchmark/xg_results/rollout.jsonl; the second
prints the disputed-position report — every disagreement scored against the Sage and
XG rollouts in turn — from Sage 3T's recorded picks.
Match play is not yet covered by this method: it needs an XG full rollout of the match positions, which we have not run. Match strength is covered instead by the Rollout PR study (Section 3) and the real-match agreement study (Section 5).
Sage and XG can also play each other directly, with XG's moves taken from its own
analysis. Sage plays both sides of an unlimited game; the transcript is batch-analysed
in XG; the first opponent decision that differs from XG's top choice is replaced by
XG's move and the game replayed from there with fresh dice, until every opponent move
is the one XG would have made. The runner plays a series of such games (seeds continue
from logs/sage_vs_xg.txt) and prints the mean points per game
with its standard error:
python scripts/run_sage_vs_xg_games.py 100 --level 3P # Sage 3P vs XG; levels 1P-4P, 1T-3T python scripts/play_sage_vs_xg.py 3P --seed 7 # one game, iterations under logs/play_sage_vs_xg/
XG must be running on the Windows desktop: every iteration drives its Batch Analyze
dialog through scripts/xg_batch_analyze.py, an Anthropic
Computer Use agent (set ANTHROPIC_API_KEY), which takes the
mouse and keyboard for about a minute per iteration. A game converges in a handful of
iterations; XG's "too good to double" verdict counts as no double.
The thirteen family benchmarks live under
backgame_ref_positions/benchmark/ — one
<family> starting.txt of seed positions and one
<family> rollout.jsonl reference each — and are scored by
scripts/score_backgame_pr.py, which reports the PR beside the
share of picks that are rollout-graded.
python scripts/score_backgame_pr.py --category "21 backgame" --level truncated3 python scripts/score_backgame_pr.py --category containment --level 3ply # XG: export every decision as a one-decision native .xg game, then per level (XG running): analyse, wait, score python scripts/export_folder_benchmark_xg.py generate python scripts/xg_folder_batch.py stamp python scripts/xg_folder_batch.py analyze --level xg3ply # drives XG's Batch Analyze dialog (pywinauto) python scripts/xg_folder_batch.py wait --level xg3ply python scripts/xg_folder_batch.py score --level xg3ply
.xg files.