A quantitative comparison of the Open Sage evaluation engine — which powers Backgammon Sage Pro — against eXtreme Gammon's bot engine (XG).
We compare the Open Sage bot engine against eXtreme Gammon (XG) across matched evaluation levels — direct neural-network evaluation, multi-ply lookahead, and truncated rollouts. Because XG exposes no programmatic interface, we score it through its Batch Analysis feature. We use three complementary methods. The first constructs a layered set of ground-truth evaluations — trusting fast evaluations where the choice is clear and escalating close calls to full rollouts — and scores each engine as a Performance Rating (PR) against that reference, across both 500 money games (17,535 decisions) and 130 five-point matches (18,292 decisions). For the money games we then rebuild that entire comparison against XG's own tiered analysis as the reference — the exact mirror — and Sage's truncated rollout stays ahead even when judged by XG's own numbers. The second method isolates the money positions on which Sage 3T and XG Roller++ genuinely disagree and adjudicates each against both engines' full rollouts. Across both methods the two engines are close, with Sage modestly ahead at the matched truncated-rollout levels most users rely on. The third method turns the question around: on 290 real tournament matches already analyzed in XG, we re-analyze each in Sage and compare the Performance Rating the two engines assign to every player — and find the average difference statistically indistinguishable from zero.
eXtreme Gammon (XG) is a standard reference for computer backgammon analysis, and its evaluation engine is widely considered one of the strongest available. Any new engine that aims to be taken seriously has to answer a simple question: how does it compare to XG? This study sets out to answer that quantitatively for Open Sage, the neural-network engine behind Backgammon Sage Pro.
Our goal was to compare Open Sage's evaluations against XG's at comparable levels of effort. The natural experiment — having the two engines play many thousands of games head-to-head and tallying the result — is not practical: the XG desktop app exposes no programmatic interface, so feeding positions and moves between the two engines is a manual, click-through process. Reaching a sample size large enough to resolve the small differences between two strong engines would take far too long.
Instead, we use XG's Batch Analysis feature, which can score hundreds of transcribed games in a single unattended pass. Open Sage plays both sides of many games; each game is exported in a standard text format that XG can import; XG analyzes the lot; and we parse its verdicts back out. On top of that shared foundation we apply two distinct methods, described in Section 3 and Section 4. A third study (Section 5) sets the simulations aside altogether and works from real tournament matches analyzed in XG.
Both engines can evaluate a position at a range of depths, trading accuracy for speed. Understanding these levels is essential to reading the results, because the comparison is always between matched levels of the two engines.
Direct evaluation (1-ply). The raw output of the neural network for a position, with no lookahead. This is the fastest setting and the foundation every deeper level is built on. (We follow XG's convention, in which 1-ply denotes the raw network evaluation.)
Multi-ply lookahead (2-, 3-, 4-ply). The engine looks several turns into the future, considering the opponent's replies and averaging over the possible dice rolls at each step, with a raw network evaluation at the end of each line. Each additional ply is more accurate and substantially more expensive.
Truncated rollouts (1T, 2T, 3T). Rather than searching to a fixed depth, the engine plays out many short simulated games, truncates them after a few turns, and evaluates the resulting position with multi-ply lookahead. Variance reduction cancels out much of the luck in the sampled dice. These are XG's "Roller" settings, and they are stronger than fixed-depth search while remaining far cheaper than a full rollout.
Full rollout. Many simulated games played all the way to completion. Run at sufficient volume with variance reduction, a full rollout is the closest thing to ground truth that a backgammon engine can produce, and we use it as the reference standard throughout this study.
| Family | Sage | XG equivalent | What it does |
|---|---|---|---|
| Direct | 1-ply | 1-ply (raw eval) | A single neural-network evaluation of the position — no lookahead. |
| Multi-ply | 2P · 3P · 4P | 2-ply · 3-ply · 4-ply | Searches the opponent's replies several turns deep, averaging over the dice. |
| Truncated rollout | 1T · 2T · 3T | Roller · Roller+ · Roller++ | Many short simulated games, truncated after 5–7 turns and evaluated with multi-ply, with variance reduction. 3T/Roller++ use 3-ply decisions; 2T/Roller+ use 2-ply; 1T/Roller use 1-ply. |
| Full rollout | Rollout | Rollout | Simulated games played to completion — the reference "truth" in this study. |
The first method scores every engine against a fixed set of decisions whose "true" best play we have established as accurately as possible. It follows the same approach used in the well-known 2012 XG study that compared XG against a number of other bots.
We began by simulating 500 money games of Sage 3P playing itself. A moderately strong level like 3P produces a realistic distribution of positions across every game plan — racing, attacking, priming, anchoring — which is exactly the variety we want to test against. We then ran the whole study a second time for 130 five-point matches, where the score on the board changes the value of every decision; results for both appear below.
Establishing the true best play for every decision by full rollout would be enormously expensive — and unnecessary, because most decisions are not close. We therefore build the reference in three escalating passes, spending rollout effort only where the choice is genuinely in doubt. The same recipe is applied to both data sets; each pass settles the decisions it can resolve confidently and hands the rest down:
Every decision is evaluated at 3-ply. Where the best move beats the next-best by more than 0.05 equity, the 3P verdict is accepted as truth and the decision is settled.
The remaining decisions are re-evaluated at 3T. Where the margin is now wider than 0.02 equity, the 3T verdict is accepted as truth.
Everything still inside 0.02 is settled by a full Sage rollout: 3P decisions for both checker play and cube actions, run in batches of 1,296 paths until the 95% confidence band on the equity falls under 0.005 — or a ceiling of 20,736 paths (16 × 1,296) is reached. The slowest back-game positions took well over an hour each.
The result is a layered reference in which every decision is resolved at exactly the depth its difficulty demands. We ran this same three-pass recipe over both data sets — the 500 money games and the 130 five-point matches — and the resulting tier compositions are shown alongside each set's results below.
With a layered set of trusted evaluations in hand, scoring any engine is straightforward: we present it with each benchmark decision, record the equity error of the move or cube action it chooses relative to the truth, and report the average error × 500 as a PR. The same total is broken out into checker-play and cube PRs, and further by game plan.
Scoring XG required the extra step described in the introduction. Because XG has
no API, we ran its Batch Analysis over the 500 money-game and 130 match transcripts
using a custom level — 3-ply decisions, upgrading to the level under test wherever it disagreed —
and parsed XG's preferred decision for each position out of the
.xg
files it produces, scoring those against the same reference equities.
Across the 500 money games, the three passes resolved 16,889 positions (17,535 decisions) as follows:
The five matched levels, scored over those 17,535 decisions. Lower PR is better; the stronger engine in each pair is highlighted.
PR for every engine and level tested, broken out by decision type and game plan. Lower is better.
| Engine | PR | Checker | Cube | Pure Race | Racing | Attacking | Priming | Anchoring |
|---|---|---|---|---|---|---|---|---|
| Sage 3T | 0.21 | 0.18 | 0.36 | 0.02 | 0.24 | 0.17 | 0.31 | 0.28 |
| XG Roller++ | 0.32 | 0.31 | 0.38 | 0.04 | 0.40 | 0.24 | 0.41 | 0.44 |
| Sage 2T | 0.26 | 0.23 | 0.44 | 0.02 | 0.32 | 0.21 | 0.39 | 0.30 |
| XG Roller+ | 0.41 | 0.41 | 0.39 | 0.05 | 0.59 | 0.31 | 0.47 | 0.54 |
| Sage 1T | 0.50 | 0.52 | 0.40 | 0.04 | 0.57 | 0.43 | 0.59 | 0.73 |
| XG Roller | 0.53 | 0.54 | 0.48 | 0.05 | 0.63 | 0.44 | 0.71 | 0.66 |
| Sage 4P | 0.41 | 0.39 | 0.50 | 0.07 | 0.51 | 0.37 | 0.45 | 0.53 |
| XG 4-ply | 0.46 | 0.45 | 0.52 | 0.06 | 0.58 | 0.40 | 0.57 | 0.58 |
| Sage 3P | 0.58 | 0.58 | 0.57 | 0.14 | 0.72 | 0.52 | 0.63 | 0.74 |
| XG 3-ply | 0.57 | 0.57 | 0.58 | 0.05 | 0.71 | 0.48 | 0.73 | 0.71 |
| Sage 2P | 1.64 | 1.39 | 2.88 | 0.40 | 1.77 | 1.83 | 1.86 | 1.71 |
| Sage 1P | 2.59 | 2.48 | 3.20 | 0.78 | 2.60 | 2.79 | 3.10 | 2.89 |
Every number above grades the engines against Sage's tiered reference. The obvious objection is home-field advantage — Sage is being measured against its own rollouts. So we built the exact mirror: the same 500 games and the same decisions, but with XG's own analysis as the truth at every tier. XG 3-ply settles the 3P-tier decisions, XG Roller++ the 3T-tier decisions, and XG's own full rollout the rolled-out decisions — XG's tier-for-tier analogue of the Sage reference. Every engine is then re-scored against it.
| Engine | PR | Checker | Cube | Pure Race | Racing | Attacking | Priming | Anchoring |
|---|---|---|---|---|---|---|---|---|
| Sage 3T | 0.26 | 0.25 | 0.33 | 0.02 | 0.37 | 0.20 | 0.30 | 0.36 |
| XG Roller++ | 0.33 | 0.35 | 0.23 | 0.04 | 0.50 | 0.20 | 0.39 | 0.47 |
| Sage 2T | 0.31 | 0.30 | 0.35 | 0.02 | 0.41 | 0.27 | 0.36 | 0.37 |
| XG Roller+ | 0.38 | 0.40 | 0.23 | 0.04 | 0.58 | 0.26 | 0.39 | 0.54 |
| Sage 1T | 0.46 | 0.48 | 0.37 | 0.03 | 0.58 | 0.40 | 0.50 | 0.64 |
| XG Roller | 0.47 | 0.49 | 0.35 | 0.12 | 0.61 | 0.35 | 0.54 | 0.62 |
| Sage 4P | 0.37 | 0.36 | 0.42 | 0.06 | 0.53 | 0.34 | 0.40 | 0.39 |
| XG 4-ply | 0.45 | 0.46 | 0.40 | 0.13 | 0.60 | 0.36 | 0.48 | 0.59 |
| Sage 3P | 0.49 | 0.48 | 0.52 | 0.08 | 0.66 | 0.46 | 0.54 | 0.55 |
| XG 3-ply | 0.47 | 0.49 | 0.40 | 0.12 | 0.63 | 0.37 | 0.54 | 0.61 |
| Sage 2P | 1.35 | 1.14 | 2.40 | 0.31 | 1.50 | 1.48 | 1.52 | 1.41 |
| Sage 1P | 2.24 | 2.14 | 2.76 | 0.74 | 2.21 | 2.44 | 2.69 | 2.39 |
The match set runs the same three passes, with one addition: each player's away-score and the Crawford flag are threaded through every evaluation, so the truth is computed in match-equity (MWC) space against the correct score. Across the 130 five-point matches, the passes resolved 17,892 positions (18,292 decisions) as follows:
The five matched levels, scored over those 18,292 decisions. Lower PR is better; the stronger engine in each pair is highlighted.
PR for every engine and level tested on the 5-point match set, broken out by decision type and game plan. Lower is better.
| Engine | PR | Checker | Cube | Pure Race | Racing | Attacking | Priming | Anchoring |
|---|---|---|---|---|---|---|---|---|
| Sage 3T | 0.19 | 0.16 | 0.40 | 0.01 | 0.17 | 0.18 | 0.25 | 0.26 |
| XG Roller++ | 0.35 | 0.35 | 0.41 | 0.08 | 0.44 | 0.32 | 0.33 | 0.45 |
| Sage 2T | 0.26 | 0.24 | 0.44 | 0.02 | 0.36 | 0.18 | 0.34 | 0.33 |
| XG Roller+ | 0.44 | 0.44 | 0.41 | 0.09 | 0.55 | 0.44 | 0.40 | 0.52 |
| Sage 1T | 0.49 | 0.49 | 0.52 | 0.05 | 0.57 | 0.44 | 0.57 | 0.63 |
| XG Roller | 0.51 | 0.51 | 0.54 | 0.09 | 0.64 | 0.50 | 0.51 | 0.62 |
| Sage 4P | 0.42 | 0.42 | 0.48 | 0.05 | 0.52 | 0.38 | 0.45 | 0.55 |
| XG 4-ply | 0.46 | 0.45 | 0.56 | 0.09 | 0.60 | 0.44 | 0.42 | 0.56 |
| Sage 3P | 0.57 | 0.56 | 0.67 | 0.05 | 0.74 | 0.56 | 0.58 | 0.68 |
| XG 3-ply | 0.54 | 0.53 | 0.61 | 0.09 | 0.69 | 0.53 | 0.52 | 0.65 |
| Sage 2P | 1.30 | 1.26 | 1.60 | 0.39 | 1.59 | 1.23 | 1.50 | 1.41 |
| Sage 1P | 2.30 | 2.29 | 2.31 | 0.69 | 2.48 | 2.43 | 2.47 | 2.56 |
The Rollout PR method scores both engines against a shared reference. A sharper, more direct question is this: in realistic play, where do the two engines actually disagree on the best move — and when they do, which one is right?
We answer it on the hardest money decisions from Method One — the rolled-out positions, each of which now carries both a full Sage rollout and a full XG rollout. Among them we find every decision where Sage 3T and XG Roller++ chose differently, and ask which choice each rollout preferred. Having both rollouts is the whole point: every disagreement is judged against Sage's rollout and XG's own, so the answer never rests on trusting a single engine's truth.
Across the rolled-out money positions — the hardest decisions in the study — the two engines disagree on 1,488 checker plays and 69 cube decisions. Each panel shows how often each engine matched the rollout's best decision — scored against XG's own rollout and against Sage's rollout in turn.
The first two methods ask which engine is stronger. A third question is just as important to anyone who uses an engine to study their own play: if you analyze a real match in XG, note your Performance Rating, then analyze the very same match in Sage — how close are the two ratings? A player who has spent years building intuition for what a given PR means in XG should get essentially the same number from Sage.
To test this directly, we took a large collection of real tournament matches that had already been analyzed in XG, re-analyzed every one of them from scratch in Sage, and compared the Performance Rating each engine assigned to each player.
The match files come from three 2026 tournaments, all 7-point matches, generously provided by Máté Fehér — already analyzed in eXtreme Gammon, exactly as a competitive player would study their own games.
290 matches and 580 individual player ratings in all. One further match was set aside as a corrupted transcription.
Pooling all 580 player ratings, the two engines agree almost exactly. The average difference is statistically indistinguishable from zero, and the spread of that difference is small next to the spread in PR itself.
| Per-player Performance Rating | XG | Sage | DifferenceSage − XG |
|---|---|---|---|
| Average | 4.36 | 4.36 | +0.002 |
| Standard deviation | 2.08 | 2.10 | 0.37 |
| 95% range | 1.52 – 9.36 | 1.44 – 9.67 | −0.76 – +0.74 |
Across the two strength studies, Open Sage 3T and XG Roller++ are roughly comparable — two of the strongest backgammon evaluations available, and close to each other. The Rollout PR study — run for both money play and 5-point matches — finds Sage stronger at every matched level except 3-ply, where the two engines are within noise of each other; and on the money games it holds up even when the reference is switched to XG's own tiered analysis. The disputed-position study, which looks at positions where the two engines disagree at the strongest evaluation level (Sage 3T and XG Roller++) and scores them against both engines' rollouts, finds Sage 3T the closer engine on checker disagreements against either rollout, with cube disagreements too rare and too split to call.
There is, then, some evidence that Sage 3T is slightly stronger than XG Roller++. The most natural objection — that the edge is an artifact of grading Sage against its own rollouts — is answered where we can test it: scored against XG's own reference, Sage's truncated levels stay ahead. The differences are nonetheless small, and the honest summary is that the two engines play at a very similar, very high level — which is precisely the bar we set out to meet.
And the third study answers the question a Sage user actually faces: across 290 real tournament matches already rated by XG, Sage produces essentially the same Performance Rating — the average difference between a player's Sage and XG rating is statistically indistinguishable from zero, and the spread of that difference is small next to the spread in PR itself. Whether the test is strength against a rolled-out truth or simple agreement on how a real game was played, Open Sage and XG land in the same place.
Open Sage is released as open source, and the complete pipeline behind both
methods ships in the engine's repository. Everything below runs against the
bgsage package and a local build of its engine alone — no
external services, datasets, or infrastructure are required. The only manual
dependency is eXtreme Gammon itself, used to regenerate the XG columns, since XG
has no programmatic interface.
bgbot_cpp extension) following the repository's instructions
for your platform. Reproducing any XG column additionally requires eXtreme Gammon
(Windows) for its Batch Analysis step. Every script resolves its paths inside the
repository, so run them from the bgsage/ root.
You don't have to drive the pipeline by hand. Claude Code — Anthropic's agentic command-line coding tool — can build the engine and run the benchmarks for you. Point it at a checkout of the repository and describe what you want; it works through the same steps detailed below:
data/money_benchmark/benchmark.json.gz and print it as a table I can compare to the study."./mybot — for each decision in the shipped benchmark, evaluate it with my bot and compute its PR against the reference equities."
The benchmark — the layered set of trusted evaluations — is built and scored by a
single script, scripts/benchmark_money.py. There are two
ways in.
The repository ships the assembled benchmark as
data/money_benchmark/benchmark.json.gz (the uncompressed
JSON exceeds GitHub's file-size limit; the scripts read the gzip transparently).
Scoring any Sage level reproduces its row in the Section 3 table:
# lower PR is better python scripts/benchmark_money.py score --level 3ply # Sage 3P python scripts/benchmark_money.py score --level truncated3 # Sage 3T # --level: 1ply 2ply 3ply 4ply | truncated1/2/3 | rollout
To score a different engine entirely, apply the same rule the script uses internally: for every decision in the dataset, take your engine's choice and compare its equity to the stored reference ("rollout") equity — the mean error × 500 is its PR.
Three adaptive passes re-create the dataset. The games are seeded (500 games from seed 1), so the decisions and reference equities come out the same:
# 1. simulate 500 Sage-3P self-play games (+ XG transcripts) python scripts/benchmark_money.py build --stages pass1 --n-games 500 --workers 6 # 2. re-evaluate every decision within 0.05 at 3T python scripts/benchmark_money.py build --stages pass2 --n-threads 16 # 3. roll out every decision still within 0.02 python scripts/benchmark_money.py build --stages pass3 --n-threads 16
Pass 3 writes data/money_benchmark/benchmark.json
(and its .gz); score it exactly as in Option A.
Omitting --stages runs all three passes in order.
XG has no API, so its results come from XG's Batch Analysis. Pass 1 writes
XG-import transcripts to data/money_benchmark/xg/ — these
are not shipped, so regenerate them with pass 1 if you started from the
dataset alone.
xg/ folder with a custom level —
3-ply decisions, upgrading to the level under test on disagreements — with
Save Games after analyze checked. XG writes one
.xg per game.
python scripts/benchmark_pr_xg.py, which reads XG's
top decision per position from the .xg files and scores
it against the same reference equities, printing the same PR breakdown.
The XG-reference money table in Section 3 — and the
disputed-position study — instead need XG's rollouts of the hardest
positions. Run XG's Batch Rollout on those positions (Save Games
checked), then python scripts/xg_benchmark_report.py parses
them and prints both the XG-reference level table and the disputed-position report.
See Method Two below.
The 5-point match study is reproduced the same way with
scripts/benchmark_match.py, the match-play twin of
benchmark_money.py — the match length and number of matches
are arguments, and the match score is threaded through every evaluation. Its ground
truth ships as data/match_benchmark/5pt/benchmark.json.gz.
# score against the shipped match dataset (no rebuild) python scripts/benchmark_match.py score --match-length 5 --level truncated3 # or rebuild: 130 seeded 5-pt matches, three adaptive passes python scripts/benchmark_match.py build --match-length 5 --n-matches 130 --stages pass1 --workers 6 python scripts/benchmark_match.py build --match-length 5 --n-matches 130 --stages pass2 --n-threads 16 python scripts/benchmark_match.py build --match-length 5 --n-matches 130 --stages pass3 --n-threads 16 # XG columns: batch-analyze data/match_benchmark/5pt/xg/, then python scripts/benchmark_pr_xg_match.py --match-length 5
The disagreement study draws from the same money benchmark built for Method One, cross-referenced against XG's own full rollout of the hardest positions. The only manual dependency is XG's Batch Rollout; a single script does the rest.
data/money_benchmark/xg/.
.xg files.
python scripts/xg_benchmark_report.py
It parses XG's rollouts into
data/money_benchmark/xg_results/rollout.jsonl, then prints
both the evaluation-level PR against the XG rollout (Section 3's second
reference) and the disputed-position report — every disagreement scored against the
Sage and XG rollouts in turn.
Match play is not yet covered by this method: it needs an XG full rollout of the match positions, which we have not run. Match strength is covered instead by the Rollout PR study (Section 3) and the real-match agreement study (Section 5).
.xg files.