XG Roller++ avg error
0.015
A quantitative comparison of the Open Sage evaluation engine — which powers Backgammon Sage Pro — against eXtreme Gammon's bot engine (XG).
We compare the Open Sage bot engine against eXtreme Gammon (XG) across matched evaluation levels — direct neural-network evaluation, multi-ply lookahead, and truncated rollouts. Because XG exposes no programmatic interface, we score it through its Batch Analysis feature. We use three complementary methods. The first constructs a layered set of ground-truth evaluations — trusting fast evaluations where the choice is clear and escalating close calls to full rollouts — and scores each engine as a Performance Rating (PR) against that reference, across both 500 money games (17,535 decisions) and 130 five-point matches (18,292 decisions). The second isolates the positions on which Sage's truncated rollout and XG's strongest setting genuinely disagree, and rolls them out to adjudicate. Across both, the two engines are close, with Sage's truncated rollout (3T) modestly ahead of XG Roller++, most visibly in match play — with the caveat that this ground truth is derived from Sage's own rollouts. The third method turns the question around: on 290 real tournament matches already analyzed in XG, we re-analyze each in Sage and compare the Performance Rating the two engines assign to every player — and find the average difference statistically indistinguishable from zero.
eXtreme Gammon (XG) is a standard reference for computer backgammon analysis, and its evaluation engine is widely considered one of the strongest available. Any new engine that aims to be taken seriously has to answer a simple question: how does it compare to XG? This study sets out to answer that quantitatively for Open Sage, the neural-network engine behind Backgammon Sage Pro.
Our goal was to compare Open Sage's evaluations against XG's at comparable levels of effort. The natural experiment — having the two engines play many thousands of games head-to-head and tallying the result — is not practical: the XG desktop app exposes no programmatic interface, so feeding positions and moves between the two engines is a manual, click-through process. Reaching a sample size large enough to resolve the small differences between two strong engines would take far too long.
Instead, we use XG's Batch Analysis feature, which can score hundreds of transcribed games in a single unattended pass. Open Sage plays both sides of many games; each game is exported in a standard text format that XG can import; XG analyzes the lot; and we parse its verdicts back out. On top of that shared foundation we apply two distinct methods, described in Section 3 and Section 4. A third study (Section 5) sets the simulations aside altogether and works from real tournament matches analyzed in XG.
Both engines can evaluate a position at a range of depths, trading accuracy for speed. Understanding these levels is essential to reading the results, because the comparison is always between matched levels of the two engines.
Direct evaluation (1-ply). The raw output of the neural network for a position, with no lookahead. This is the fastest setting and the foundation every deeper level is built on. (We follow XG's convention, in which 1-ply denotes the raw network evaluation.)
Multi-ply lookahead (2-, 3-, 4-ply). The engine looks several turns into the future, considering the opponent's replies and averaging over the possible dice rolls at each step, with a raw network evaluation at the end of each line. Each additional ply is more accurate and substantially more expensive.
Truncated rollouts (1T, 2T, 3T). Rather than searching to a fixed depth, the engine plays out many short simulated games, truncates them after a few turns, and evaluates the resulting position with multi-ply lookahead. Variance reduction cancels out much of the luck in the sampled dice. These are XG's "Roller" settings, and they are stronger than fixed-depth search while remaining far cheaper than a full rollout.
Full rollout. Many simulated games played all the way to completion. Run at sufficient volume with variance reduction, a full rollout is the closest thing to ground truth that a backgammon engine can produce, and we use it as the reference standard throughout this study.
| Family | Sage | XG equivalent | What it does |
|---|---|---|---|
| Direct | 1-ply | 1-ply (raw eval) | A single neural-network evaluation of the position — no lookahead. |
| Multi-ply | 2P · 3P · 4P | 2-ply · 3-ply · 4-ply | Searches the opponent's replies several turns deep, averaging over the dice. |
| Truncated rollout | 1T · 2T · 3T | Roller · Roller+ · Roller++ | Many short simulated games, truncated after 5–7 turns and evaluated with multi-ply, with variance reduction. 3T/Roller++ use 3-ply decisions; 2T/Roller+ use 2-ply; 1T/Roller use 1-ply. |
| Full rollout | Rollout | Rollout | Simulated games played to completion — the reference "truth" in this study. |
The first method scores every engine against a fixed set of decisions whose "true" best play we have established as accurately as possible. It follows the same approach used in the well-known 2012 XG study that compared XG against a number of other bots.
We began by simulating 500 money games of Sage 3P playing itself. A moderately strong level like 3P produces a realistic distribution of positions across every game plan — racing, attacking, priming, anchoring — which is exactly the variety we want to test against. We then ran the whole study a second time for 130 five-point matches, where the score on the board changes the value of every decision; results for both appear below.
Establishing the true best play for every decision by full rollout would be enormously expensive — and unnecessary, because most decisions are not close. We therefore build the reference in three escalating passes, spending rollout effort only where the choice is genuinely in doubt. The same recipe is applied to both data sets; each pass settles the decisions it can resolve confidently and hands the rest down:
Every decision is evaluated at 3-ply. Where the best move beats the next-best by more than 0.05 equity, the 3P verdict is accepted as truth and the decision is settled.
The remaining decisions are re-evaluated at 3T. Where the margin is now wider than 0.02 equity, the 3T verdict is accepted as truth.
Everything still inside 0.02 is settled by a full Sage rollout: 3P decisions for both checker play and cube actions, run in batches of 1,296 paths until the 95% confidence band on the equity falls under 0.005 — or a ceiling of 20,736 paths (16 × 1,296) is reached. The slowest back-game positions took well over an hour each.
The result is a layered reference in which every decision is resolved at exactly the depth its difficulty demands. We ran this same three-pass recipe over both data sets — the 500 money games and the 130 five-point matches — and the resulting tier compositions are shown alongside each set's results below.
With a layered set of trusted evaluations in hand, scoring any engine is straightforward: we present it with each benchmark decision, record the equity error of the move or cube action it chooses relative to the truth, and report the average error × 500 as a PR. The same total is broken out into checker-play and cube PRs, and further by game plan.
Scoring XG required the extra step described in the introduction. Because XG has
no API, we ran its Batch Analysis over the 500 money-game and 130 match transcripts
using a custom level — 3-ply decisions, upgrading to the level under test wherever it disagreed —
and parsed XG's preferred decision for each position out of the
.xg
files it produces, scoring those against the same reference equities.
Across the 500 money games, the three passes resolved 16,889 positions (17,535 decisions) as follows:
The five matched levels, scored over those 17,535 decisions. Lower PR is better; the stronger engine in each pair is highlighted.
PR for every engine and level tested, broken out by decision type and game plan. Lower is better.
| Engine | PR | Checker | Cube | Pure Race | Racing | Attacking | Priming | Anchoring |
|---|---|---|---|---|---|---|---|---|
| Sage 3T | 0.21 | 0.18 | 0.36 | 0.02 | 0.24 | 0.17 | 0.31 | 0.28 |
| XG Roller++ | 0.32 | 0.31 | 0.38 | 0.04 | 0.40 | 0.24 | 0.41 | 0.44 |
| Sage 2T | 0.26 | 0.23 | 0.44 | 0.02 | 0.32 | 0.21 | 0.39 | 0.30 |
| XG Roller+ | 0.41 | 0.41 | 0.39 | 0.05 | 0.59 | 0.31 | 0.47 | 0.54 |
| Sage 1T | 0.50 | 0.52 | 0.40 | 0.04 | 0.57 | 0.43 | 0.59 | 0.73 |
| XG Roller | 0.53 | 0.54 | 0.48 | 0.05 | 0.63 | 0.44 | 0.71 | 0.66 |
| Sage 4P | 0.41 | 0.39 | 0.50 | 0.07 | 0.51 | 0.37 | 0.45 | 0.53 |
| XG 4-ply | 0.46 | 0.45 | 0.52 | 0.06 | 0.58 | 0.40 | 0.57 | 0.58 |
| Sage 3P | 0.58 | 0.58 | 0.57 | 0.14 | 0.72 | 0.52 | 0.63 | 0.74 |
| XG 3-ply | 0.57 | 0.57 | 0.58 | 0.05 | 0.71 | 0.48 | 0.73 | 0.71 |
| Sage 2P | 1.64 | 1.39 | 2.88 | 0.40 | 1.77 | 1.83 | 1.86 | 1.71 |
| Sage 1P | 2.59 | 2.48 | 3.20 | 0.78 | 2.60 | 2.79 | 3.10 | 2.89 |
The match set runs the same three passes, with one addition: each player's away-score and the Crawford flag are threaded through every evaluation, so the truth is computed in match-equity (MWC) space against the correct score. Across the 130 five-point matches, the passes resolved 17,892 positions (18,292 decisions) as follows:
The five matched levels, scored over those 18,292 decisions. Lower PR is better; the stronger engine in each pair is highlighted.
PR for every engine and level tested on the 5-point match set, broken out by decision type and game plan. Lower is better.
| Engine | PR | Checker | Cube | Pure Race | Racing | Attacking | Priming | Anchoring |
|---|---|---|---|---|---|---|---|---|
| Sage 3T | 0.19 | 0.16 | 0.40 | 0.01 | 0.17 | 0.18 | 0.25 | 0.26 |
| XG Roller++ | 0.35 | 0.35 | 0.41 | 0.08 | 0.44 | 0.32 | 0.33 | 0.45 |
| Sage 2T | 0.26 | 0.24 | 0.44 | 0.02 | 0.36 | 0.18 | 0.34 | 0.33 |
| XG Roller+ | 0.44 | 0.44 | 0.41 | 0.09 | 0.55 | 0.44 | 0.40 | 0.52 |
| Sage 1T | 0.49 | 0.49 | 0.52 | 0.05 | 0.57 | 0.44 | 0.57 | 0.63 |
| XG Roller | 0.51 | 0.51 | 0.54 | 0.09 | 0.64 | 0.50 | 0.51 | 0.62 |
| Sage 4P | 0.42 | 0.42 | 0.48 | 0.05 | 0.52 | 0.38 | 0.45 | 0.55 |
| XG 4-ply | 0.46 | 0.45 | 0.56 | 0.09 | 0.60 | 0.44 | 0.42 | 0.56 |
| Sage 3P | 0.57 | 0.56 | 0.67 | 0.05 | 0.74 | 0.56 | 0.58 | 0.68 |
| XG 3-ply | 0.54 | 0.53 | 0.61 | 0.09 | 0.69 | 0.53 | 0.52 | 0.65 |
| Sage 2P | 1.30 | 1.26 | 1.60 | 0.39 | 1.59 | 1.23 | 1.50 | 1.41 |
| Sage 1P | 2.30 | 2.29 | 2.31 | 0.69 | 2.48 | 2.43 | 2.47 | 2.56 |
The Rollout PR method scores both engines against a shared reference. A second, more direct lens asks: in realistic play, where do the two engines actually disagree on the best move — and when they do, which one is right?
To find out, we simulated games of Sage 3P self-play, had XG analyze them, and isolated the decisions where Sage 3T and XG Roller++ chose differently. The procedure filters carefully so that only genuine 3T-vs-Roller++ disagreements survive:
The first two methods ask which engine is stronger. A third question is just as important to anyone who uses an engine to study their own play: if you analyze a real match in XG, note your Performance Rating, then analyze the very same match in Sage — how close are the two ratings? A player who has spent years building intuition for what a given PR means in XG should get essentially the same number from Sage.
To test this directly, we took a large collection of real tournament matches that had already been analyzed in XG, re-analyzed every one of them from scratch in Sage, and compared the Performance Rating each engine assigned to each player.
The match files come from three 2026 tournaments, all 7-point matches, generously provided by Máté Fehér — already analyzed in eXtreme Gammon, exactly as a competitive player would study their own games.
290 matches and 580 individual player ratings in all. One further match was set aside as a corrupted transcription.
Pooling all 580 player ratings, the two engines agree almost exactly. The average difference is statistically indistinguishable from zero, and the spread of that difference is small next to the spread in PR itself.
| Per-player Performance Rating | XG | Sage | DifferenceSage − XG |
|---|---|---|---|
| Average | 4.36 | 4.36 | +0.002 |
| Standard deviation | 2.08 | 2.10 | 0.37 |
| 95% range | 1.52 – 9.36 | 1.44 – 9.67 | −0.76 – +0.74 |
Across the two strength studies, Open Sage 3T and XG Roller++ are roughly comparable — two of the strongest backgammon evaluations available, and close to each other. The Rollout PR study — run for both money play and 5-point matches — finds Sage stronger at every matched level except 3-ply, where the two engines are within noise of each other. The disputed-position study finds Sage resolving more of the genuine disagreements correctly, with a smaller average error, most clearly in match play.
There is, then, some evidence that Sage 3T is slightly stronger than XG Roller++, especially in matches. But the differences are small, and part of the measured edge may reflect the use of Sage's own rollouts as the reference truth rather than a true difference in strength. The honest summary is that the two engines play at a very similar, very high level — which is precisely the bar we set out to meet.
And the third study answers the question a Sage user actually faces: across 290 real tournament matches already rated by XG, Sage produces essentially the same Performance Rating — the average difference between a player's Sage and XG rating is statistically indistinguishable from zero, and the spread of that difference is small next to the spread in PR itself. Whether the test is strength against a rolled-out truth or simple agreement on how a real game was played, Open Sage and XG land in the same place.
Open Sage is released as open source, and the complete pipeline behind both
methods ships in the engine's repository. Everything below runs against the
bgsage package and a local build of its engine alone — no
external services, datasets, or infrastructure are required. The only manual
dependency is eXtreme Gammon itself, used to regenerate the XG columns, since XG
has no programmatic interface.
bgbot_cpp extension) following the repository's instructions
for your platform. Reproducing any XG column additionally requires eXtreme Gammon
(Windows) for its Batch Analysis step. Every script resolves its paths inside the
repository, so run them from the bgsage/ root.
You don't have to drive the pipeline by hand. Claude Code — Anthropic's agentic command-line coding tool — can build the engine and run the benchmarks for you. Point it at a checkout of the repository and describe what you want; it works through the same steps detailed below:
data/money_benchmark/benchmark.json.gz and print it as a table I can compare to the study."./mybot — for each decision in the shipped benchmark, evaluate it with my bot and compute its PR against the reference equities."
The benchmark — the layered set of trusted evaluations — is built and scored by a
single script, scripts/benchmark_money.py. There are two
ways in.
The repository ships the assembled benchmark as
data/money_benchmark/benchmark.json.gz (the uncompressed
JSON exceeds GitHub's file-size limit; the scripts read the gzip transparently).
Scoring any Sage level reproduces its row in the Section 3 table:
# lower PR is better python scripts/benchmark_money.py score --level 3ply # Sage 3P python scripts/benchmark_money.py score --level truncated3 # Sage 3T # --level: 1ply 2ply 3ply 4ply | truncated1/2/3 | rollout
To score a different engine entirely, apply the same rule the script uses internally: for every decision in the dataset, take your engine's choice and compare its equity to the stored reference ("rollout") equity — the mean error × 500 is its PR.
Three adaptive passes re-create the dataset. The games are seeded (500 games from seed 1), so the decisions and reference equities come out the same:
# 1. simulate 500 Sage-3P self-play games (+ XG transcripts) python scripts/benchmark_money.py build --stages pass1 --n-games 500 --workers 6 # 2. re-evaluate every decision within 0.05 at 3T python scripts/benchmark_money.py build --stages pass2 --n-threads 16 # 3. roll out every decision still within 0.02 python scripts/benchmark_money.py build --stages pass3 --n-threads 16
Pass 3 writes data/money_benchmark/benchmark.json
(and its .gz); score it exactly as in Option A.
Omitting --stages runs all three passes in order.
XG has no API, so its results come from XG's Batch Analysis. Pass 1 writes
XG-import transcripts to data/money_benchmark/xg/ — these
are not shipped, so regenerate them with pass 1 if you started from the
dataset alone.
xg/ folder with a custom level —
3-ply decisions, upgrading to the level under test on disagreements — with
Save Games after analyze checked. XG writes one
.xg per game.
python scripts/benchmark_pr_xg.py, which reads XG's
top decision per position from the .xg files and scores
it against the same reference equities, printing the same PR breakdown.
The 5-point match study is reproduced the same way with
scripts/benchmark_match.py, the match-play twin of
benchmark_money.py — the match length and number of matches
are arguments, and the match score is threaded through every evaluation. Its ground
truth ships as data/match_benchmark/5pt/benchmark.json.gz.
# score against the shipped match dataset (no rebuild) python scripts/benchmark_match.py score --match-length 5 --level truncated3 # or rebuild: 130 seeded 5-pt matches, three adaptive passes python scripts/benchmark_match.py build --match-length 5 --n-matches 130 --stages pass1 --workers 6 python scripts/benchmark_match.py build --match-length 5 --n-matches 130 --stages pass2 --n-threads 16 python scripts/benchmark_match.py build --match-length 5 --n-matches 130 --stages pass3 --n-threads 16 # XG columns: batch-analyze data/match_benchmark/5pt/xg/, then python scripts/benchmark_pr_xg_match.py --match-length 5
This method ships no pre-generated artifacts — its intermediate files live under the
git-ignored logs/ tree and depend on the manual XG step — so
it is reproduced end to end. Each experiment is three stages: simulate, analyze in
XG, then aggregate and roll out.
# 1. 200 Sage-3P self-play money games -> logs/sage_vs_sage/ python scripts/run_sage_vs_sage_games.py 1 200 --level 3P --workers 6 # 2. (manual) XG Batch-Analyze: XG 4-ply, upgrading to Roller++ # on disagreement, Save Games after analyze -> one .xg per game # 3. find disputes and roll them out (5,184 paths, 3-ply) python scripts/aggregate_xg_pr.py --threshold 0.02 --rollout-threads 24 --n-trials 5184
# 1. 70 Sage-3P self-play 5-point matches -> logs/sage_vs_sage_match/ python scripts/run_sage_vs_sage_match.py 5 70 --level 3P --workers 6 # 2. (manual) XG Batch-Analyze with the same custom level # 3. match-aware aggregation + rollouts python scripts/aggregate_xg_match_pr.py --threshold 0.02 --rollout-threads 24 --n-trials 5184
Both aggregators re-evaluate each Sage-3P-vs-XG disagreement at 3T, keep only the genuine 3T-vs-Roller++ disputes, and roll out those whose XG-measured gap exceeds 0.02 — the procedure described in Section 4.