Two Models, One Ledger: How an Audit and a Bucket-by-Bucket Drill-Down Narrowed Our Whole System to a Single Niche.
Saturday, July 11, 2026
This is the story of one day of work — July 10 — that changed what this system bets more than anything since we went live. No new signal was discovered. Instead we did the opposite: we put our own results under two different models' microscopes, watched most of our "edge" dissolve under honest statistics, and kept the one narrow slice that survived every cut. Then we bet only that.
The data we were sitting on
By July 9 the system had accumulated three parallel datasets: 1,400 graded games (every MLB slate since March 27, 67 columns each — open/close lines, ticket% vs money% splits, results), a real-odds feed snapshotting Polymarket's actual asks every 30 minutes, and — the newest and most honest layer — 239 real-money legs placed by the live daemon, every fill verifiable on-chain. That third dataset is small, but it's the only one with no phantom prices in it: every number is cash that actually left the wallet.
Pass one: GPT-5.6 audits the machine
We gave an outside model (GPT-5.6, run on the server with full file access) a standing brief: assume nothing, reproduce everything. The core held — all 773 post-lock CSV rows reproduce mechanically, and all 232 settled chain buys reconciled to the ledger with no survivorship. But the audit cut deep everywhere else, and publishing that is the point of this site:
- Our headline P&L was overstated. The ledger recorded nominal stakes; actual chain cash ran ~1.4% richer per fill. True lifetime P&L at the time was +$15.84, not +$30.68 — and after the next day's losses, −$4.78 on $1,122 staked. The books now reconcile to the chain nightly, to the penny.
- A stake-sizing model we'd just shipped was retracted. Its backtest was future-leaked (each historical decision quietly used statistics from the full sample). Leak-free, it lost to flat staking. It now runs shadow-only.
- Two real money-path bugs (order sizing that could overspend the stake; a response handler that could log a rejected order as placed) were found and fixed the same day.
Separately, a wide-net mining pass over everything else we collect — 785 disclosed cells across static MLB cuts, intraday price paths, and five other sports — produced nothing that survived multiple-comparison correction. The market is efficient almost everywhere we can see. That's the backdrop for what follows: any claimed pocket of edge starts life as probably-noise.
Pass two: Fable 5 drills the buckets
With the books chain-true, we pointed Claude (Fable 5) at the as-bet ledger itself and asked a narrower question: of the money we've actually wagered, where did the profit come from? Bucket by bucket:
| Bucket (fee-true, through 7/9) | Record | ROI |
|---|---|---|
| Moneyline legs on non-Rule-C signals | 19-31 | −20.6% |
| Moneyline legs on Rule C fires | 45-41 | +4.6% |
| Moneyline legs below a 40¢ ask (longshots) | 7-14 | −17.6% |
| Rule C, dog is the away team — ML | 37-32 | +10.9% |
| Rule C, dog is the home team — ML | 8-9 | −17.9% |
| Rule C away dog — runline (+1.5) | 26-9 | +11.7% |
| Rule C home dog — runline (+1.5) | 7-6 | −26.1% |
One shape kept repeating: everything positive routes through Rule C fires on away underdogs; home-dog fires are underwater in both markets. The line-cooling signal appears to mean something different when the crowd's dog is at home — so the drill-down suggested not just skipping home fires but inverting them: back the away favorite −1.5 instead. That's the niche. A tempting second "sweet spot" also showed up — a specific 40–47¢ ask band running +12.4% — and we'll come back to what happened to it.
Pass three: the models check each other
Finding your own sweet spots in your own ledger is exactly how bettors fool themselves, so every claim from the drill-down went back to GPT-5.6 as a numbered list with a hostile brief: reproduce, correct, or kill. The verdicts, unvarnished:
- Confirmed with corrections: every point estimate reproduced (one was off by $0.91 — the most recent legs hadn't been chain-trued yet; fixed, and the nightly reconcile now exists because of that finding).
- Weakened: "Rule C is profitable" is not robust — under stricter cohort definitions (the CSV's canonical dog rather than the as-bet team) it's roughly flat. What's robust is only the gap: Rule C beats non-Rule-C under every definition tried.
- Direction confirmed, significance denied: the home/away split keeps its sign under every cohort definition and both markets — but the honest date-clustered interval on the away-minus-home effect is [−12.9, +72.4] percentage points. It crosses zero. This is a lean, not a law.
- Killed: the 40–47¢ "sweet spot." Include the 47¢ endpoint and +12.4% collapses to +2.0%; the bootstrap interval spans −21% to +44%; nine neighboring endpoint choices were disclosed and none survives correction. A mined boundary, dismissed. (The 40¢ floor survives only as a risk cap on longshots — the auditor's framing, which we adopted verbatim: "a decision under uncertainty, not proven alpha.")
What we actually changed
The result is RULE_C_SPLIT_V1, live since July 10, frozen in a spec file before the first bet: live money only on strict decision-time Rule C fires at $7.50 a leg — away-dog fires get the dog moneyline (40¢ floor) plus the +1.5; home-dog fires get the fade: away favorite −1.5. Everything else the old policy would have bet — Rule A, Rule B, every secondary tag, home-dog legs, sub-40¢ moneylines — is now logged to a counterfactual paper ledger at the real asks we declined, graded nightly.
That paper ledger isn't bookkeeping; it's the tripwire. After 30 live legs or three weeks, if the pool of bets we stopped making outperforms the pool we kept, the change failed and we revert. Here's the running scoreboard, rebuilt every night (56 live legs settled so far):
| Pool since 7/10 | Record | Staked | P&L | ROI |
|---|---|---|---|---|
| Live pool (Rule C only, $7.50/leg, fee-true) | 27-29 | $316.59 | −$35.65 | -11.3% |
| Policy-skip paper pool (everything we stopped betting) | 77-62 | $695.00 | −$31.03 | -4.5% |
2 live leg(s) still open (postponement holds).
Day one was almost comically kind to the new policy: the live pool went 5-0 — including the first-ever fade leg, which cashed on a favorite winning by exactly two runs — while the declined home-dog legs lost on paper. We enjoyed it and we don't believe it: one day proves nothing, which is why the gate is thirty legs, not five.
The honest bottom line
Nothing above is statistically proven. The home/away split is p≈0.30 territory on the covers; the profitable-looking cells are dozens of legs, not thousands; and the auditor's search-accounting section reminds us we've examined hundreds of cells across this project, so any single survivor carries a heavy discovery-bias discount. What we'd defend is the process: chain-true books, two models adversarially checking each other's work, preregistered rules frozen before the first bet, a counterfactual ledger of every bet we decline, and a public revert condition we can't quietly ignore. That's the whole edge we're sure of. Nothing here is betting advice — it's a research log, and the honest status is "under test."
← All research