Most back-tests are not wrong because the maths is wrong. They are wrong because the test knew something the punter could not have known at the time, or because the rules were bent until the past looked perfect. Both are easy to do by accident, and both produce a system that looks excellent on paper and loses money from the day you start following it.

This is how to avoid that. The rules are simple to state and tedious to apply by hand, which is why most people do not. The second half shows how the RaceMetrics System Builder applies them for you.

The two rules

A horse racing back-test is only honest if every number it uses was knowable before the off. That means two things.

First, point-in-time data. A trainer's strike rate, a horse's best speed figure, a sire's record with two-year-olds, a rating — each must be computed from racing strictly before the race being tested, never from today's database. If the test can see the result it is judging, it is not a test.

Second, a holdout. Build the rules on one slice of history and confirm them on a slice they have never seen. A system that only works on the data it was built from has learned the noise, not the edge.

Everything below is one of those two rules applied somewhere specific.

1. Use only what was knowable

The obvious version of hindsight is filtering on the result. Nobody does that on purpose. The version that catches people is filtering on a figure that contains the result.

Take a trainer's strike rate. If you filter for "trainer strike rate over 20%" and the figure is today's career number, then for a race in 2019 you are using wins from 2020 to 2026 to decide whether to back the horse in 2019. The rule will look better than it was, because trainers who went on to do well are over-represented. The honest figure is the trainer's strike rate on the morning of that race, and it is different for every race.

The same applies to ratings. Every run carries two figures that answer different questions: what the rating said going into the race, and what the horse earned in it. The pre-race figure is a selection you could have made. The earned figure is a description of what happened. Filter on "earned rating rank 1" and you will see a strike rate above 60%, because you are describing the winners rather than picking anything. If a back-test looks too good to be true, check which of the two it is on.

In the System Builder, every form statistic — trainer and jockey form over 14 and 28 days, course and distance records, sire and damsire records, the RaceMetrics Ratings, speed figures, the Official Rating carried into the race — is stored as it stood before the race date. Sire, damsire and owner records are taken as of the first of the race's month. Ratings are labelled (pre-race) where a hindsight twin also exists, and the My Ratings fields follow the same split: back-test the pre-race pair, read the earned pair as research.

One caveat worth knowing. The ready-made draw and pace tables you see on a racecard — how a stall has performed at a course and distance, how a track is usually run — are refreshed to the current state of play, and earlier states are not archived. They are right for today's race and wrong for a back-test, because a 2021 race would be judged against a table that includes 2024. The per-run rows are different: every run's draw, field size, finishing position and running style is stored back to 1997, so a genuinely point-in-time draw or pace figure can be computed from them, and the System Builder's draw and pace filters are built that way.

2. Hold something back

A rule that fits the whole of history tells you nothing about the future, because you chose it by looking at the whole of history. The only way to find out whether it generalises is to test it on races it was not built from.

There are two ways to split, and they catch different problems.

Split by time. Build on the older years and test on the most recent twelve months. This is the stricter test, because it asks the question you actually care about: would this have worked recently, after the market had a chance to catch up? A system that was profitable from 2018 to 2023 and flat since is historical, not live.

Split at random. Assign every race to one of two buckets and build on one, confirm on the other. This catches a different failure: rules that fit a handful of specific races rather than a real pattern. The split has to be by race, not by runner, and it has to be fixed, so the buckets never leak into each other between runs.

The System Builder does both. The Robustness tab trains on the older data and re-tests the final twelve months separately, then gives a plain-English verdict. Consistent means the recent return is within ten points of the training return. Weaker means it has faded by more than that. Flipped means a system that made money in training lost more than 5% in the year it never saw. Insufficient means the holdout has fewer than thirty runs or fewer than five expected winners, and no verdict would be honest. The Data split setting gives you the random version: Half A / Half B, or a train-80% / test-20% holdout, both assigned by a fixed hash of the race so a given race always lands in the same bucket.

A system that passes the time holdout and the random split is not guaranteed to work. It is a great deal less likely to be curve-fitted, which is the most anyone can say before the money goes on.

3. Do not add a rule because it helps

This is where most systems go wrong, and it feels like diligence while you are doing it. You have a system at 4% return. You notice the losers are mostly in fields over fourteen. You add "field size 14 or fewer" and the return jumps to 9%. Then you notice the remaining losers skew to a particular going, and so on. Each rule is defensible on its own. Together they describe the lucky winners of history so precisely that nothing in the future fits the mould.

Three habits defend against it.

Sweep the threshold before you trust it. If a rule says "odds under 8", ask why 8 and not 7 or 10. Run the test at every value and look at the shape of the curve. A smooth decline means the edge degrades gently and your threshold is a preference. A cliff — profit at 8, loss at 9 — means one value is carrying the whole result, which is the classic over-fitting signature. The System Builder's Frontier tab does this sweep, up to twelve values per rule, and charts bets against return so the shape is visible.

Test every rule by removing it. For each rule you have, re-run the system without it. If the return goes up, the rule was excluding an edge, not protecting one. If it barely moves, the rule is a volume lever you can loosen for more bets. If it falls, the rule is load-bearing. The Trim tab runs exactly this and labels each rule Drop, Loosen, Keep or Redundant. Most systems that have been tuned by hand for a while carry two or three rules that should go.

Let the machine add rules only against a holdout. If you are going to have software suggest rules, it must be judged on data it did not select from. Auto-build in the System Builder proposes one rule at a time, ranks candidates on the training years only, then checks each against the held-out twelve months and keeps it only if it holds up out-of-sample. When the best remaining candidates only work in training, it stops and says so. "Stopped to avoid over-fitting" is the tool refusing to hand you a system that would not have worked going forward.

4. Count every bet you would have had

Three quiet omissions flatter almost every home-made back-test.

Fallers and pulled-ups are losing bets. A horse that fails to finish never wins, and you would have backed it. Leaving non-finishers out of a jumps back-test improves every figure by roughly three per cent for no reason. Genuine non-runners are the only runners that should come out — and when they do, the draw renumbers, exactly as it does on the day.

Price matters more than strike rate. A 30% strike rate at average odds of 2/1 loses money. Test against a price you could actually have got: Betfair SP after commission, the starting price, or a recorded early price if you bet in the morning. Do not test against the best of several prices unless you would genuinely have shopped every bet.

Hundreds of runs, not dozens. A filter with forty qualifiers and a 40% return has told you about forty horses. Pair any strike-rate or A/E rule with a minimum-runs floor, and treat a thin sample as a lead to investigate, not a result.

The System Builder counts non-finishers as the losers they were, removes only declared non-runners, settles at Betfair SP net of the commission you set, SP, or the 8.30pm, 7.30am and 10.30am bookmaker prices where they were recorded, and greys out any breakdown row below your minimum-runs floor rather than showing it as a result.

5. Read the shape, not the total

A system that returned 12% over six years with a 140-bet losing run in the middle of it is a system you would have abandoned in month nine. The headline return is not the number that decides whether you could have followed it.

Look at the longest losing run and the maximum drawdown — the worst peak-to-trough fall in the bank — before the return. Then look at the yearly split: a steady line beats one bumper season propping up the total. The System Builder's Streaks & Drawdown card shows both, the equity chart shows the year-by-year shape, and the Staking tab re-runs the same bets under ten staking plans with a Monte-Carlo estimate of how often each would have bust the bank.

6. Then follow it, and keep score

The final test is the one nobody can fake: the record since you started following it. Save the system on the day you trust it, and from then on judge it on what came after. The System Builder tracks every saved system's live record from its save date, separately from the back-test, and lists its qualifiers among today's and tomorrow's declared runners using the same pre-race conditions it was tested on — so what you follow is what you tested.

The checklist

  • Every filter uses a figure as it stood before the race. Pre-race ratings, not earned ones.
  • The rules were built on one period and confirmed on a later one. The verdict was consistent.
  • Each threshold was swept; no cliffs.
  • Each rule was tested by removal; nothing labelled Drop is still in.
  • Non-finishers counted as losers; only non-runners removed.
  • Settled at a price you could have taken, after commission.
  • Hundreds of qualifiers, not dozens.
  • You have looked at the longest losing run and decided you could sit through it.
  • The live record since saving is the number you now watch.

Frequently asked

How do I back-test my own horse racing ratings without hindsight?

Use the rating as it stood before each race, not the figure the horse earned in it. For your own ratings that means the pre-race value — typically the best of the horse's recent rated runs — and never a figure derived from the result you are testing. In RaceMetrics, My Ratings stores both and labels the pre-race one; back-test that, and read the earned one as a description of what happened.

What is a holdout in a betting system?

A slice of history that the rules were not built from, kept back to confirm them. The strictest version holds back the most recent twelve months: build on everything older, then check whether the edge still shows up in the year the rules never saw. A random split by race is the other version. A system should pass both.

How do I know if my betting system is over-fitted?

Three signs. The return collapses when you move a threshold by one step. Removing a rule improves the return. The system is profitable in training and flat or losing in the most recent year. Any one of them means the rules are describing past winners rather than a repeatable pattern.

What is a train/test split in horse racing back-testing?

Dividing the historical races into a set used to build the rules (train) and a set used only to confirm them (test). The split can be by time — older years train, recent year tests — or at random by race, commonly 80/20. The test set must never influence which rules you choose, which is why software that proposes rules should rank them on the training set only.

Does a high strike rate mean a system is good?

No. Strike rate without price is meaningless; a 30% system at short prices loses. Judge on return at a price you could have taken, after commission, and on the A/E — actual winners against the number the market expected — which tells you whether you are beating the price at all.

The System Builder is part of RaceMetrics Pro. The full guide to every tab is in the System Builder help.