Why a backtest flatters
A backtest is a search. Each run tries one configuration. The more you look, the better the best result looks, whether or not the idea works.
Orbital counts that search and discounts the Sharpe ratio by it. Six rules keep the search small and the result honest.
Six rules
1. Start with a purpose
Write the idea down before the first run: what should happen, why, and what would prove it wrong. A rule found by sweeping numbers is one you can only fit. When you save a version, or a run creates one, What changed? takes an optional note; use it to say what the change was for.
2. Be forward-looking
A decision may use only what was known when it was made. The engine decides on closed bars and fills at the next bar's open by default, so it does not peek; see One bar of delay. Hindsight enters through what you give it: a list of today's winners, signal rows built with later knowledge, or a Universe preset. A preset is a curated list of current index members. Choose the universe as you would have chosen it on the period's first day.
3. Test several periods
Another window adds a run but not a trial for a Rules or Program strategy, and a new configuration for a Signals strategy, whose rows are built for their period.
Pick the window
On the strategy's Runs view, drag the window over the data.
Run it
Click Run this window…, check the settings and click Run backtest.
On the Report, Market events shows the strategy's return through named stress periods beside the benchmark's. Rolling statistics shows whether its Sharpe ratio holds through time. The Drawdown table lists the deepest falls behind the max drawdown and when each recovered.
4. Mind trade frequency
The win rate of a few trades can be a lucky handful. The Strategies list marks fewer than 30 closed trades as too few under Profitable trades. Many trades carry the opposite risk: costs decide the result. Check Trades per day and Turnover in the Report. Returns are net of the commission and slippage the run charges, but the bid-ask spread and market impact are not modelled.
5. Change parameters sparingly
Every new combination of values is a new configuration and one more trial. Prefer round, defensible numbers, and a setting whose neighbours also work. If a 14-day lookback wins while 12 and 16 lose, the 14 is probably luck. The Runs table's Setting column shows the parameter values each run was given, so a sweep is on record.
6. Use a reality test
Judge the configuration on data it never saw: a later period you held back, or a forward test. On the Report, Resampling redraws the closed trades 1,000 times and shows the Median resample and the Worst 1 in 20. A result that holds on a held-back period, in a forward test and under resampling is worth paper trading. See Backtest, forward test and paper.
How much searching is too much
These bands are rough rules of thumb, not Orbital measurements. Orbital records the first row; keep the other two yourself.
| Measure | Low risk | Take care | High risk |
|---|---|---|---|
| Backtests per strategy | Under 30 | 30 to 70 | Over 70 |
| Parameters in the strategy | Under 10 | 10 to 20 | Over 20 |
| Hours spent tuning | Under 8 | 8 to 16 | Over 16 |
Backtests are every run across the strategy's configurations; parameters are the values you could set differently. Past the middle band, stop tuning and test on data the strategy has not seen.
What Orbital shows
The trial count
The trial count counts the distinct configurations of a strategy that have run, whoever launched them, and the Report's Run properties states it once there are two or more. See Versions and configurations.
The deflated Sharpe
The deflated Sharpe ratio discounts the raw Sharpe for the trial count. It needs at least two trial Sharpes that differ, and a negative figure means the search alone explains the result. The Report's Probabilistic Sharpe ratio is a different figure: the chance the true Sharpe is above zero.
The deflated figure never appears without the raw Sharpe and the count. In the Strategies list's List view, open Columns and pick Full; on a narrow window the list keeps the Simple columns. The Evidence column shows the deflated figure over the raw one and the count. Hover it for the whole sentence, such as "Sharpe 1.62 raw · 1.08 after deflating for 47 configurations tried." A strategy whose runs all predate configuration tracking reads raw · search not tracked.
Chat and get_backtest return the same three figures. The Sharpe tile, the statistics tables, the Runs table and Compare show the raw Sharpe only.
Hold out with a forward test
A forward test is the cleanest holdout, since everything after its start is data the strategy never saw. Finish the search, start one from the run you would trade, and judge the forward stretch on its own after enough sessions and trades; a week says little. See Start a forward test.