Systematic trading researchWhat does picking a grid setting on a short backtest cost?
Huyen X. Vu · Quantex Research, AlgoTrade Pte. Ltd., Singapore · ORCID 0000-0002-2209-842X
Abstract
We simulate 36 settings of one averaging grid on one-minute gold bars from 2005 to 2024. Over 20 years every setting loses money, yet in 30 of 34 seven-month windows at least one setting is profitable. The setting picked on 7 months earned a median of +104 USD a month in its window and -160 the next, a drop of 233 (95% CI 209 to 607). The whole family dropped only 21 a month over the same windows, so most of the drop comes from the selection itself. Picking did not beat a random setting. After longer backtests the drop was much smaller: 261 after 7 months, 144 after 21 and 35 after 53. Results hold when settings are scaled to the gold price.
Keywords: backtest overfitting; selection bias; winner's curse; averaging grid; gold; XAUUSD; out-of-sample
1. Introduction
Grid settings can be chosen on a short backtest that looks profitable. We measure what that choice costs when the grid itself has no edge.
We build a family of 36 settings of one simple averaging grid with no long-run edge, pick the best on a short window, and measure what happens next. Because every setting loses over 20 years, any "good backtest" here comes from the window and the choice alone.
2. Data and method
- Data: one-minute XAUUSD bars, January 2005 to December 2024 (240 months), from the public Kaggle dataset by novandraanugrah (CC0), as copied on Hugging Face (ZombitX64, revision ff6469dc). 2025 is left out because of missing days.
- Grid family: one side at a time, starting in the direction of the last one-minute candle; an order is added each time price is at least step USD worse than the last; order sizes grow by mult; the basket closes at average + tp, at a 300 USD floating loss, or before a weekend. Step 0.5/1/2/4 × mult 1.0/1.1/1.2 × tp 0.5/1/2 = 36 settings. Spread fixed at 0.20 USD (0.40 in a stress run).
- Selection test: 34 blocks of 7 months; in 17 non-overlapping pairs, the setting with the best mean in the first block is followed in the next. Six tests were listed before running, with Holm and Benjamini–Hochberg corrections. Bootstraps, the price-scaled version and the other robustness runs are exploratory.
- Price-scaled version: step, take profit, stop and spread scaled by the gold price at basket start, so a setting means the same at 410 and at 2,790 USD.
3. Results
| Measure | Settings in USD | Price-scaled |
|---|---|---|
| Picked setting, its own window (median, USD/month) | +103.7 | +48.1 |
| Same setting, next window | −159.9 | −84.9 |
| Drop (block-bootstrap 95% CI) | 232.5 (209–607) | 173.0 (124–250) |
| Next window negative | 15 of 17 | 12 of 17 |
| Whole family's drop, same pairs (median) | 21.1 | 7.3 |
| Pick beats a random setting next window? | No (median rank 0.49) | No (0.66, p 0.31) |
| Rank persistence (mean Spearman) | 0.32 | 0.32 |
Over 20 years all 36 settings lose (best −66.1 USD/month, worst −821.0), yet in 30 of 34 seven-month blocks at least one setting shows a profit. The whole family barely changes from one window to the next (median drop 21 USD/month), so most of the picked setting's drop comes from the selection itself — the winner's curse.
Longer backtests
| Backtest length | Windows | Drop (USD) | Drop, price-scaled |
|---|---|---|---|
| 7 months | 19 | 261 | 237 |
| 21 months | 18 | 144 | 108 |
| 53 months | 15 | 35 | 24 |
With 16 blocks of 15 months, the best in-sample setting fell below the out-of-sample median in none of 12,870 splits (3.0% in the price-scaled version): long samples rank this family reliably.
What did not hold
- At the higher spread, the pick seemed to beat chance (median rank 0.80). In the price-scaled version this disappears (0.66, p 0.11): it came from wide-step settings doing well when gold was cheap, so we do not claim it.
- Ranks persist partly because a few tight grids are always bad; without the 6 worst, mean Spearman falls to 0.19 (exploratory, and partly a range effect).
- Cost, volatility-regime and stop-out effects were not found, but these tests had little power.
4. What it shows
- In this family, a profitable 7-month result was weak evidence: the picked setting's next window was negative in 15 of 17 pairs.
- The drop after picking is mostly a selection effect, and it was much smaller after longer backtests.
- These findings apply to one simulated family of averaging grids, not to all grids or all EAs.
5. Limitations
- One generic grid family; fixed spread; one-minute bars; no slippage, swap or commission.
- One public feed with an unknown provider and time zone.
- 17 pairs is a small sample; blocks of two pairs only partly handle dependence between windows.
Sources
- Bailey, D. H., Borwein, J., López de Prado, M., Zhu, Q. J. The probability of backtest overfitting. Journal of Computational Finance (online 2016). https://doi.org/10.21314/JCF.2016.322
- Benjamini, Y., Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B 57(1), 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x
- Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6(2), 65–70.
- novandraanugrah. XAU/USD Gold Price Historical Data. Kaggle dataset, CC0: Public Domain.
- Quantex Research WP 04: Tail moves in gold, and what they mean for an averaging grid.
Declarations
- Author: Huyen X. Vu, Quantex Research, AlgoTrade Pte. Ltd., Singapore (ORCID 0000-0002-2209-842X).
- Type: Quantex internal research report, published on this site only; not peer reviewed. Reviewed before posting by two AI reviewers (Gemini, ChatGPT); all numbers re-checked against the script output.
- Use of AI tools: Data analysis and drafting were assisted by AI tools (Claude, Anthropic). The author checked the results and is responsible for the content.
- Conflict of interest: The author is the founder of AlgoTrade Pte. Ltd., which develops and runs systematic trading models.
- Funding: No external funding.
- Comments: welcome at [email protected].
- How to cite: Vu, H. X. (2026). What does picking a grid setting on a short backtest cost? Quantex Research, AlgoTrade Pte. Ltd. https://quantex.com.sg/research/short-backtest-selection/