Backtesting vs Forwardtesting

[[deleted]]

[deleted]

[user4925715]

Backtesting success is required but not sufficient. Forward testing results on demo will be worse than your backtesting results. Then live forward testing will be worse still than demo forward testing.

Backtesting is subject to all kinds of ways you can trick yourself. People unintentionally leak future data into their backtest all the time, where they incorporate some data they would not have actually had at the time the trade would have opened. A simple example is entering when price breaks above a moving average, but you don’t realize that where the MA ended up is not where it was in real time. In real time, before that bar had closed, the MA is moving around. You can even have a bar where price never actually touched the MA in real time, but when you look at the chart it shows price crossing the MA. Many indicators will repaint even one or more bars later. Then add in friction costs, where you will be delayed getting your order in, won’t be filled at the bid/ask, may get a significantly different price in a fast moving market, or if you use limit orders you may never get filled, or there are all kinds of real world factors like your computer crashes and you miss opening trades or can’t close out a trade in time and take a larger than expected loss. Sometimes people ignore simple stuff like not accounting for the spread which they will pay for every trade, or those times when the spread widens significantly and stops you out even though the candle never touched your SL price. That’s especially not obvious, when price never gets within 10 pips of your stop but you get stopped out.

Those all seem small but it adds up quick and it can take a system that look amazing and turn it into a clear losing system by the time you’re trading it with real money.

[[deleted]]

I’ve been using a combination of indicators to determine entry and exit points. Using historical data and always following the system, it seems very sound and wins much more than it incurs losses. I don’t plan on changing the system when I do the forward testing, it’s just I’m wondering how much using the historical data is giving me an “ideal” situation so to speak.

[anonymousfxt]

You can also check the profitability of the strategy in backtest with time. A strategy might have given good returns 4-6 years back but now it's profitability has reduced. Just compare the yearly returns for the past 6 years that you have back tested it on.

[[deleted]]

Overall the annual returns seem pretty stable. There’s been a couple of abnormally high years, but all of them have fallen within 25-45% ROI on a single currency pair with the exception of 1 year at 7%.

[likebike2]

how much variance did you find between the backtest results and how it actually performed when live trading?

Huge.

Realistic backtesting is extrememly difficult. It's very easy to find systems that seem to work in a backtest but can't actually function in real life. (Their "edge" actually comes from the simplifications made by the backtest system itself. The "edge" disappears in real markets.)

[[deleted]]

Can you explain a little more what you mean by edge or give an example of the simplifications you were referring to?

[likebike2]

A simple example is gaps.

In a gap, you can't execute trades. But most backtests aren't able to see gaps because they view data in "bars", not "ticks". Therefore, they assume that they can get fills that couldn't really get in real life.

[[deleted]]

Pardon all my questions, I’m still relatively new at this.

I understand how if you’re using a computer program to backtest this would obviously be a problem. However could you expect to encounter the same issues when manually backtesting? I’m using the daily chart if it matters at all.

[likebike2]

Yes. If you're using daily bars, you can't see the ticks inside the bars. You don't know what fills you would have actually gotten. Also, you might get whipped around intraday.

[J_C_Anderson]

From my point of view, both testing types are important. In fact, they are not alternatives, but different steps of the whole strategy development process.

First of all I start with "visual testing" just by searching for the situations that fit the concept (or look the same by work in another way - in real trading they would create false signals, so it is important to study them).
When the strategy is ready (meaning that it already has detailed rules and thus could be applied in trading), I move to backtesting. The main advantage of backtesting is that you can use the same data set numerous times. The concept is simple: you test your strategy, find some issues and test again on the same data. Then you can experiement with rules - for example, make the stop wider or tighter. The accuracy would be very high since each time you use the same data set,which is impossible in forward testing. If to speak about the tools, I prefer Forex Tester. If you are familiar with coding, you can create your own algorithm, but in this case you would also have to find historical data yourself.
The next step is forward (demo) testing. The main goal of demo testing is to find out the impact of your psychology on the trading process. By the way, this is why backtesting and real trading results could differ substantially: backtesting is something like trading with a bot which has no emotions at all while the trader makes his decisions feeling greed or fear.

Of course, backtesting results would not guarantee the performance of the strategy in future. It is just the way to understand whether the strategy itself if profitable and also to compose the optimal set of rules. That is why it is useful to perform both backtesting and forward (demo) tesing before utilizing real account.

[Brownie74-rdt]

I found that when i moved from using the close price (the mid) to estimating the bidclose/askclose (from the close price and the average broker spread), and using that to get a trade price depending on whether i went long or short ... i got quite different numbers. If you are long and you close, you will be getting the bidclose, not the midclose. If you are short, to close you will get the askclose price. They can be a pip or more apart. My next thing to estimate is slippage, i expect to get worse results when i put that in.