TRADING SYSTEMS
System Comparison
Walk-Forward Testing
Here’s how you can use walk-forward testing as a tool to compare trading systems and gain insight into comparing the future performance of systems.
Recent advances in trading system software and powerful hardware capabilities have provided a valuable tool in evaluating the future viability of trading systems — walk-forward testing. This technique allows the trading system developer to quickly assess if the system will survive (or even prosper) in the future. It does this by segmenting the past into in-sample and out-of-sample periods, and by optimizing each successive in-sample period to provide the values for the next out-of-sample period. In addition, testing can be used to compare one system with another. Walk-forward testing provides a unique insight into comparing the projected performance of trading systems.
Optimization and its uses
Optimization of trading systems is used in order to learn from the past those values that are most likely to succeed in the future. This is at the heart of technical analysis — the concept that the future (especially the near-term future) can be predicted from the most recent past. The values determined from optimization of the most recent past should be used over the next time period. In the context of walk-forward testing, the most recent past is called the in-sample period, and the next time horizon is called the out-of-sample period.
Optimization gives the trading system designer the ability to determine the best values for variables in the in-sample period. When a trading system designer puts too many constraints into the system rules, then the system is curve-fitted. Walk-forward testing will show that if a system is curve-fitted it may work for a period in the past, but it will fail for periods that are out of sample.
FIGURE 1: WALK-FORWARD TESTING. Here you see the dates used for in-sample and out-of-sample data.
Importance of walk-forward testing
Walk-forward testing is one of the most powerful tools developed to analyze the future performance of trading systems. For the purposes of this comparative analysis, it was decided that the period over which the walk-forward optimization would be conducted would be eight years, with the first four being the first in-sample period. Walk-forward testing works as follows: the first in-sample period is selected, and the variables selected are optimized for a chosen objective function. Those values are then used for the first out-of-sample period. The process is then repeated for each set of in-sample and out-of-sample periods.
In Figure 1, you can see the sets of dates that were used in this article’s analysis of the systems. With walk-forward testing we can stitch together the out-of-sample periods to get a clear idea of how the system would have performed over the total out of sample period from January 1, 2004, to December 31, 2007.
Optimization objectives
A key question to be asked by any system designer when implementing optimization is: What is the objective to be optimized? Most system trading software will allow optimization of net profit, but typically, this is not what the real world trader feels should be the most important objective.
To compare systems, I chose the K-ratio measure. Developed by Lars Kestner and detailed in an article in Stocks & Commodities, this measure can be called one of the most important measures of the performance of a system. The K-ratio is a unitless measure of performance that can be compared across markets and time periods. Together, the Sharpe ratio and K-ratio are the most important measures when evaluating trading strategy performance. For more information on choosing an objective for a trading system, please see Quantitative Trading Systems by Howard Bandy.