New Techniques  |  JUL 1997

Rainbow Charts by Mel Widner, Ph.D.

Here, in part 1 of two, these Contributing Writers explain the steps necessary to evaluate trading system behavior with the use of statistics. Most systems I have seen are simply optimized by being tested with different parameters, examined for profitability and sometimes assessed for a few other characteristics. On occa-sion, I have encountered systems tested on perhaps one or two out-of-sampleY´ periods to determine whether performance would hold up. However, few who write about system development or developers themselves have attempted to estimate probabilities or compute statistical analyses on trading systems to assess likely future system performance. For the sake of simplic-ity, I have often been guilty of omitting such analyses from my own work. However, I often get comments regarding issues that statistics directly bear upon, such as sample size, replicability, curve-fitting, generalization and determining whether a system will trade profitably or fall apart when finally put to the test. I have also had discussions with system developers who are reluctant to apply any optimization strategy because of a fear of curve-fitting rooted in a lack of knowledge about the statistical issues involved. Traders could benefit greatly from additional knowledge of statistics and statistical issues, but it is difficult to gain such when little has been published about the subject. Therefore, I have decided to divert my attention from the search for good systems or system elements to a discussion of some of the ways with which I evaluate trading system behavior with the aid of statistics.

by Mel Widner, Ph.D.