Orthodox, established methods of statistical science are used to assess the trustworthiness of each of the programs that are presented here. But original research and development is also carried out.
The Traded Portfolio project conducts out-of-sample testing in the time domain and also cross-sectionally in order to arrive at an unbiased characterization of any given asset allocation rule’s likely future performance. Several statistics are compiled that bear on the significance of the outcomes. For example, two tests attempt to confound the scheme under study by presenting it with randomized data to see if it reacts properly. A third involving cross validation investigates to see if it can reasonably be claimed that there was no “data snooping” (which refers to an improper form of “data mining” that amounts, in effect, to fitting the scheme to a set of data without realizing that to be all that you’re doing and then wrongly presenting the fit as if it were a prediction). Project code is always made to contain a nopeekforward() function that that introspectively tests to see if the code improperly computes allocations by making use of data that are not solely from periods prior to the period during which the allocations are put into effect.
The writeups of the Notes menu provide some details and go over some of the opportunities and complications. You may find the little essay Real vs. Hypothetical to be of interest because it provides an example of an out-of-sample testing procedure that illustrates how involved the process generally is and why it’s necessary. And that’s just one of the out-of-sample procedures in use, cross validation being another.