I am skeptical of back-testing until confirmed by testing with real money.
I believe the best test trading is done using a limited pool of real money conducting real trades that fully exercise the full chain of brokers and technology. Other approaches seem to provide good data that can help during the development process but I have personally experienced too several instances where something works slightly different and allows the back-test to show results better than they can really deliver.
I favor testing each strategy with the smallest pool of money that can reasonably deliver realistic results. I carefully limit the exposure to this limited pool of money. This isn’t perfect because at small-scale transaction fees take a larger bite of the earnings which can make a viable strategy appear less successful but it appears to offer relatively lowest risk and most reliable approach. The amount placed at risk will vary depending on the total capital pool and the effective burn rate for the organization.
I have developed a strong suspicion of all internalized back testing systems because I have seen too many instances where they showed favorable results that could not be duplicated in the real world. I believe they are valid ways to test the optimizer and to measure the results from changes to ML algorithms. Most back-test systems seem to make a simplifying assumption which allowed a Strategy to cheat and deliver results different than what would have occurred during real order execution. This holds true for the back-test systems I have used popular systems produced by name brand brokers and my own.
By extension any project or company that shows only back trace results without actual live test trading should be approached skeptically.
I strongly believe the only trustworthy way to prove a strategy actually works is to force it to make predictions or recommendations against which actual trades are executed and then measure the results. The actual trading should be done on a system that is completely isolated from the ML trading or recommendation engine. It is only after this fully independent trading under true trading conditions with all of the plumbing and intermediate brokers involved that I would want to trust any Strategy regardless of the source. It is OK to take the recommendations and use them to manually enter trades in 3rd party systems or use a API to enter the trades in an automated fashion provided they go through the full order fulfillment pipeline.
Manually trading the strategy recommendations does involve more manual effort it can allow test trading to start sooner which provides time to market benefits and provides rapid feedback for agile development. I have also found that it can help identify and isolate problems fast.
I have modified this stance to allow us to use completely isolated virtual trading systems such as the virtual accounts provided by OptionsHouse, IB and Ameritrade where we can perform early strategy testing as if we are trading real money. This is a good way to gain a good feel for the performance of the strategy with minimal financial risk while minimizing places for accidental cheating. Unfortunately I have found that even these systems seem to cheat so their virtual order execution timing is not truly identical to the execution you get with real money which causes their results are likely to be different than those delivered from real trading. No paper trading system which allows an order to be back-dated or entered after the fact should be considered trustworthy when testing new strategies as they completely ignore the execution timing.
There is always a trade off between money placed at risk versus an organizations burn rate. Every trading organization has a burn rate that is consuming capital over time. Since you can only partially trust the results of paper trading you must still conduct real trading tests before ramping up the amount risk the paper trading time represents more accumulated expenses. Ultimately it becomes a ROI tradeoff and it may provide a better ROI to conduct real trading on a limited investment earlier in the test life cycle than would be intuitively obvious. Depending on the burn rate it may be worth considering real trading simultaneously with or in lieu of the paper trading.
Simply put the results from real trading is essential before risking larger amounts of money. Larger amounts of money invested are needed to produce sufficient profit so there is still a net-profit after paying for the accumulated burn rate. There are also time to market factors where the ability to deploy real money a few month earlier may be worth enough to risk real money by starting live test trading sooner rather than later. We normally start with virtual trading and then transition to live test money trading as soon as the virtual trading has shown consistent profits.
Forex, futures, stock, and options trading is not appropriate for everyone. There is a substantial risk of loss associated with trading these markets. Losses can and will occur. No system or methodology has ever been developed that can guarantee profits or ensure freedom from losses. No representation or implication is being made that using this methodology or system or the information in this letter will generate profits or ensure freedom from losses. Forex and Option trading can result in losses that exceed the original principal balance.
Hypothetical or simulated performance results have certain limitations. Unlike an actual performance record, simulated results do not represent actual trading. Also, since the trades have not been executed, the results may have under-or-over compensated for the impact, if any, of certain market factors, such as lack of liquidity. Simulated trading programs in general are also subject to the fact that they are designed with the benefit of hindsight. No representation is being made that any account will or is likely to achieve profit or losses similar to those shown.
Even results from Live cash trading can be subject to specific market conditions that may not repeat in the future and as such, duplicate results from future trading is unlikely to duplicate past results. Changing the dollar amount traded can cause different behavior in live trading markets especially when trading large positons that can exceed the liquidity available in the market and cause changes in pricing behavior.
Bayes Analytic LLC provides software that can produce trading signals. The customer is responsible for choosing a configuration and parameters for the software that meets their own goals. The customer is responsible for conducting their own tests and only the customer can activate the software to start trading. The software runs in an account the customer has logged into and then activated the software. Bayes Analytic has no control of, influence over or visibility to the signals specific to given user because we have no visibility into configuration parameters the user has chosen to operate with. The Bayes Analytic software is provided without Warranty on a As Is, Where is basis. It is the customers responsibility to test the software to ensure it meets their trading requirements. Every time Bayes Analytic releases a new version of the software the customer should conduct new tests to validate the new version continues to meet their requirements because every software change could have unexpected side effects that may not be obvious until the customer has tested it in their environment with their configurations. The Bayes Analytic software may run as a script inside of other software packages or talking to API that Bayes Analytic has no control of or Influence over so the customer should test entire ecosystem to ensure it meets their trading requirements. Bayes Analytic may provide the software in source form since that is required by some trading systems but it remains the exclusive copyrighted property of Bayes Analytic and may not be reverse engineered or redistributed. The customer is responsible for choosing their own broker and installing the Bayes Analytic software so it can trade using the desired account. Bayes Analytic has no control over or influence of the broker and many brokers have different ways of quoting spreads, charging commissions, flow of orders and latency of information. As such a strategy and software that performs well at one broker may and probably will require changes to perform well at other brokers. It is the customers responsibility to test the software with their selected broker to ensure it meets their trading requirements.