Wall Street Steward Blog

A Crock of Backtesting

Backtesting is a beautiful thing….or, it can be a horrible thing.  It depends on which side of the table you are sitting on.

In investing, backtesting simply means taking a set of investment criteria and then applying them over the course of history to see what “would have happened.”

For example, if we assume that these day trading programs are not a scam (they are), and they showed specific criteria on how they pick a stock, they could then test those selection rules for every calendar year going back to the early 1900’s. 

Why is this a bad thing you ask?  Because it is totally hypothetical, and it assumes that the strategies that worked well in the past will work well going forward.  “Past performance is not indicative of future results.”  In redneck, that means that “just cause dat dog barked last time don’t mean it will dis time.”  Just because an investment averaged 10% per year for the past 10 years does NOT mean it will do so for the next 10 years.  In fact, due to the cyclical nature of investing, sometimes a great track record represents investments that you should NOT buy.

Backtesting helps financial firms create track records that do not exist.  There are so many things that go into a buy or sell decision, and some of those things are not scientific.  Gut feel, intuition, the ability to improvise, etc. are very important.  Furthermore, you never know how you are going to react until you are in the situation.  Backtesting throws all of that out.  Essentially, it allows you to use your 20/20 hindsight and lie to yourself like this:

Obviously, I would’ve loaded the boat with long term municipal bonds in 1981.  I mean, they were paying 12%.  Any idiot would’ve done that.

Of course I would have sold EVERY single technology stock in early 2000.  I mean, we all knew it was a bubble that had to pop.  Clearly, I would have ridden the ride for the previous 5 years (1995-1999) and then had the courage to punt the tech stocks at the all time high.

I’m being sarcastic here, but I have seen “systems” that make claims such as these.  Backtesting even shows up on the infamous Ibbotson chart.  The chart shows the average annual returns for the S&P 500 from 1926-2011 as approximately 10% per year.  The S&P 500 was formed in 1950…so how did they calculate the returns for years 1926-1949?  Yep, backtesting.

The only real thing to do with backtesting is to completely ignore it.  Act as if you had never seen the data presented to you, and do not use it as part of your decision making process.  If the investment stands on its own merit, and you decide to buy it anyway, that is fine.  However, if you buy it because “look what it would’ve done in 2002 when the market was down 22%,” you are a glutton for punishment. 

I mean, look, backtesting is a ridiculous concept that would only be accepted in an industry as promiscuous as the financial industry.  Don’t believe me….watch this.

  • If we had just enacted the Patriot Act in 2000, we would have caught every major terrorist act, including 9/11.  And because it is in place now, we can guarantee that no terrorist will ever attack us again.

  • If we had mandated a roof and bulletproof glass on presidential vehicles, nobody would know Lee Harvey Oswald’s name, and JFK would have served out his presidency.  Of course, because we would have prevented that, this also guarantees that no other president would have been attacked since then.

  • If I had just decided to NEVER drive on rainy days, I would not have been on that road at all, and would have, therefore, not had an accident.  Since I have decided to not drive on rainy days in the future, this means that I will never have another auto accident.

  • If I had walked away from the blackjack tables in Las Vegas after being up 3 grand, I would have never lost it all back, along with a thousand of my own scratch.  Therefore, if I use that system from now on, I can take down the “house.”

Surely now you can all see how asinine these theories are.  To come up with some far-fetched ideas that would have helped IF they had existed AT THE TIME is “woulda coulda shoulda.”

Even explaining backtesting to an investor, I sound like an idiot.  If you have used backtesting in a sales presentation, stop and ask yourself if YOU would accept that theory if you were investing your own money.  I know that I would not.   

For the investor, before considering an investment, ask what role backtesting had in the creation of the track record.  Ask if the investment existed during the years it shows the performance.  Then, read the fine print no matter what the advisor tells you.  Like President Reagan said, “Trust, but verify.”