« VoIP Nostalgia | Main | Why Mary Doesn’t Have a New Dell »

October 14, 2007

Backtested Models – Hot Dogs

Way back in the early 60s I earned my tuition by programming the IBM 7090 in the college computer lab. This monster lived in an air-conditioned space the size of a basketball court and probably had less computing power than your watch, certainly less than your cell phone; but it was awesome for its time. When it was idle, usually in the middle of the night, I could sneak in some projects of my own. Having this unprecedented computing power available, I thought, certainly I can make my fortune at the dog track in Revere. I chose dogs rather than horses because eliminating jockeys made the problem simpler and lessened the possibility of fixed races defeating my algorithms.

What I needed to do was calculate the “actual” odds of each dog winning each race. Then, if the odds at the track just before the end of betting were much worse for a particular dog than I had calculated, betting on that dog was good strategy. Didn’t mean I’d get a payoff in any one race or even any set of races but, over time, should make money. But how to know how to weight all the masses of data available (or the data I had time to keypunch) to come up with the real odds.

That, of course, was where the computer would help. I fed it a mass of data for the past season. I programmed it to try one set of weighting factors after another until it found a strategy that overall made money if bets were placed at the actual track odds at post time. Having found a money-making strategy, the computer then made it better by tweaking the factors in small increments. Eureka! A particular strategy showed an expected 17% on money bet each racing day even given the various ways the track takes a cut. I was off to the races – literally.

Lost 50% of my stake on the first night. Made 5% the second; lost it all the third.

This wasn’t an adequate sample to test the theory; but I was out of money so had to go back to playing on the computer instead of at the track. Of course, could still simulate how I would’ve done if I’d actually been betting. Lousy, was the answer;  but this stimulated a new approach: I tested my method derived from last year’s results against the data from the previous year. That simulation showed a loss of 22% of the stake each racing day. Hmmm….

OK, scientific method says a theory is no good unless it predicts something that can be tested. I shouldn’t have trusted my model just because it fit the past. So I modified my technique to generate algorithms using last year’s data and then “predict” the first half of this year which had already happened. Aha..  That eliminated a lot of methods but one survived with flying colors. Meanwhile, I was working overtime at my regular night job so I had a new stake to take back to the track and make my fortune.

This time it took five nights to go broke. But, by the fifth night, the people who always stood in the same place I always stood, had noticed my green-and-white sprocket-punched fanfold computer paper. I explained my theory to them and boasted about my access to a computer (remember, rare and almost mystical in those days). They offered to buy printouts from me. I warned them that I hadn’t done very well yet but the authority of the fanfold paper was unimpeachable; they suspected I was holding back on them.

On the sixth night I had only enough money for subway fare and admission to the track but also had a sheaf of my computer-generated tip sheets. Sold out in almost no time and, as luck would have it, the tip sheet tips did pretty well so my circle of prospects grew rapidly.  Next night I brought a whole box to the track.

That’s when my first career in information processing ended. A very small and intense man with a very large but vague companion told me that I didn’t have a permit to sell tip sheets at the track. “Where do I get one,” I asked. “You don’t” is a polite rendition of what he told me. The large companion kindly disposed of the rest of my box of printout. I was escorted outside the track

But the lesson is really about backtested models. You can always find a correlation between two sets of data if you have a computer look hard enough. Trouble is, you can as easily find a correlation between unrelated datasets as you can between datasets where there may be some causal relationship (astrology, anyone?). If your algorithm (model) can’t make predictions, it’s worthless. If it can make predictions, you test the predictions. Testing can easily falsify your model but can’t actually prove it (a black swan could always show up in the next test). However, the more successful predictions a model makes, the more it makes sense to rely on it provided that you know that the stuff you don’t put into your model is going to be the same in the future as it was in the past (a big if).

As you may have suspected, this has a lot to do with predicting global warming. Stay tuned.

BTW, my predicting the unknown past was not equivalent to predicting the future. It was just another way of doing backtesting.

| Comments (View)

Recent Posts

Grapes of Wrath

Who Outed Jeff Bezos?

The Noes Have It

FireTVStick Thrashes at&t’s DIRECTV

An Invaluable Lesson in Colonial Williamsburg

TrackBack

TrackBack URL for this entry:
https://www.typepad.com/services/trackback/6a00d83451cce569e200e54ef75fa58833

Listed below are links to weblogs that reference Backtested Models – Hot Dogs:

Comments

blog comments powered by Disqus
Blog powered by TypePad
Member since 01/2005