Pre-Season Predictions Are Worthless

tag:blogger.com,1999:blog-38600807.post-38143814435125546162007-09-02T23:10:00.003-04:002008-02-24T23:29:42.628-05:00Pre-Season Predictions Are WorthlessI recently came across online discussions regarding pre-season predictions of win totals for each team in the NFL. In this post I'll show why you shouldn't put much faith in any of the predictions you read before the first snap of the 2007 regular season. For comparison purposes, I'll use Football Outsiders' predictions and the Vegas over-under lines for the last two NFL seasons as representative statistical and consensus predictions. As it turns out, neither fare very well in their predictions. I'll judge the accuracy of predictions by mean absolute error (MAE). This is the average of the absolute value of the error. If the MAE is 3.5, it would mean the predictions were off by an average of 3.5 games in either direction. The smaller the MAE, the better the prediction. For the 2005 and 2006 NFL seasons the accuracies for Football Outsiders and Vegas betting lines are listed in the table below. <style type="text/css">.nobrtable br {display: none} table {border-collapse: collapse; border-width: 1px 1px 1px 2px; border-style: solid;} td {padding: 3px;} th {padding: 3px;} #colorcol {background-color:#ffffe0;} </style> <div class="nobrtable"> <table border="1"><col id="colorcol"> <tbody><tr bgcolor="#aad5ff"> <td>Year</td> <td>FO</td> <td>Vegas</td></tr> <tr> <td>2005</td> <td>3.0</td> <td>3.0</td></tr> <tr> <td>2006</td> <td>2.3</td> <td>2.3</td></tr> <tr> <td>Avg</td> <td>2.6</td> <td>2.6</td></tr> </tbody></table> </div><div class="linespacing"> Over the past two years, the Football Outsiders predictions are no better than the consensus Vegas lines. They are wrong by an average 2.6 wins for each team. Although the NFL is difficult to predict, 2.6 games is quite a bit in a 16 game season. How do we judge whether 2.6 games is a good or bad prediction? To judge if predictions are worth anything, we should compare them to obvious knowledge. I compared the FO and Vegas predictions to two sets of obvious predictions. The first is if I mindlessly predicted 8 wins for every team. The second is just using a regression of last year's wins. The resulting comparison of average errors is listed below.</div> <style type="text/css">.nobrtable br {display: none} table {border-collapse: collapse; border-width: 1px 1px 1px 2px; border-style: solid;} td {padding: 3px;} th {padding: 3px;} #colorcol {background-color:#ffffe0;} </style> <div class="nobrtable"> <table border="1"><col id="colorcol"> <tbody><tr bgcolor="#aad5ff"> <td>Year</td> <td>FO</td> <td>Vegas</td> <td>8 Wins</td> <td>Last Year</td></tr> <tr> <td>2005</td> <td>3.0</td> <td>3.0</td> <td>3.0</td> <td>2.9</td></tr> <tr> <td>2006</td> <td>2.3</td> <td>2.3</td> <td>2.1</td> <td>2.2</td></tr> <tr> <td>Avg</td> <td>2.6</td> <td>2.6</td> <td>2.5</td> <td>2.5</td></tr> </tbody></table> </div><div class="linespacing"> The average errors for '05 and '06 was 2.5 games for both the mindless 8-game predictions and last year's records. Both "obvious" methods actually do slightly better than the expert predictions. Pre-season predictions are completely worthless, at least those of Football Outsiders and Las Vegas. In fact, they're worse than worthless. One final note--If each division is ranked by their current Vegas over-under lines, the results are virtually identical to last year's final standings. The only exceptions are two ties. If you want to know how to take advantage of bad expert predictions, <a href="http://www.bbnflstats.com/2007/09/how-to-beat-over-under.html">read this</a>.</div> Hat Tip: Some data for this analysis came from <a href="http://footballpredictionnetwork.blogspot.com/">Football Prediction Network</a>. Original idea stemmed from the <a href="http://thesportseconomist.com/2007/08/nfl-predictions-writers-v-vegas.htm">Sports Economist</a> via <a href="http://sabermetricresearch.blogspot.com/2007/08/how-well-can-anyone-predict-nfl.html">Sabermetric Research</a>. End note: The regression of season win totals based on previous year wins yielded the following result (n=96): Next Yr Wins = 5.7 + 0.29 * Last Yr Wins. r-squared=0.12. Significance for Last Yr Wins was p=0.00.Brian Burkenoreply@blogger.com