tag:blogger.com,1999:blog-38600807.post6059233476312944139..comments2007-10-08T23:34:35.366-04:00Brian Burkenoreply@blogger.comBlogger9125tag:blogger.com,1999:blog-38600807.post-43051044481620799392007-10-08T23:34:00.000-04:002007-10-08T23:34:00.000-04:00

The thing to then watch for, year to year, are years where the error is significantly larger. You can figure out what the "expected error distribution" is by assuming the error truly is binomial (which is what you're presuming in the regression anyway, since it's a chi-squared fit), and doing a Monte Carlo.If the error is always within the expected error distribution, then you've got a model

Patrickhttp://www.blogger.com/profile/05228159984123927949noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-91888100900848815412007-10-06T11:04:00.000-04:002007-10-06T11:04:00.000-04:00

Patrick-Thanks. All great stuff I was not familiar with. I used some different software that can randomly select cases as training cases and validation cases. I posted the results in graph format to the original post.My interpretation is that you'd want to see two things. One, the test and validation plots are tightly intertwined. And two, they both follow the diagonal path tightly so as not to

Brian Burkehttp://www.blogger.com/profile/12371470711365236987noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-21042841886605741252007-10-06T04:07:00.000-04:002007-10-06T04:07:00.000-04:00

But how does one know what an expected accuracy would be?Run through the season. Predict each pair of games. The regression will give you some number that is related, somehow, to the probability that team A will beat team B (and obviously the probability that B will beat team A). You obviously know that conversion from the calibration. Average the larger of the two numbers for all games (the

Patrickhttp://www.blogger.com/profile/05228159984123927949noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-2637746111077029932007-10-04T06:43:00.000-04:002007-10-04T06:43:00.000-04:00

Patrick-Are you refering to "calibration?" I think we just have some differenct terminology. Here's how last year's calibration numbers looked.http://www.bbnflstats.com/2007/03/assessing-models-accuracy.html

Brian Burkehttp://www.blogger.com/profile/12371470711365236987noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-43015339344063465812007-10-04T06:27:00.000-04:002007-10-04T06:27:00.000-04:00

Pat-Thanks.Retrodictive--I couldn't remember that word.Help me out. You're saying my observed accuracy is 69.5%. But how does one know what an expected accuracy would be? Last year, this model (or one very close to it) was correct 65% of the time, and was well calibrated, i.e. 80% winners won 80% of the time, etc. But 2006 was a very odd year in which home teams only won 53% of games when they

Brian Burkehttp://www.blogger.com/profile/12371470711365236987noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-68115037950075612192007-10-04T02:34:00.000-04:002007-10-04T02:34:00.000-04:00

The word is "retrodictive", not "retrospective," incidentally.And the more interesting number in that case isn't the prediction accuracy - that's determined by the set of games used for the test - but the expected accuracy versus the observed accuracy (i.e. the error).The observed accuracy is mostly irrelevant without knowing the expected accuracy: if one model expected 60% accuracy, observed 70%

Patrickhttp://www.blogger.com/profile/05228159984123927949noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-44825610208739176282007-10-02T21:20:00.000-04:002007-10-02T21:20:00.000-04:00

Derek-Yes.

Brian Burkehttp://www.blogger.com/profile/12371470711365236987noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-64995914281146898652007-10-02T20:58:00.000-04:002007-10-02T20:58:00.000-04:00

By which I mean, did you test the 2002-2006 games on using the logistic regrssion model produced by training on the same set of games?

Derekhttp://www.blogger.com/profile/17941314072950152029noreply@blogger.comtag:blogger.com,1999:blog-38600807.post-80855003269063496362007-10-02T20:57:00.000-04:002007-10-02T20:57:00.000-04:00

By 69.5% accuracy retrospectively, do you mean that you're testing on games that the model was trained on?

Derekhttp://www.blogger.com/profile/17941314072950152029noreply@blogger.com