Elo Ratings

The Elo rating system is a method of ranking players or teams in sports and games. It only considers wins and losses, and it ignores margin of victory. The system was originally created to rate international chess players by Arpad Elo, a physics professor who was himself a master chess player.

In a nutshell, the system estimates the probability one opponent should beat another. If an opponent wins more often than expected, his rating would improve, and vice versa. The algorithm needs to start with a prior expectation of how good each player (or team) is. Then, as the players complete matches, their ratings are adjusted upwards or downwards based on who won. The size of each adjustment is based on how significant the win was. For example, if a grand master chess player beats a novice, his rating would hardly budge, but if a novice beat the master, both ratings would move significantly.

The actual algorithm is based on the function below. E_A is the expected win probability of player A. R_A is player A's rating, and R_B is player B's rating.


After a game between opponents A and B, player A's new ranking (R'_A) is revised as:

where K is a maximum size of adjustment, and S_A is the actual result of the match. The K value has traditionally been 32 for chess, but it can be adjusted to tailor the system to various other games and sports. Ratings are typically set to have an average of 1500, but this is arbitrary and can be adjusted also.

For example, if player A's rating is 1655 and player B's rating is 1500, then according to Elo's function the probability A would beat B is 0.65. If player A defeats player B, then the actual outcome is 1.00. Player A's new rating would be:

R'_A = 1655 + 32 * (1.00 - 0.65) = 1666

One interesting way to look at the ratings is to create a generic win probability. By using the Elo algorithm to compute the expected win probability against a notional average rating, we can get a sense of each team's expected winning percentage.

Sagarin's Application of Elo

Jeff Sagarin uses a version of the Elo system to create NFL team ratings. He transforms them to produce ratings that are predictive of a game's point spread. So the difference between two opponents' ratings, plus an adjustment for home field advantage, predict the margin of victory. Sagarin's adjustment is a straightforward linear transformation of the original Elo system, as you can tell from the graph below. (I suspect Sagarin may over-weight recent games, however.)

Elo Mimicked

Using the same method as I described in my last post, we can mimic Elo ratings. That method computed team ratings based on margin of victory from each game. Instead of using margin of victory we can simply replace the score of each game with a 1 or 0 based on who won. Then we can solve for the ratings that best estimate the game outcomes. Because the ratings are linear we can transform them into individual game probabilities or generic win probabilities using a logistic transformation:


These rating systems can be adapted for any type of game or sport. Recently, on-line games have been using similar algorithms to rank players. The primary advantage to this type of system is that it discounts victories over very weak opponents. Often players will set up phony opponents to beat in order to inflate their own scores.

To get a sense of what these rankings would look like for the most recent (2007) NFL season, the table below lists several ratings for each team. The Elo column lists the ratings I derived from the actual Elo algorithm. The Sagarin column lists Jeff Sagarin's version of Elo--his final 2007 season ratings . Lastly, based on the Elo algorithm, the win probability column lists the probability each team would beat a league-average team on a neutral site. All ratings include results from the playoffs and Super Bowl.

Team	Elo	Sagarin	Win Prob
NE	2315	36.2	0.99
DAL	1913	29.89	0.92
NYG	1877	32.35	0.90
GB	1865	28.97	0.89
SD	1836	28.46	0.87
IND	1807	27.23	0.85
JAX	1692	25.62	0.75
WAS	1656	23.44	0.71
PHI	1624	23.63	0.67
TEN	1573	22.63	0.60
DET	1553	21.47	0.58
MIN	1523	22.22	0.53
HOU	1523	20.25	0.53
TB	1512	19.82	0.52
DEN	1490	19.82	0.49
CHI	1480	21.61	0.47
CAR	1453	17.93	0.43
SEA	1440	20.43	0.41
PIT	1438	18.64	0.41
NO	1429	17.42	0.40
CLE	1413	18.71	0.38
BUF	1384	18.24	0.34
ARI	1374	16.45	0.33
OAK	1309	14.34	0.25
CIN	1284	14.83	0.22
ATL	1262	13.34	0.20
KC	1254	14.71	0.20
BAL	1240	12.56	0.18
SF	1231	12.49	0.18
NYJ	1198	11.98	0.15
STL	1103	9.54	0.09
MIA	946	4.8	0.04

Homemade Sagarin Ratings

Since the early 1970s, Jeff Sagarin has been publishing sports team ratings. For most sports, including the NFL, his ratings are calculated so that the difference between two opponent's ratings, plus a home field adjustment, forecast a game's point spread. His ratings are widely recognized as some of the best around. They can be found every week of the NFL season on the USA Today site. Sagarin has never published his exact algorithms, but we can easily build a very good facsimile.

Excel has a powerful tool called "Solver." It's one of those thousand or so features that Microsoft packs into its Office products that no one ever knows about. In fact, you do don't even see it on the Tools menu until you enable it from the "Tools|Add Ins..." command.

If you go to Microsoft's on-line help site for Solver, the example problem provided is an exercise estimating point spreads for NFL games. The sample spreadsheet is for all the game scores from the 2002 season.

Basically all you do is create a table of ratings for each team. The ratings don't have to mean anything yet. For now they can be your best guess, or all ones, or anything. Solver will calculate them later. Then for each game, you calculate what the ratings suggest should be the point spread. The ratings are intended to work just like Jeff Sagarin's ratings. If team A's rating is 5 and team B's rating is 8, then when team A plays team B the point spread should be 3 in favor of team B. Factoring in a league-wide value for home field advantage, say 3, and the spread becomes 6 if team B is at home.

Next, using the LOOKUP function to grab the ratings from the table, you calculate the error between the expected spread and the actual result for each game. Square the error (as every good statistician would). In a cell, sum all the squared errors for all the games in the season. In another cell, enter a point value for home field advantage--3 points is a good initial guess. Solver takes over from here.


In the Solver dialog box, you tell it to minimize the value in the cell for the sum of squared errors. Then you tell it to do so by varying the values in the table for the team ratings and the cell for the home field advantage. (You can also add in a constraint that says the average for all the teams' ratings should be zero, so that good teams will have positive ratings and poor teams will have negative ratings.)

Solver will compute the team ratings necessary to best fit the actual point spreads. And now you have your very own homemade Sagarin ratings.

I noticed that Sagarin's average rating is 20 instead of 0, which makes sense because the average NFL score is about 20 points. So I altered the Solver constraint accordingly. For the 2007 season, including the playoffs, the homemade ratings were nearly identical to Sagarin's.

Comparison of Sagarin and MS Solver Team Ratings for 2007

Team	Sagarin	MS Solver
NE	36.4	37.7
IND	30.2	30.6
SD	30.0	28.8
GB	29.4	28.9
NYG	28.3	25.2
DAL	28.0	28.2
JAX	27.9	26.1
PHI	24.9	24.8
PIT	23.6	24.6
MIN	22.8	23.6
WAS	22.2	22.7
SEA	22.0	22.0
TEN	21.8	20.2
CHI	20.5	21.2
HOU	20.3	19.7
TB	19.8	20.5
CLE	19.4	18.8
DEN	17.5	15.8
DET	17.2	16.7
BUF	16.4	20.0
CIN	16.2	17.6
NO	16.1	17.4
ARI	16.0	16.2
OAK	15.7	14.3
NYJ	15.5	16.1
KC	15.2	15.1
CAR	15.0	14.3
MIA	13.2	11.7
BAL	13.0	14.6
ATL	10.2	9.9
SF	8.3	8.9
STL	7.1	7.8

The differences beween Sagarin's and our homemade ratings may come from the method of solving. Solver uses a "brute force" numerical iteration method, and Sagarin's method is unknown. Sagarin may also weight recent games heavier. Notice how the difference in the Giants' rating is one of the more significant. The Giants finished the 2007 season on quite a win streak.

Doug Drinen of Pro-Football-Reference.com discusses a very similar method for ranking teams based on margin of victory which he calls the Simple Rating System (SRS). His post includes a good discussion on the advantages and disadvantages of a pure margin of victory ranking system.

Sagarin actually uses two different systems. One is called Pure Points, which is based solely on point differential. This is the system which the method discussed above mimics. His other method is called Elo Chess, which considers only wins and losses, and ignores points. This system is based on a method devised to rate chess players by Arpad Elo, a physics professor and master chess player. In the next post I'll demonstrate how to mimic the Elo ratings.

The Ellsberg Paradox and 4th Down

The Romer paper and other research provide fairly conclusive evidence that NFL coaches should go for it on 4th down more often than they currently do. The Ellsberg Paradox might help explain why.

Say there are two jars of 100 balls of which some are red and some are blue. Jar A has 50 red balls and 50 blue balls. Jar B has a random unknown mix of red and blue balls. You'll be given $100 if you pick a red ball from a jar. Which jar would you choose to pick from?

In clinical experiments, people almost universally choose jar A. This is the Ellsberg Paradox, a violation of the utility theory in economics. The expected value of each choice is equal. There is a 50/50 chance of winning $100 from either jar, so we wouldn't expect one option to be significantly preferable to the other.

The Ellsberg Paradox demonstrates the difference between risk and uncertainty. Risk is measurable but uncertainty is not. People almost always prefer a known risk to an unknown uncertainty, even if the expected results are equal.

People prefer Jar A according to the equation above. U() is the utility function.

Punting seems a lot like Jar A, for which the risks and potential outcomes are known. Going for the first down seems more like Jar B, for which the potential outcomes are vague and hard to measure. So at the equilibrium point between going for it and punting, where each decision provides equal chances of ultimately winning, coaches would be heavily biased toward punting. Even beyond the equilibrium point, where going for it would be favorable, coaches would still be biased toward the relatively certain (but less favorable) outcome of the standard 40-net-yard punt.

In a strict analogy, the $100 would be a win, and the red balls would represent the probability of winning the game. There would actually be some uncertainty in each strategy, but far more uncertainty in the go-for-it strategy--perhaps something like 40 to 60 red balls in the punt jar and 20 to 80 balls in the go-for-it jar. The Ellsberg Paradox suggests coaches would naturally prefer punting, the less uncertain option. Only when the advantage of going for it is beyond obvious would a coach choose to go for the 1st down--say 10 to 20 red balls for punting and 15 to 60 red balls for going for it.

I think NFL coaches typically employ the maximin strategy. In game theory the maximin strategy is one that selects the alternative with the best worst-case-scenario. It maximizes the minimum possible payoff. This is a conservative strategy in comparison to the maximax strategy, which selects the alternative with the greatest maximum payoff.

Continuing the jar and red ball analogy, compare jar X with 10 to 90 red balls and jar Y with 30-40 red balls. Utility theory would suggest the rational option is jar X with a higher overall chance of success. The maximin choice however, would be jar Y because it has a higher minimum chance of success.

Calculating the probability distributions of a football game's outcome given the combinations of score, time remaining, field position, etc. is far more complex than being told how many red balls are in a jar. It would be overwhelming for a human brain even to attempt it. In such a situation, coaches, like everyone else, use heuristic shortcuts such as the maximin strategy. Punting on every 4th down is a known risk, especially because coaches can count on opposing coaches to follow the same strategy (which suggests that always punting is a Nash Equilibrium). Punting usually presents the best worst-case-scenario despite being a sub-optimum decision.

How Draft Experts Are Graded

It seems that everyone has a mock draft board these days. The object of this parlor game seems to be to predict which players will be picked by each team. So draft gurus tend to be judged by how many correct predictions they make. When you think about it, it's pretty ridiculous. A draft expert has to get four things right. He needs to not only evaluate players and team needs, but also evaluate all 32 teams' own perceptions of each player and its needs. And if one prediction at the top of the first round is off the mark, the house of cards collapses.

I think a much better way to evaluate draft experts is to wait several years, then see which players actually turned out to be more productive. Look at how they rated each player, not how well they read the minds of the league's GMs.

Besides, in my eyes the real value of a draft expert is be able to tell me in 30 seconds everything there is to know about that strong safety from Alcorn State my favorite team just picked up midway through the 4th round...without any notes or any hesitation...and has a full head of hair. And for that, there is only one man.