Belichick Cheating Evidence?
One of the more apparent signs that a spy or corrupt official is cheating is that he is living a lifestyle beyond his means.
If Belichick's Patriots exploited unfair advantages in stealing signs from opposing sidelines we would expect to see some sort of evidence that they won games "beyond their means." By means I am referring to the Patriots' passing and running performance on offense and defense.
By successfully exploiting stolen signs, we might expect the Patriots to choose to use that advantage on critical plays--3rd downs in the 4th quarter for example. These critical plays would heavily "leverage" performance on the field to be converted into wins. In other words, the Patriots would win more games than their on field stats would indicate.
This is exactly what we see in the data. Year-in and year-out, Belichick's Patriots have won about 2 more games than expected given their offensive and defensive efficiencies, including turnovers and penalties. No other modern team has even come close to the Patriots in consistently winning more games than their stats indicate. Could those extra wins be due to cheating?
For a comparison of other teams' actual/expected wins charts, see this article. When I first discovered this pattern, I believed it was evidence of Belichick's in-game "genius." He was known for some unconventional tactics, such as going for it on 4th downs more often than his counterparts. (I'd go for it on 4th down too if I knew the play the defense would run.)
In a previous article, I ranked every head coach since 1983 in terms of how many excess wins they had above their expected wins based on team efficiency. Belichick's career, as a whole, was rather average due to his poor results in Cleveland. But by isolating his tenure with New England, his excess wins per season would rank him as the best tactical coach ever, with an extra +2.33 wins per season. The rest of the pack is far behind with the 2nd best coach at +1.83 wins per season. Belichick is a true outlier, at almost 3 standard deviations above the mean.
This is only circumstantial evidence of cheating, but it is evidence. And although hardly damning, we can be sure of one thing about Belichick--he is willing to cheat. If someone has crossed the line by breaking one rule, what makes you think he's not willing to break others?
On some other sites I've read some emotional comments rationalizing the Patriots' cheating. Whether other teams have attempted this or not is not relevant. Patriots fans and Belichick fans must accept the possibility that much of their success has been due to cheating. What I've done here has added weight to that possibility and quantified the potential scope of this scandal. At the very least, we would be justified in continuing to investigate the Patriots' methods.
Note: Some other examples of teams' expected vs. actual wins can be found here. Also, since this post is appearing on many other sites recently, I've posted a response to many of the excellent comments and criticisms over the past several months.
36 comments:
I understand that you're not offering this as any sort of definitive evidence, but for argument's sake...
If it's true that the best teams tend to out preform their stats due to the short season; and if it's true that Bellichick's Patriots are one of the best teams, then this result isn't unreasonable.
But this assumes that the Pats are great independent of any cheating.
Have you (or anyone else) looked into the reasons why teams out preform their stats (be it just luck or some controllable variable)? I wonder how possible it is to isolate any evidence of cheating. My guess is not very.
Not sure that makes any sense or is even interesting.
Regardless, I agree with your conclusion.
Pris-I share the same interpretation. It could be due to greatness, cheating, or luck--or a combination of any or all of the above.
But, Belichick's score is SO high, at an amazing 3 standard deviations above the mean, and the result is SO consistent, it's hard to believe that just luck and skill are at play, given the fact we know he's willing to cheat.
It's possible cheating had no effect, but why do it when your team is the odds-on favorite to win the Super Bowl and you're the world's best coach?
VERY - interesting!!! And I've forwarded this link to all my outspoken Patriot's fan friends. A thought that came to mind is that this analysis doesn't incorporate turnovers in as much as a team with a high positive turnover ratio might have a shorter field to score and their points/yds ratio would be higher while their opponent's points/yards ratio would be lower since many drives would have ended with a turnover.
Of course, one way to increase your turnover ratio is to know what the other team is going to do so you would need to find a way to normalize for this which may be impossible.
tb-It would take a month to find all the turnover data going back to 1983, but it would help. But for coaches with longer careers, the turnover factor will tend to even out.
But, I do use all turnover data for recent years. So the graph in this post that illustrates how Belichick's Patriots have overperformed their stats by 2 wins per year does include turnover data.
I don't have time to post all the other team's graphs (with turnover data). But trust me, no other team has a profile anything like the Patriot's. (Except maybe the Lions, who tend to lose more games than they should.)
Embarrassingly bad piece. Since they were stealing defensive signals, then that would benefit their offensive numbers across the board.
So why wouldn't this show up in your stats?
Feeble.
To the anonymous Pats fan-
1. The article does not assume that the Patriots rule-breaking activities were limited just to videotaping opponent's signals, and so the advantage would not be limited to improving offensive stats.
2. The article points out that the Patriots would likely use their knowledge of opponent's defensive calls in high-leverage situations--4th and short, for example. So if It's 4th and 2, and the opponent calls a run blitz, the Pats could call a WR screen. If the play picks up 4 yds the Pats get a first down, but their average yds per pass would actually suffer.
Please read the entire article before throwing out words like embarrassing and feeble.
I'm curious how you would explain the Patriot's over-performance? The possibilities are:
a. They are the luckiest sports team in the history of all sports.
b. Belichick is not a cheater, but a football genius the likes of which come along once in 10,000 years.
c. They cheated sometimes.
We know one of the above is true already.
Since most of your "statistics" here are based on an assumption of cheating then why not assume that all other teams cheat in the same way, only not from on-field cameras. That's the only difference, an on-field camera versus one from the stands or box. Therefore, if all teams steal signals then your assertion of Belichik's over achieving is negated. What teams don't "scout" their opponents? When coaches look at films, what do you think are they looking at - vacation movies? When coaches speak behind their play sheets is it because they're all afraid THE BILL will be watching? Seriously, all coaches fear Belichik will be watching? Even the ones who will never play against NE? You're taking one exposed instance of what happens in every NFL game by every NFL team and blowing it into a big balloon filled with hot air. New England wins because the players are better prepared to play 60 minutes every game, especially in a very complex defense. They also have the best QB/receiver(s) combination in the league. That's why they win, not because they cheat.
Why not just report and work with fact as they are instead of employing such useless speculation. If you're going to factor in drivel why not use blown calls by the officiating crews? How many of those have gone against NE compared to other teams? Do you have those stats? Everyone who has eyes can see those and it's not just left to speculation; so, they should be given more weight in your various equations than a supposition of spying. I doubt if you'd take that route – it's not in the interest of your hypothesis that NE just can't be good but must win by cheating. In the game of unbeaten teams, the Colts ran well in the first half and then were stopped in the second. The Patriots must have cheated better in the second half accoding to your theory. Phantom interference calls on both offense and defense against NE kept the Colts in the game, not the Colts outstanding play. Manning was shut down and then Addai was shut down. Brady again proved he's the best late-drive QB in the game. Is Belichik a better coach than Dungy? Yes, in this game, because the Colts had no answer for what NE was doing (Brady) and couldn't make second half adjustments and close the game. But you would put it down to cheating.
Stick to real facts and numbers and leave the speculation to the idiots on the pre-game shows. They get paid to be stupid.
You're obviously a very loyal Pats fan and want to defend your team. I get that. But no speculation is needed to know that Belichick cheated. It's just a matter of knowing how much and measuring the effect, right?
The only speculation here is when people suggest everyone else cheats too.
All I did here is show that NE consistently wins more games than we'd expect given their efficiency stats. No other team is remotely close to that record over the past several years. It's just evidence, not proof.
A simple test of your cheating hypothesis is to repeat the analysis in another couple years and see if there is a statistically significant decrease in their relative performance in that late-game, high-leverage situations you've focused on.
If cheating was responsible for their over-production, then you'll see a significant decrease. If it was coach/team greatness then you won't.
Of course, this is nullified if something traumatic happens like a major Brady injury, or something else that seriously confounds the study.
If the cheating has a positive effect for the Patriots, it affects their play on the field...so wouldn't the positive result of the cheating already be factored into their efficiency stats?
Only if they're using the advantage most of the time. My guess is that they're reading defensive signals (and possibly others) and using them mostly on very high-leverage plays--plays that don't have a large impact on their stats but have a very large impact on the outcome of the game. If it's 3rd and 5 and they get a 6 yd pass, that doesn't help their passing stats much, but it keeps the winning drive alive in the 4th quarter.
That would explain the "excess" wins above what would be expected given their on-field stats.
The only speculation here is when people suggest everyone else cheats too.
My guess is that they're reading defensive signals (and possibly others) and using them mostly on very high-leverage plays.
Look at the first synonym in this dictionary entry.
This article is nothing but speculation. That's fine, but you should admit it. I mean there is definitely no evidence given for the "possibly others" remark.
In your previous post, the Lions we far greater outliers than the Pats most seasons. Did they reverse cheat? Are they the unluckiest team ever?
I think the part that is specualtive is the assumption that the information gleaned from the cheating was only used at certain times--enough to affect wins and losses but not enough to affect efficienct stats, thereby nicely fitting this thesis. Maybe this is how it went down, but at this point it's definitely speculation.
Doug-I think that's a fair point. I might characterize it as a hypothesis supported by at least some circumstantial evidence, and not as pure speculation. My use of the word "guess" wasn't wise. And Vaclav, that's incorrect about the Lions. The Pats are more consistent outliers, but I do think the Lions may be poorly run from top to bottom.
My response about "the only speculation here" was related only to the previous rant from the Pats fan going on about unfair refs and the "every other team does it" argument.
I don't mean to be lecturing people here, but that's how science and research work. People "speculate"--have theories--then test them with observation. If the observations support the theory, that's one more reason to believe it.
Here's how I came to the theory:
1. The Patriots did cheat. They were caught, warned, but continued. It's difficult, time consuming, and risky. From that, we'd be safe saying the filming had a purpose and a value. The Patriots themselves must have believed the practice gave them some kind of advantage.
2. We know the Patriots' performance in 2nd halves were not significantly better than in 1st halves (see the related FO article). And we know that their performance against teams they faced twice in a season was not any better in second games than first games. It follows that they probably didn't systematically use an unfair advantage *all the time*.
3. Accordingly, if they did gain an advantage (which they themselves believed), perhaps it was only applied in certain critical situations. The alternate (null) hypothesis is that they never gained an advantage--the Pats did all that risky work that was ultimately fruitless.
I then remembered my look at "gameday" coaches and how Belichick's W-L record was off-the-charts above and beyond what you'd expect given his teams' on-field performance. I put 2 and 2 together and the numbers appear to support the theory.
Thanks for keeping me honest.
This still makes major assumptions.
1- The equal applicability of your statistical methods across every team. If an agent acts in such a way that the system does not accurately predict outcomes, its not logical to assume the system is correct and the agent is breaking assumed rule sets in reaching outcomes outside your predictions. Since you seem to have a yardage based efficiency system, the most obvious point is the Patriot's tendency to have a bend-but-don't-break system. The inability of your projection system to account for this is more an indictment of your system than the agent that is outside your norm. When the Colt's broke the DVOA stats at FO, they didn't assume cheating, they reworked their system.
Let's look at the Football Outsider's win projections/Pythagorean wins.
Year Estimated Wins Actual Wins
2006 11.0/12.2 12
2005 8.9/9.1 10
2004 13.1/12.4 14
2003 11.6/11.4 14
2002 9.3/8.9 9
Only in 03 and 04 is there a variance of greater than 1 game, and any system that estimates wins is going to tend to have a greater error rate, usually towards the mean, with the great teams.
Jake-Great comments. I agree with you on one point and disagree on the other.
First, I agree that we can't just say any residuals/errors not accounted for by a model can automatically be chalked up to cheating. My original theory was that part of the residual was luck, part was coaching, and part was due to other factors not accounted for by the model (officiating for example). With enough seasons for each coach, the luck and 'other factors' would tend to average out, and we'd be left with the additional wins due to 'gameday' coaching.
In fact, I did this research weeks/months before the Jets game cheating was revealed. Belichick's tenure at NE stood out far beyond any other coach in terms of beating their win expectations (3 standard deviations above the mean). At first, I attributed the results to his coaching genius--going for it on 4th down more often than other coaches for example. Then the cheating revelations hit and made me consider if a lot of those 'extra wins' could be due to an advantage gained by the cheating techniques employed by the Patriots.
So I didn't have a result and "assumed" cheating. I had a noteworthy result, then cheating was revealed, then I examined how the cheating could explain the prior result.
On the point about FO DVOA, I'm less inclined to agree. DVOA is an ad-hoc play-by-play scoring system, not a statistical regression model.
It's clever and novel, but its results are severely over-fit. Their system would be blind to a cheating outlier because DVOA assigns bonus points for success in high-leverage situations. If the Patriots indeed used unfair advantages in crtical situations, their successes would be captured inside the DVOA system. So to DVOA it would always appear that the Pats are winning as many games as they "should."
Regarding the point about the Colts, I was always taught that adjusting a model to fit an outlier was the opposite of good model construction. Outliers happen. The extraordinary thing about Belichick's Patriots is that they were consistently the biggest outlier every year. The model works really well for 31 agents and the 1 agent that defies it has been caught cheating. I'm not so sure that's an indictment of the regression.
Here's an analogy to consider. There's a kid in high school who consistently gets B-s on all his quizzes, homework, and assignments. But you notice he gets A+s on all of his final exams. Later, it's revealed he was caught trying to bring crib notes into some exams. Isn't it reasonable to think that the other A+ exams are due to cheating? Is that an "assumption?"
An interesting point I didn't see mentioned is that the second highest 'over-achiever' in terms of wins is Tony Dungy and the Colts. Granted it's not as much as an outlier as the Patriots, but it seems significant.
1) A cynic would argue perhaps the Colts were cheating too.
2) Another could argue that dome teams have an inherent advantage in 1/2 their games that over time leads to some over achievement. Perhaps due to crowd noise, real or artificial.
3) Or perhaps having one of the best QB's in the history of the game tends to lead to more wins than expected because QB's have a higher than normal direct effect on the outcome of game.
Just a thought. Both Brady and Manning are generally considered to be in that team picture of greatest QB's on their current career paths.
The two coaches that leap far ahead of any established level of 'excess' wins per year happen to have the two best QB's of the pass heavy era. And one of those coaches drops well down the list if you include his non Brady years.
nater-Very good point. I'd completely agree, but I don't think NE's pass offense was all that spectacular until this year. In 2006, it was very average but they still won +2 games above what a similar team would be expected to win.
Also, here is a link to more 'actual/expected wins' charts for other teams. IND is included, and you'll see that they don't really exceed their expected wins at all.
http://www.bbnflstats.com/2007/06/next-years-wins.html
Completely unconvincing. What does the point about using the information in critical situations have to do with anything? The Pats might outperform their statistics for a variety of reasons including competence, training, and ability to perform in the clutch. Surely the 16-0 exceeded expectations too, post cheating. What a load!
Ability to perform in the clutch? Are you suggesting the Patriots are so good they only have to try when it really matters? Give me a break.
Patriots fans--please hide your bias better!
Brian,
Have you ever considered writing a piece on say, a 38 year old quarterback performing at a level higher than 90% of the league and suddenly playing his best season in 12 years?
It seems like another statistical outlier and one that can be backed up with circumstantial evidence (38 year old pitchers who shouldn't be throwing 100 mph fastballs suddenly getting exposed as steroid users perhaps?). But why would you write such an article?
I mean there IS a possibility that Brett Favre is cheating in the unholiest of ways (you know, the kind that requires Congress get involved?), but there is ALSO a possibility that he is just a great quarterback and a great quarterbacks can sometimes accomplish incredible things.
Brady and Favre will go down as two of the greatest quarterbacks to ever play the game and while there are possibilities of improprieties on their parts and the parts of many other legends, I'd prefer to watch and enjoy their mastery of the position. Not to say that I'd like to turn a blind eye to potential cheating or even go through my football watching life in blissful ignorance; but I'd prefer to believe in the magic of the game and in the players than in the numbers that surround them.
Reading the article and all the comments, I still don't understand the rationale of the Pats' cheating making enough difference to affect wins and losses but not enough to affect Brian's efficiency stats.
Joe-If they used any advantage only in high-leverage critical situations, the efficiency stats wouldn't be affected nearly as much as the win/loss record.
Say NE has the ball down by 4 in the 4th quarter. It's 4th and 2. What if the Patriots know the defense's call to overload the left side, and counter with a 3 yd run to the right? NE probably goes on to win, but their offensive running efficiency isn't really affected.
Brian,
Didn't you use this exact same formula to predict that The Ravens would be a "solid, above-average team in 07"?
Here is what I said (and since then I've learned that pre-season predictions are a waste of time).
"They only real difference was the quarterback--Steve McNair replaced Kyle Boller. The biggest changes in team fortunes are due to quarterback changes. We can also note that although Baltimore won 13 games, they had about 10 expected wins. This tells us not to hold our breath for another 13 win season in Baltimore, but still count on a solidly above-average team."
Unfortunately, the Ravens were forced to revert back to Boller after it became apparent McNair was hiding a pretty serious injury. (Note the sentence above about quarterbacks.) Injuries to their Hall of Fame left tackle, #2 pass rusher, and both all-pro starting cornerbacks may have had something to do with their decline as well.
I think my main point was that in 2006 the Ravens were really a 10-win team that got a little lucky, and not to count on another 13-win season. Any prediction for 2007 should start with 10 wins rather than 13, and then be adjusted for schedule/personnel changes/etc.
Sorry that I don't have a regression model for predicting groin pulls and turf toe!
I need to make a correction to a comment above. I said "Outliers happen. The extraordinary thing about Belichick's Patriots is that they were consistently the biggest outlier every year."
In some years there were teams with a bigger difference between expected and actual wins than the Patriots. The extraordinary thing is that the Patriots were consistently an outlier on the positive side, without any neutral or down years. But over all years, they were the biggest outlier by far.
"Joe-If they used any advantage only in high-leverage critical situations, the efficiency stats wouldn't be affected nearly as much as the win/loss record.
Say NE has the ball down by 4 in the 4th quarter. It's 4th and 2. What if the Patriots know the defense's call to overload the left side, and counter with a 3 yd run to the right? NE probably goes on to win, but their offensive running efficiency isn't really affected."
Brian,
The part I don't follow is why the the advantages they got from cheating would result in the type of play you describe rather than an 80 yd TD pass for example. Or a mix of both types of plays.
From 2002 to 2006, the Patriots averaged about 2 wins above the expected number of wins. Between 2003 to 2006, their wins-above-expected was consistantly between 2.0 and 2.5. The author of this article hypothesizes that this anomaly was due to their demonstrated habit of illegally videotaping opposing defensive signals. In the first quarter of the first game of this season, the Patriots were forced to cease that habit. All their previous tapes of defensive signals were confiscated. Presumably, they were under close scrutiny by the league all year to prevent them from pulling similar stunts.
And yet they finished the 2007 season 2.1 wins-above-expected, just like 2003 to 2006. Changing the input variable (taping vs not taping opponents defensive signals) did not change the outcome (number of wins-above-expected).
So you could conclude:
a) Videotaping opponents defensive signals did not give them an unfair in-game advantage and is not the factor that explains the anomalous win data (perhaps they were simply storing the info for future scouting purposes, as a record of each opponents defensive play calling tendencies vs down and distance - somewhat more accurate than analyzing the game film)
b) They were somehow obtaining the same knowledge of opponents signals by other means and using it to regain the same unfair advantage, despite the close scrutiny by the league.
c) They had and continue to have other methods of cheating that the league does not know about which gives them an unfair advantage. Defensive signal stealing was only a small part.
Occam's Razor points me to a).
I'd have to agree, LK. The over-performance again this year would be evidence against cheating as the reason for previous over-performance.
Then again, this is the biggest outlier season of all outliers. It might not be a good counter-example. It also makes the "luck" theory even more astronomically improbable.
We're left with these possible explanations:
1. genius
2. randomness (luck)
3. some quality about Brady
4. cheating
5. flaw in the model
6. some combination of the above
I'd have to say if 5 is true, then the real answer is 1. The other 31 teams all conform to the model relatively well.
Brian,
One last comment...
Two facts we know: 1) the Patriots have won games consistantly above their model-predicted wins in each of the last 5 seasons (well beyond expected random variation, apparently), and 2) they were caught illegally recording opponents defensive signals in one game this season. It seems reasonable to speculate that they also videotaped in previous games over the years (although, to be honest, we really don't know if they did, or how long they had been doing it, or if other teams also did it.)
But if we accept the reasonable premise that the videotaping had been going on for several years and that no other team was doing it, then there does appear to be a correlation between videotaping and winning above expected wins.
However, as a statistician, you know that correlation does not imply causation. I think another hypothesis that fits the data is this: Bill Belichick is an obsessed, focused, single-minded coaching genius. A symptom of that obsession is the desire to record and have at his disposal every available bit of information, even if recording that information violates the letter of a league rule. The videotaping was not the cause of Belichick's success, but a side effect of his obsessive coaching style.
Watching the Patriots over the years, I've noticed that Belichick and his team do a lot of the little things extremely well: impeccable clock management; bold, creative, and unexpected play calling on both sides of the ball (such as going for it on 4th down instead of punting); excellent discipline in football fundamentals (such as falling on fumbles instead of trying to pick them up and run); capitalizing quickly on the misfortunes of their opponents (such as a bad call by the refs or a fluke turnover), while recovering quickly when those same misfortunes befall them. All these things might not show up in offensive or defensive stats, but could add up to winning one or two close games a season, where a poorer-coached, less disciplined team would lose.
One last, last comment...
If you get a chance, compare Ted Williams' famous 1941 batting average (.406) versus the average and standard deviation of the rest of the league that year. Among the 136 players who could qualify for the batting title, Williams' BA was FOUR(!) standard deviations above the mean. This means that, even if there were 15,000 players in the league, Williams would STILL have the highest BA.
Phenomenal outliers do sometimes occur, even without cheating!
LK-You're dead right about the correlation/causation. But there is another possibility--it could be a little of both.
I agree on the going for it on 4th down. I am working on a big project on just that subject for the off-season. If you read the original 'gameday' coach I said some of the same thing.
And no surprise about Ted Williams--he was a naval aviator after all.
Brian, (no relation)
I have a few bones to pick.
1) You state that the reasons the patriots won were:
a) Luck
b) good coaching
c) unknown factors like cheating
Why not add d) Great players? Underrated players like Mike Vrabel, Tedy Bruschi, and Troy Brown. Then you have pro-bowlers like Ty Law, Seymore, Rodney Harrison, Lawyer Milloy, Willie McGuinnest.
2) Since when are a team's projected wins *ever* accurate? Every year there is a surprise team in the superbowl. Every year there are a few teams that surprise everybody. When have the analysts *ever* been very accurate with their predictions? What a horrible use of statistics. Basing analysis on people's opinions.
3) As others have pointed out, turnovers play a huge role in the success of a team. For some reason or another, every surprise team each season has a huge + turnover ratio. Bears 2006, Falcons 1998 are the ones that come to my head.
4) Using offensive numbers is one thing, but a better comparison is points against for a defense rather than yards given up. IN each of the past 7 years, Patriots have been near the top of the list and points allowed.
5) In 2001, the Patriots had incredible special teams which turned a number of games (AFC championship game for one!). Please tell me how exactly do you cheat with special teams?
6) In 2003, the Patriots had an incredible defense. Ted Washington archoring the middle Ty Law, etc, etc. etc.
7) Then, 2007. With the NFL, media, and fans having the Patriots under the microscope, how exactly did they cheat with 16 wins? A QB that had record in touchdowns with what, only 10-11 interceptions? That's just crazy shit.
8) If you want to talk about *real* cheating, how about the teams that cheat in free agency and the salary cap. I talked a little about this here in where's the outrage?
flawed argument from the start. Stealing signs is allowed. The rule broken was using a camera.
Stealing signs by other means even you guys can understand.
http://sportsillustrated.cnn.com/inside_game/magazine/life_of_reilly/news/2002/01/09/life_of_reilly/
offensive signals in 2002, but same idea.
"Flawed argument?" Bill Belichick might disagree with you. Taping signals was important enough to him to risk fines and possibly a suspension. Remember, he was warned previously after being caught.
Neat article though. I got HD a year ago and was amazed how easy it was to read coaches' lips. During last year's Navy-Note Dame game I could read Charlie Weiss' calls, even the snap count. "It's on three." Too easy.
And if you watch the NFL Films replay of Super Bowl XXXVI (Rams/Patriots), you can see a sideline discussion between NE defenders of what could only be the Rams' offensive signals.
Stealing signs, and doing so consistently, would obviously give a team a clear advantage.
I no statistician. I'm not even good with math. But wouldn't the team that makes it to the super bowl have to win a couple more games than "expected" by converting on crucial plays? Isn't that how you differentiate yourself in competitive sports? And wouldn't the fact that the patriots have made it to the super bowl half a dozen times since the late 90's mean they had to win more, throwing off their curve of wins relative to the rest of the pack?
I could show you a mean graph depicting how a team's winning percentage in football is inversely proportional to the length of their coaches hoodie sleeves - and it would be just as true.
Post a Comment