Sonas: Assessment of the EU performance calculation

Assessment of the EU performance calculation

By Jeff Sonas

With almost 400 players competing at the recently-concluded 2011 European Individual Championship in Aix-les-Bains, France, including 85 players rated 2600+, it is no surprise that the final leaderboard was extremely crowded at the top. Four players tied for the overall lead scoring +6 (8.5/11), eleven players finished half a point back with +5 (8/11), and there was a big group of 29 players on +4 (7.5/11). The exact final ranks were not determined by rapid playoffs, but by a series of tiebreaker calculations, the first of which was performance rating. The performance rating calculation can be fraught with controversy, and at least one player has already lodged an official protest against the final standings.

Because there was so much at stake in the performance rating calculation here (including determining the individual European Champion as well as a number of qualification spots into the World Cup) I thought it might be interesting to take a closer look at the calculations and the resultant numbers.

Part 1 – Reading the rules

Of course we ought to start with the official rules for the tournament itself. Section 8 clarifies that (per ECU Tournament Rules D3.2) there are no tie-break matches, and that the tiebreaker rules will follow section B6.2 of the ECU Tournament Rules, namely:

Note that if you had an unplayed game (such as a forfeit or a bye), it mentions "the current FIDE Tournament Rules". I think it is referring to this part of the FIDE Rating Handbook, where it simply says that in the handling of unplayed games for tie-break purposes, the result shall be counted as a draw against the player himself.

Okay, fair enough. So if players finish on the same score, we will break the tie by calculating a "Performance Rating", where the highest and the lowest rated opponent will be deleted and the maximum rating difference of two players shall be 400 points. Unfortunately there is no further detail provided in either the Aix-les-Bains or EU regulations regarding the exact method for calculating performance rating.

Part 2 – Deciding how to calculate performance rating

So with all of those ties at the top, and 23 qualification spots available for the World Cup, the specific details of this performance rating calculation are quite significant. Unfortunately there are a few different ways to calculate performance rating; let's look at some of these.

(2a) Simple Performance Rating

The simplest approach, and I think the most generally accepted one, is to treat all of a player's games identically, where we average the opponent ratings and then add a modifier, taken from the standard Elo table (see section 8.1(a) here) for converting from fractional score to rating difference. That follows the recommendation provided by FIDE (see section 5. Tiebreak Rules using Ratings). As an example let's look at the four players who finished on +6:

So by this simple measure, the final standings would have been #1 Potkin, #2 Wojtaszek, #3 Moiseenko, and #4 Polgar. However, the actual standings had Polgar finishing in third place, and Moiseenko finishing in fourth place. Why was this? It is because this particular tournament used a different performance rating calculation.

There is a concern that extreme rating differences can skew the performance rating too much. It is easy to construct examples wherein a player's performance rating can go down despite a victory, because their latest opponent's rating was so low, or conversely that it can go up despite a loss, because their latest opponent's rating was so high. For instance if you start a ten-round event by scoring 50% in 9 games against 2200-rated average opposition, that is (of course) a 2200 performance rating through 9 rounds.  If you then defeat a 1200-rated player in the final round, it yields a 55% score against average opponent strength of 2100, for a performance rating of 2136 (less than 2200 despite your victory). Conversely, if you had instead lost to a 2700-rated player in the final round, that gives a 45% score against average opponent strength of 2250, for a performance rating of 2214 (more than 2200 despite your loss). These examples, though they are somewhat artificial, lead to the desire to "blunt" the impact of extreme rating differences, such as the approach taken in the Aix-les-Bains tournament.

(2b) EU 2011 Performance Rating

The key difference with Aix-les-Bains comes from that special clause "the highest and the lowest rated opponent will be deleted". Instead of an eleven-game percentage score and an average of eleven different opponent ratings, the calculation used for this tournament identifies each player's highest-rated opponent, and lowest-rated opponent, and ignores both games. Thus it effectively is looking at a nine-game percentage score, and an average of nine different opponent ratings, based on the selected subset of nine games. So instead we get this:

You can see that for the first three players, the impact of removing the games against highest-rated and lowest-rated opponents had the effect of removing 1.5/2, leaving a 7/9 (78%) score whereas for Moiseenko, it actually removed 2/2, lowering his score to 6.5/9 (72%). This has such a large impact that it hardly matters what you calculate for the average opponent ratings; Moiseenko already has a gap of more than 50 points to make up (since the others will get +220 for their 78% scores and he will only get +166 for his 72% score). And ultimately his performance rating under this scenario (2755) is more than 40 points behind the other three.

You can go through all four players and point to small variations that would have made a big difference to their performance rating calculations. Potkin's opponents in rounds 8 and 10 were within five Elo points of each other; had their ratings been switched then it would have been Potkin who had 2/2 subtracted from his score, and he would have ended up in 4th place instead. Wojtaszek's first two opponents were rated 2423 and 2422; had those been switched then he would have only had 1.0/2 subtracted, leading to a fabulous 83% score from the remaining nine games and an easy first place. The same thing for Polgar; her first two opponents were rated 2415 and 2409; had it been switched then she would have only had 1.0/2 subtracted and she would have won first place. And for Moiseenko, until the final round his two strongest opponents were both rated 2626 (one of which he beat and the other he lost to) so it would have been a further tiebreaker that would have left him either having 2/2 subtracted from his score (and fourth place in the tiebreak) or having 1.0/2 subtracted from his score (and likely first place).

I don't like this approach. I think it is too discontinuous, too dependent on tiny rating point differences as we have just seen. And I find it quite unfortunate that in an attempt to compensate for perceived problems in averaging opponents' ratings together, the result was a calculation that was dominated by the outcomes (rather than the opponent ratings) in those two removed games. It is basically saying that the secondary goal for these players, in addition to scoring as many overall points as possible, is to do particularly well against the middle of the pack. I really think it would be fairest to say that the 11 outcomes all have equal relevance.

(2c) Performance Rating Removing Top/Bottom ratings

A methodology that perhaps would have been more in the spirit of the "middle of the pack" approach, would be to say that we retain all eleven game outcomes, but discard the highest-rated opponent and lowest-rated opponent, for purposes of calculating the average opponent rating. One might even argue that this is a perfectly valid way to read the phrase "the highest and the lowest rated opponent will be deleted". It certainly seems to me to be a fairer and smoother way. So under this approach, these four players would still retain their 77% score, but we would only use the middle nine opponent ratings when calculating the average opponent rating:

And so we can see that under this approach, it would have Moiseenko rather than Polgar who would have finished in third place. I will also show you at the end how the World Cup qualifiers would have differed, depending on which performance rating calculation was used.

(2d) Thompson Performance Rating

Ken Thompson has previously suggested an eminently reasonable calculation for performance rating, though it is much harder to calculate than the above approaches. The idea is actually quite simple: if somebody scores 8.5/11 against a particular set of ratings, then the "performance rating" will be whatever FIDE rating would have yielded an expected score of exactly 8.5/11 against those exact opponents. More generally, performance rating ought to represent the rating that would have led to an expected score exactly the same as the score that was actually achieved, given the individual opponents' ratings.

Another great thing about this calculation is that we don't have to care at all what the player's current rating is, yet we can still apply the "400-point rule" that limits the expected score in the case of very large rating differences. It does seem a bit strange to penalize/reward a player in their performance rating calculation, based on what their current rating actually is, but any approach that wants to implement that "maximum rating difference of two players shall be 400 points" constraint would have to either use the Thompson-like approach, or use the player's own rating, which seems a bit counter-intuitive. The whole point of performance rating is that it provides a way to describe the player's performance skill based only on results, without having to use the player's current rating.

This calculation becomes trickier than we would like, because the Elo Expectancy Table is not continuous; it only provides the expected score out to two decimal points, and sometimes this leads to zero solutions or multiple solutions. So you would ideally use Elo's exact formula instead of the Expectancy Table, although I like to split the difference and use linear interpolation on the official Expectancy Table. In addition the only way to really perform the Thompson calculation is to do a "binary search", where you keep trying different proposed ratings, then see whether it leads to an expected score that is too low or too high, and then adjust the proposed rating up or down accordingly until you get closest to the desired expected score. Finding the best fit out to 0.1 Elo points gives us these performance ratings for the four top finishers:

Part 3 – Alternative final standings

In the detailed examples provided above, we can see that for three reasonable alternative ways to calculate performance rating, the third place finisher would have been Moiseenko rather than Polgar. And as you might expect, there would be similar variations in the qualification for the World Cup. There were 23 World Cup slots available in this tournament, with three of the top finishers (V.Potkin, P.Svidler, and R.Mamedov) already having qualified, as had two other players who also finished on +4 (B.Jobava and A.Pashikian). Since the top four scored +6, and the next eleven scored +5, and three of those fifteen players were already in, that meant twelve of the 23 World Cup spots were claimed by players on +6 or +5, leaving 11 slots for the players having the best tiebreak scores among the whole group on +4. So the lucky ones were those finishing in #16 through #26, and the unlucky ones were those finishing in #27 through #44:

 Clearly the most unlucky player was Mircea-Emilian Parligras, who had a top-ten performance rating by most measures, the second-best Median Buchholz and Buchholz scores in the entire tournament (those were the tiebreakers after performance rating), and in fact had the best performance rating out of the entire +4 group in all three of the alternate calculations. Yet he finished in 14th out of the 29 players on +4, for an overall placement of #29 in the tournament, mostly because the two results that were deleted were both wins, and so he failed to qualify for the World Cup. Ivan Ivanisevic and Peter Heine Nielsen also would have qualified for the World Cup under any of these other three measures, yet failed to qualify under the calculations actually used (finishing #27 and #32, respectively). And of course you can see from the above listings that several players would have failed to qualify for a World Cup spot under any of the three alternative calculations, yet did qualify, typically because they had 1.0/2 removed from their score during the tiebreak calculation. 8 of the 12 who had 1.0/2 removed from their score did qualify, whereas none of the five who had 2.0/2 removed from their score (including Parligras) managed to qualify.

I don't really have a strong recommendation for what should be done here, because I don't know what was communicated to the players beforehand, or even during the tournament. It wouldn't seem right to use either the traditional or Thompson performance measures, given the wording of the rules about discarding the highest and lowest rated opponents, but on the other hand I really dislike the idea of discarding the actual outcomes and letting the remaining percentage score play such a large relative role. So if you had asked me in the middle of the tournament, I would have suggested interpreting the rules to use the "Removing Top/Bottom Ratings" calculation. If you had asked me after the tournament… is it too late?

So certainly one major lesson here is that tournament regulations need to be unambiguous with regard to how performance rating is calculated. This applies wherever performance rating is used: tiebreak rankings, board prizes, etc. However, it still doesn't answer a very interesting question: what method for calculating performance rating works best? I am currently working on analyzing several of these formulas for predictive accuracy, and the preliminary evidence reveals that the most accurate performance rating measure (out of the four listed above) would either be the standard performance rating formula or the "Removing Top/Bottom Ratings" formula.

Interestingly, for 9-round events in particular, the "Removing Top/Bottom Ratings" formula appears to be slightly more accurate than the standard calculation. GM David Navara has also made a very interesting suggestion wherein you calculate the standard performance rating, but exclude any wins that have decreased your overall performance rating, or any losses that have increased your overall performance. This is a neat idea that needs to be carefully assessed, as it does lead to the possibility of comparing performance ratings between two players across differing numbers of games, which is not ideal. In a few days I will release the results of my analysis of predictive accuracy.

Copyright Jeff Sonas / ChessBase

Previous articles by Jeff Sonas on ChessBase.com