Rybka's Monte Carlo Analysis revisited

Several columns ago in ChessBase Workshop, we took a close look at the new "Monte Carlo" analysis form in the Rybka playing program. Making a long story short, the Monte Carlo analysis feature is started at a particular board position, and causes the Rybka engine to play a large number of very fast games against itself from that position; the idea is for the engine to essentially say "given best play for both sides, this is what is expected to happen". Instead of storing all of these games in a database (as occurs in the "Shootout" mode), the games are stored as a tree when using the Monte Carlo analysis mode.

In the course of that column, I presented the following illustration:

...which was accompanied by the following explanation:

This display reminds us of the engine which is being used for the analysis and the search depth we set at the start of the process. We see how many games have been played so far (54 at the time I made the screen capture) and the results of those games. In this case, White won twenty-five of them while Black won just three; twenty-six of the games were drawn. The next value is the success rate, given from White's perspective (70.4%). "TP" is a "performance rating" value, again presented from White's viewpoint. And I'll confess that the remaining values have me stumped so far.

Small wonder that I was stumped as I've never taken a formal statistics class; consequently I'd never encountered these values before. Two of ChessBase Workshop's readers rode to the rescue and have generously provided an explanation for the remaining values.

The first e-mail comes from Johannes Köhler, who writes:

My understanding of Monte Carlo Analysis (ANY Monte Carlo Analysis), is that it is a simulation (in this case of many games played by the engine against itself) and the output is always a probability function (normal distribution, usually) of the referred simulation. In Rybka's case, the output is a tree and the probability refers not only to the outcome but also for the Performance Rating you refer to. Therefore, those numbers you mention that have you "stumped so far" appear to be intervals of performance rating and the percentages are the probability or the percentage of cases that the performance falls in those intervals.

In your example, the average performance rating is +150, and in 68% of the cases the performance rating in the games played in the simulation have fallen in the interval +117 to +186, 95% of the cases in the interval +87 to +225 and 99.7% of the cases in the interval +57 to +270. The percentages follow the known Chebichev theorem that in a normal distribution 68% of the observations will fall within +/- 1 standard deviation, 95% of the observations will fall within +/- 2 standard deviations and 99.7% (or virtually all observations), will fall within +/- 3 standard deviations.

I hope this helps, as I found this the best explanation for those numbers.

This helps a great deal, sir, and I thank you for sending this explanation to me.

A similar, but somewhat less involved explanation, comes our way from Jim Weiner, who sent me this:

Concerning the Chessbase Workshop article recently posted on Rybka Monte Carlo analysis: the values in the bar that have you stumped are what statisticians call confidence intervals for the tournament performance rating change (TP). In the first screen shot, the displayed TP is +150 Elo, but that's actually just the best (highest-probability) estimate. "68%->[117,186]" means there's a 68% chance that the true rating change is somewhere between +117 and +186 Elo, etc. Note that the higher the confidence (the percentage value), the wider the interval. In other words, if we want to have a greater confidence of bracketing the true value, we have to widen the bracket. That's why the 99.7% interval has the wider endpoints of +57 and +270. In statistics, as in life, there's no free lunch (unless you're a banker, I guess).

My thanks to both of you gents for the wonderful explanations! I'd never have figured out this information on my own. Many thanks!

And now for another of those little controversies which seem to pop up about once a year or so. In a column about the various ChessBase playing engines, I was guilty of committing a shortcut in which I said that "if you need to ask if you have a multi-processor computer, you probably don't have one" based on the cost of such a machine.

Numerous readers e-mailed to take me to task for that. And you're right to do so. While I still maintain that a true multi-processor machine is more expensive than a single-processor one, I wasn't taking into account the numerous single-processor computers which allow multi-threading. Further complicating matters, however, were the e-mails which mentioned that multi-threading may or may not behave in the same way as multiple processors and that this has been a hot topic on several computer chess message boards (and this is, in fact, the main problem with my initial comment -- I was speaking strictly of computers which physically contain more than one processor).

So I will amend my previous statement to this: when in doubt, contact the vendor or manufacturer of the hardware in question (whether it's a computer or processor you already own or one which you're considering purchasing) to obtain the specifications, since it's not possible for me (or anyone else) to provide a comprehensive list of what makes/models/processors will allow a "Deep" version of a playing program to take advantage of any increased processing ability. The aforementioned computer chess message boards are also good places to go to make inquiries, as you'll often find someone who has prior experience with the specific hardware about which you're curious.

Until next week, have fun!

© 2009, Steven A. Lopez. All rights reserved.