The Silver Openings Suite
By Albert Silver
Before GM John Nunn published his first “Nunn Positions”, trying
to find out a chess engine’s strength as opposed to another was an unsure
task at best. Programs were usually pitted against each other in long matches,
results were compiled, and their ratings calculated. A recurring issue was that
new programs could often kill older ones straight from the opening thanks to
more up-to-date opening books, sometimes even prepared specifically against
them. This made it hard to answer the question of which was the best engine
to analyze one’s games.
Nunn’s idea was quickly accepted as simple and brilliant, helping provide
a controlled testing environment for engines. Instead of their opening books,
engines would be tested against each other using a set of pre-selected opening
positions, played as both white and then black, so that only the engine’s
playing strength was tested, and not the quality of its openings book. Nowadays,
all the major playing interfaces support this, including Fritz, for which instructions
are provided below.
Over time, the concept has validated itself repeatedly, and the original suites
have become a deserved standard, though a few shortcomings have also appeared:
the chosen positions, while excellent, don’t actually cover the full gamut
of openings or types of positions encountered in the opening phase, and thus
the middlegame. Also, some positions have been known to initially play out almost
always the same way in the first moves, as the top engines all agree on a specific
continuation. The following openings suite was designed to address some of these
issues, and provide a complementary testbed for engine testing.
This openings suite is comprised of 50 positions, leading to a round 100-game
match, covering not only the major openings, but also standard pawn structures
and opening systems. Therefore not only will you find an Alekhine, some French
Defenses, and several flavors of the Sicilian, but there are also a Hedgehog,
a Stonewall, a Colle, and a King’s Indian Attack for example.
To determine their frequency, the number of positions for each opening found
in the suite is proportionate to the number of times played as per statistics
culled from Mega Database 2010. In other words, the number of Sicilians found
in the suite (9), is roughly proportionate to the number found in the 4.5 million
game database. This isn’t true in an absolute sense, as minor concessions
were made to favor variety as well.
Finally, to ensure diversity, the positions lead to junctures in theory, where
each engine is free to choose the direction it prefers. Each and every position
was tested with several top engines (notably Rybka 3, Fritz 12, Shredder 12,
and Stockfish 1.6.3) for a few minutes, and their choices were compared. If
they all chose the same continuation, the position was rejected. This preliminary
testing, followed by several test matches, overseeing the games, helped refine
the positions chosen.
That said, if your goal isn’t only to test engine strength, but also
to watch the engines duke it, the most entertaining moments are often when all
hell breaks loose. So, a second suite of 25 positions was put together, based
entirely on gambits. In fact, three of the positions even start with a piece
sacrifice, and while they may not hold up in long time-controls, if you
watch them at five-minute games, chaos and cutthroat chess are virtually guaranteed.
Nevertheless, two criteria were applied in the selection:
-
The engines could not start out believing they were dead lost, which would
have been dreadfully unfair. Thus a maximum -0.70 self-evaluation limit
was set.
-
The theory could not be reduced to finding ‘only-moves’, meaning
only one move is playable, since this would be tantamount to a tactical
test suite rather than an openings playing suite.
For those who use such tests, or enjoy watching interesting games, I hope both
suites are found to be useful and fun.
Engine matches in Fritz 12
Setting up a match between engines, using one of the opening suites is as easy
as could be in Fritz 12. At the top, first select Engine, then Engine Match.
A small window will open allowing you to specify your preferences. Let’s
start by choosing a time control for the match. The default is 4 minutes per
side with a 2 second increment per move. If one clicks on the radio button (that
little round button circled in red), a new window opens to permit you to change
the time control.
Now let’s choose the engines. Click on the Define button under
White, and in the window that opens, choose the engine. Remember that with an
openings suite, the engines will play both colors, so it doesn’t matter
which one is white or black. Now do the same for the black side, so that both
engines are chosen.
All that is left is to choose the Openings Suite. Click on Openings DB
and select the file in question.
I saved mine in \My Documents\ChessBase\Testsets, but if you saved it somewhere
else, go to the directory select it, and click Open.
That’s all there is to it. Now that you have chosen your options, just
click Ok and have fun!