Test your engines – the Silver Openings Suite

The Silver Openings Suite

By Albert Silver

Before GM John Nunn published his first “Nunn Positions”, trying to find out a chess engine’s strength as opposed to another was an unsure task at best. Programs were usually pitted against each other in long matches, results were compiled, and their ratings calculated. A recurring issue was that new programs could often kill older ones straight from the opening thanks to more up-to-date opening books, sometimes even prepared specifically against them. This made it hard to answer the question of which was the best engine to analyze one’s games.

Nunn’s idea was quickly accepted as simple and brilliant, helping provide a controlled testing environment for engines. Instead of their opening books, engines would be tested against each other using a set of pre-selected opening positions, played as both white and then black, so that only the engine’s playing strength was tested, and not the quality of its openings book. Nowadays, all the major playing interfaces support this, including Fritz, for which instructions are provided below.

Over time, the concept has validated itself repeatedly, and the original suites have become a deserved standard, though a few shortcomings have also appeared: the chosen positions, while excellent, don’t actually cover the full gamut of openings or types of positions encountered in the opening phase, and thus the middlegame. Also, some positions have been known to initially play out almost always the same way in the first moves, as the top engines all agree on a specific continuation. The following openings suite was designed to address some of these issues, and provide a complementary testbed for engine testing.

This openings suite is comprised of 50 positions, leading to a round 100-game match, covering not only the major openings, but also standard pawn structures and opening systems. Therefore not only will you find an Alekhine, some French Defenses, and several flavors of the Sicilian, but there are also a Hedgehog, a Stonewall, a Colle, and a King’s Indian Attack for example.

To determine their frequency, the number of positions for each opening found in the suite is proportionate to the number of times played as per statistics culled from Mega Database 2010. In other words, the number of Sicilians found in the suite (9), is roughly proportionate to the number found in the 4.5 million game database. This isn’t true in an absolute sense, as minor concessions were made to favor variety as well.

Finally, to ensure diversity, the positions lead to junctures in theory, where each engine is free to choose the direction it prefers. Each and every position was tested with several top engines (notably Rybka 3, Fritz 12, Shredder 12, and Stockfish 1.6.3) for a few minutes, and their choices were compared. If they all chose the same continuation, the position was rejected. This preliminary testing, followed by several test matches, overseeing the games, helped refine the positions chosen.

That said, if your goal isn’t only to test engine strength, but also to watch the engines duke it, the most entertaining moments are often when all hell breaks loose. So, a second suite of 25 positions was put together, based entirely on gambits. In fact, three of the positions even start with a piece sacrifice, and while they may not hold up in long time-controls, if you watch them at five-minute games, chaos and cutthroat chess are virtually guaranteed.

Nevertheless, two criteria were applied in the selection:

• The engines could not start out believing they were dead lost, which would have been dreadfully unfair. Thus a maximum -0.70 self-evaluation limit was set.

• The theory could not be reduced to finding ‘only-moves’, meaning only one move is playable, since this would be tantamount to a tactical test suite rather than an openings playing suite.

For those who use such tests, or enjoy watching interesting games, I hope both suites are found to be useful and fun.

Engine matches in Fritz 12

Setting up a match between engines, using one of the opening suites is as easy as could be in Fritz 12. At the top, first select Engine, then Engine Match.

A small window will open allowing you to specify your preferences. Let’s start by choosing a time control for the match. The default is 4 minutes per side with a 2 second increment per move. If one clicks on the radio button (that little round button circled in red), a new window opens to permit you to change the time control.

Now let’s choose the engines. Click on the Define button under White, and in the window that opens, choose the engine. Remember that with an openings suite, the engines will play both colors, so it doesn’t matter which one is white or black. Now do the same for the black side, so that both engines are chosen.

All that is left is to choose the Openings Suite. Click on Openings DB and select the file in question.

I saved mine in \My Documents\ChessBase\Testsets, but if you saved it somewhere else, go to the directory select it, and click Open.

That’s all there is to it. Now that you have chosen your options, just click Ok and have fun!

• Click to download all Silversuite positions in PGN
• Click to download all Silversuite Gambit positions