Winning starts with what you know
The new version 18 offers completely new possibilities for chess training and analysis: playing style analysis, search for strategic themes, access to 6 billion Lichess games, player preparation by matching Lichess games, download Chess.com games with built-in API, built-in cloud engine and much more.
Last week we looked at the first of the new ChessBase DVDs and I posed the question, "Why a DVD?" The answer was self-evident for that disk. But when I saw that the new Fritz Powerbook 2004 was on DVD, I was brought up a bit short. "Is the opening book that big? I wondered.
Before we seek an answer, let's talk about the philosophy behind the Powerbook. If you're a Fritz owner (or you own one of Fritz' "sister" playing programs) you already know that chessplaying programs come with an opening book: a library of opening moves the program refers to when it starts a chess game. This is to keep a computer from "reinventing the wheel" with each new game; if left to their own devices and forced to rely on the engine's chessplaying algorithm, most chess programs will tend to play the same thing game after game. The opening books that come with chessplaying programs are not made up of every move in every known opening -- in fact, a program's opening book is usually tailored to that program's particular style of play to try to get into middlegames that the program will be exceptionally good at playing. "Closed" openings are usually left out. If a chess program has a particular weakness (perhaps it's not great at fianchetto openings), the book will be "tweaked" to avoid those kinds of openings. There are many other general examples, but you get the idea: each program's opening book is "tuned" to maximize that program's unique strengths while minimizing its weaknesses. The opening book is tweaked for use by a computer; a human's style of play isn't the primary factor.
However, the primary user of a chessplaying program is a typically a human being -- and humans like variety. After you've played a few dozen games against your chess program, you might notice a certain sameness to the kinds of positions you see: open games in which tactical sharpness is the rule. But if you're using a chess program to try to hone your own skills and your big nemesis is the elderly guy down at the chess club who always plays closed d-pawn openings, you might find your playing program's choice of openings to be, um, less than helpful.
That's where Powerbook comes in. It's a "human" opening book -- derived from games played by humans and designed to be used as a training tool for humans. The idea is simple: take the best games played by top human players spanning more than 100 years of chess history and merge their openings into a single book -- then load that book into a chessplaying program to give the engine access to all of those openings, regardless of whether or not those openings fit a computer program's style of play. The human user of the chess program will then see that program play a broader range of openings; he'll get exposed to a greater variety of opening types, and (hopefully) become accustomed to them and become a stronger player himself.
It's a pretty simple concept. But within the past couple of years a new chess subculture has sprung up: players who like to pit their hardware/software combinations against other like-minded users' machines in games played on the Internet. Many of these players are always looking for an "edge" and over the last few years some of them have thought (erroneously) that using the Powerbook instead of their program's default opening book was going to make their hardware/software combo a killer. I can't begin to count the number of phone/e-mail exchanges I've had with these folks in which they've expressed bewilderment at their programs' poor performance against other computers when they use the Powerbook in computer vs. computer games. The fact was that using the Powerbook would very often lead to opening choices which aren't congenial to the way a computer plays chess. These users didn't understand the point of the Powerbook -- it's a tool geared toward human players who want to improve their own play by exposing themselves to a wider variety of openings, not a "killer book" to be used in computer vs. computer play.
The Powerbooks of previous years were typically derived from databases of 600,000 or so games and limited in the length of the variations, all of which ensured that the whole thing would fit comfortably on a CD. With the increased storage available on a DVD, the ChessBase developers were able to go all out with Powerbook 2004: the book is compiled from more than a million games played by strong chessplayers in the years 1900 through 2003.
Let's start by looking at that database. It contains 1,002,084 games played by players of International Master status and up. Right away a red flag should go up with most readers. "Since the Elo system wasn't adopted until the early 1970's, how do we know that the older games are from strong players?" ChessBase has devised a unique solution: they've used a system (devised by Ken Thompson and included as a feature in Fritz) to retroactively rate players from the decades before the 1970's. And a quick look at the database included with Powerbook 2004 will show this: estimated Elo ratings (assigned by Thompson's algorithm) are included in the game headers for the pre-1970 games. You just need to remember that these ratings are estimates and that pre-1970 players didn't hold numerical ratings.
As for the Powerbook opening tree itself, this sucker is huge -- over a gigabyte in size (that's why it ships on a DVD instead of a CD). It contains 18,071,012 unique opening positions along with statistical information for every position in the tree: the number of games in which it appeared, the success rate for each move from that position, the Elo average for the players who played each move.
And it's keyed to the million-plus game database from which it was derived. Here's how you use it. Fire up Fritz (or any of its sister player programs). Load both the database and the opening book. Open up the book and step through the tree. When you come to an interesting position, stop and right-click in the opening book pane; you'll get a popup menu of various commands. Select the first command: "Search games" and the program will find all of the games in the database in which the position appeared, displaying a list of them. You can double-click on any of these games to load it and start at the position you were just examining in the tree -- now you can see how the middlegame and endgame played out. Simple!
You can copy both the Powerbook opening tree and database to your hard drive for easy access, or just run them straight from the DVD. The developers also realize that not everyone has an extra gigabyte of free hard disk space, so they've covered such users with a second book -- a "compact" version: the "Strong Database" and its corresponding opening book. It's a smaller opening book and database which is actually a subset of the full Powerbook: 50,235 selected games from top-echelon players, along with an opening book derived from these games (and containing 902,294 unique opening positions). Together they take up just under 76 MB of hard drive space.
The Powerbook 2004 DVD's fun doesn't stop there. Also included on the disk is an updated set of detailed "Globe" files for use in the Playchess section of the Fritz program. You copy these files into your ..\ChessProgram8 folder and you'll see updated detailed geographic maps in the Globe view when you're playing chess on the Playchess server.
So if you're a chessplayer looking to broaden your horizons with a wider range of openings for your chess program, you'll certainly do just that with Powerbook 2004 -- with the bonus of broadening your geographic horizons, too.
Just as computer chessplaying programs use an opening book to be able to play a wider range of openings (or a set of openings specifically tailored to that program), they can also use endgame databases (called "tablebases") to be able to play the endgame perfectly.
The idea's not a new one. Ken Thompson (back in his Bell Labs days more than a decade ago) devised just such a series of endgame databases. Each database contained every possible position with a certain set of pieces on the chessboard (for example, King and Queen vs, King and Rook, sans pawns). When a game reached a position that was covered in one of these databases, the program could "look up" the position and see the consequences of every legal move, basing its move decision on the perfect endgame knowledge contained in these databases.
But, good as these databases were, there were some hitches with them. A program couldn't access them until a position "known" in one of the databases was actually on the board -- it couldn't "look ahead" into the databases during late middlegame positions. And, in the case of the few "single pawn" databases, a program couldn't see what happened after a pawn promoted to another piece -- it couldn't switch to another database reflecting the post-promotion material to see what would happen after the promotion.
All of this changed in the late 1990's with the development of a new set of endgame tablebases by Eugene Nalimov. Chess programs could "look ahead" to these tablebases even if that reduced set of material wasn't already on the chessboard (it could look ahead to the point when material had been traded down to a simplified endgame covered in the tablebases and use that information as part of its decision-making process). And if some future pawn promotion or further reduction of material led to information contained in another tablebase, the program could switch tablebases to access the new material.
Perhaps best of all, Nalimov gave users the ability to generate their own tablesbases through a freeware program called TBGen, so players no longer had to wait for a new endgame tablebase to be released in some secret proprietary format. The downside was that it required a very powerful computer to generate them in a reasonable amount of time -- and in the case of the larger five-man tablebases, this could require a computer to run around the clock for days to be able to crank out a single new tablebase file (even with a super-fast computer).
Consequently, chess software vendors began to sell pre-generated Nalimov tablebases on CD. It still required a pretty high-end computer to be able to use them, since a full set of tablebases required about 8 GB of disk space (which doesn't sound like much now, but this was a lot of storage back in the late 90's). ChessBase released a tablebase set called the Fritz Turbo Endgame, which contained a partial set of tablebases (all three-man, four-man, and the most commonly encountered five-man endings) on four CDs requiring about 2.5 GB storage space.
In recent years, Eugene Nalimov has been working on six-man tablebases (and, as a brief point of clarification, the Kings do count against the total number of pieces in a tablebase -- so, for example, in a six-piece tablebase the material consists of the Kings and four other pieces or pawns). The six-man files are enormous; a single six-piece tablebase file can't be contained on a CD. This is why ChessBase has chosen to release the new Turbo Endgame 2 on DVD -- so that the six-piece tablebases can be included.
Turbo Endgame 2 ships on five DVDs. Here are the contents of each DVD:
Of course, the question that leaps immediately to everyone's mind is, "How much hard disk space will be needed for all of that?" It's a whopping 23+ GB of disk space!
That's a lot of room and not all of us will have that kind of storage space available (in my case, I'd have to uninstall all of my robotic combat games -- and we know that ain't gonna happen!), so the ChessBase developers have provided us with two sets of installation instructions which cover other options for installing partial sets of tablebases. You'll find the information contained in a booket that comes with the Turbo Endgame 2 DVD set. The booklet also explains the ideas behind endgame tablebases and instructs you on how to use them (including using them for analysis within ChessBase 8).
If you're a Fritz8 user, there's an additional bonus for you in this DVD set -- an upgraded Fritz8 engine which replaces any previous version you have.
Please keep in mind one caveat, though: not all chessplaying programs can access the six-man tablebases. At present, Fritz8 and Hiarcs9 are the only ChessBase playing programs that can use the six-piece files. The others (such as Shredder, Junior, and Chess Tiger) can access only the three-, four-, and five-piece endings.
Unlike the Powerbook opening tree (which can actually hurt your program's performance in games against other chessplaying programs), Turbo Endgame 2 will definitely help your program's performance immeasurably. Many endgames require specific knowledge that chess programs can't figure out for themselves within the limited constraints of a timed game. Turbo Endgame 2 will give Fritz and Hiarcs instant and perfect endgame knowledge (which, as previously discussed, will help the engines' analysis even before these positions actually occur on the chessboard).
So if you're an online computer vs. computer chessplayer and you're looking for a way to "rev up" your program's performance, Turbo Endgame 2 is absolutely a great way to go. They don't call it "Turbo" for nothin'!
Until next week, have fun!
