Hartmann – choosing a chess engine

9/28/2014 – Three chess programs today dominate the scene, with new versions supplanting each other in the computer rating lists. Currently Komodo is leading, and conveniently is released as a brand-new ChessBase engine. But is it really the one that ambitious chess players should use (or switch to)? Chess book reviewer John Hartmann offers some very useful advice.

Choosing a Chess Engine

By John Hartmann

Increasingly I’m convinced that a serious chess player must make use of chess technology to fully harness his or her abilities. This, as I have previously discussed, involves three elements: the GUI, the data, and the engine. ChessBase 12 is the gold standard for chess GUIs, and I will be reviewing a new book about proper use of that GUI in the near future. Here, however, I want to take up the thorny issue of choosing a chess engine. Which engine is ‘best’ for the practical player to use in his or her studies?

I put ‘best’ in scare-quotes because there are two ways to look at this question.

  1. There is little question at this point that the best chess engines of the past five years can beat 99.9% of human players on modern hardware. So one way that engines are tested now is in a series of engine vs engine battles. While many people process private matches, there are three main public rating lists: IPON, CCRL and CEGT.

  2. From my perspective, however, analytical strength is more important. If all the engines are strong enough to beat me, I think that the quality of their analysis – the ‘humanness’, for lack of a better word – is critical. It used to be the case that humans could trick engines with locked pawn chains, for example, or that engines would fail to understand long-term compensation for exchange sacrifices. Such failings have largely been overcome as the engines and hardware have improved; nevertheless, there remain certain openings and types of positions that are more problematic for our metal friends. Michael Ayton offers one such position in the ChessPub forums; if you want have a laugh, check out the best lines of play on offer by the engines reviewed here:

Among the multiple engines available, there are three that stand above the fray. These are Houdini by Robert Houdart, Komodo by the late Don Dailey, Larry Kaufman and Mark Lefler, and Stockfish. Houdini and Komodo are commercial engines, while Stockfish is open-source and maintained by dozens of contributors.

How can we understand the differences between the engines? Let’s consider two key components of chess analysis: search and evaluation. Search is the way that the engine ‘prunes’ the tree of analysis; because each ply (move by White or Black) grows the list of possible moves exponentially, modern engines trim that list dramatically to obtain greater search depth. Evaluation is the set of criteria used by the engine to decipher or evaluate each position encountered during the search.

In a very general sense, what differentiates Houdini, Komodo and Stockfish are their search and evaluation functions. How they are different on a technical/programming level, I cannot say: Houdini and Komodo are closed-source and I can’t decipher code in any event. What I can do, however, is cite what some experts in the field have said, and then see if it coheres with my experience of the three engines.

Larry Kaufman, who works on Komodo, said in an interview on the Quality Chess blog that:

Komodo is best at evaluating middlegame positions accurately once the tactics are resolved. Stockfish seems to be best in the endgame and in seeing very deep tactics. Houdini is the best at blitz and at seeing tactics quickly. Rybka is just obsolete; I like to think of Komodo as its spiritual desceendant, since I worked on the evaluation for both, although the rest of the engines are not similar. Komodo’s assessment of positions is its strong point relative to the other top two, which overvalue the queen. Komodo has the best sense for relative piece values I think. Komodo is also best at playing the opening when out of book very early.

Stockfish is, as Kaufman suggests, very aggressive in the way that it prunes the tree of analysis, searching very deeply but narrowing as the ply go forward. It is important to remember that each engine reports search depth and evaluation differently, so that the way that Stockfish ‘razors’ the search means that its reported depth can’t be directly compared to Houdini or Komodo. Still, it does seem to search more deeply, if narrowly, than do its competitors. This has advantages in the endgame and in some tactical positions.

Houdini is a tactical juggernaut. It tends to do best on the various tactical test sets that some engine experts have put together, and it is fairly quick to see those tactics, making it useful for a quick analysis of most positions. Its numerical evaluations also differ from other engines in that they are calibrated to specific predicted outcomes:

A +1.00 pawn advantage gives a 80% chance of winning the game against an equal opponent at blitz time control. At +2.00 the engine will win 95% of the time, and at +3.00 about 99% of the time. If the advantage is +0.50, expect to win nearly 50% of the time. (from the Houdini website)

Kaufman argues that his engine, Komodo, is the most positionally accurate of the three, and I don’t disagree. Kaufman is involved in the tuning of Komodo’s evaluation function; as he is a grandmaster, it does not seem outrageous to believe that his engine’s positional play might benefit from his chess expertise. The engine is slightly ‘slower’ (anecdotally, and not judging by NPS, or nodes per second, and ply count) than are Stockfish and Houdini, but Komodo seems to benefit more from longer analysis time than do Houdini or Stockfish.

I’ve been using Komodo 8 in the Fritz GUI from ChessBase for a few days now. The GUI is the same as the Houdini 4 and the Deep Fritz 14 GUIs; in fact, when you install Komodo 8, I think it just adds some configuration files to your ChessProgram14 folder to allow for a Komodo ‘skin’ to appear. The Komodo 8 engine is slightly faster than 7a judging solely by NPS. While coding changes mean that the two can’t be directly compared, Mark Lefler has said that 8 is approximately 9% faster than 7a. The ChessBase package comes with a 1.5 million game database, an opening book, and a six month Premium membership at Playchess.com; all are standard for Fritz GUI releases such as Deep Fritz 14 or Houdini 4.

From my perspective, I tend to use all three engines as I study chess or check analysis for review purposes, but two more than the third. When I look at my games, which aren’t all that complex, I generally use Houdini as my default kibitzer. It seems to be the fastest at seeing basic tactical problems, and its quickness is a plus on some of my antiquated computers. I also tend to bring Komodo into the mix, especially if I want to spend some time trying to figure out one position. Stockfish serves more as a second (or third) option, but I will use it more heavily in endgame positions – unless we get into tablebase territory, as Stockfish does not (generally) use them.

As I was working on this review, I thought that I might try to ‘objectively’ test the engines on positions that were more positional or prophylactic in nature, or perhaps in some difficult endgame positions. I took 11 positions from books on hand, including a number from Aagaard’s GM Preparation series, and created a small test suite. Each engine (including Deep Fritz 14 for comparison’s sake) had 4 minutes to solve each problem on my old quad-core Q8300, and each engine had 512mb of RAM and access to Syzygy (5-man) or Nalimov (selected 6-man) tablebases as they preferred. You can see the results on the following JavaScript board:

(select games from the dropdown menu above the board) or as summarized below:

Deep Fritz 14, curiously enough, solved more problems than did Houdini 4, Komodo 7a/8 or Stockfish 5. None could solve the famous Shirov …Bh3 ending. None could solve the Polugaevsky endgame, which illustrates a horizon-related weakness still endemic among even the best engines. Only Komodo 7a, Komodo 8 and Deep Fritz 14 solved position #2, which I thought was the most purely positional test among the bunch. This test is only anecdotal, and perhaps the engines would have gotten more answers right on faster hardware; nevertheless, I was a little surprised.

Test #2: Jon Dart (author of Arasan) has created a series of test suites to torture his engine and others. I took the first 50 problems from the Arasan Testsuite 17 and ran Houdini 4, the two Komodos, Stockfish 5, Deep Rybka 4.1 and Deep Fritz 14 through their paces. (I would have added Crafty 23.08, installed with Komodo 8, but it kept crashing the GUI when I tried to include it in the test.) Here the engines only received 60 seconds to solve the problem – the same standard Dart uses in his tests of Arasan, albeit with a much faster computer. You can see the results on the following JavaScript board:

(select games from the dropdown menu above the board) or as summarized below:

Stockfish 5 and Houdini 4 each solved 38/50 problems in the one minute time limit. Komodo 8 solved 30 problems, improving by one over Komodo 7a’s 29 solved problems, and doing so with a faster average solving time. Deep Rybka and Deep Fritz each solved 28 problems correctly. Given the shorter ‘time control’ and the relatively tactical nature of the test set, these results seem representative of the various engines and their characteristics.

So now we have to answer the real question: which engine is best? Which one should you use? Let’s begin by admitting the obvious: for most analytical tasks you throw at an engine, any one of the three would suffice. Most of the other major ‘second-tier’ engines, including Crafty (free to download), Deep Fritz (commercial), Hiarcs (commercial) and Junior (commercial), are also sufficient to analyse the games of amateurs and point out our tactical oversights. If you’re just looking for an engine to blunder-check your games, you have plenty of options.

If, however, you’re using engines for heavy analytical work or on very difficult positions, I think you need to consider buying both Houdini and Komodo and also downloading the open-source Stockfish. Each engine, as discussed above, has relative strengths and weaknesses. The best strategy is to see what each of the engines have to say in their analysis, and then try to draw your own conclusions. Were I forced to decide between Houdini 4 and Komodo 8, I’d probably – at this moment, anyway! – choose Komodo 8, simply because it seems stronger positionally, and its slight comparative tactical disadvantage doesn’t outweigh that positional strength. Both Houdini and Komodo are well worth their purchase price for the serious player and student. Downloading Stockfish should be mandatory.

Order Houdini 4 Standard or Houdini 4 Pro.
Order Komodo 8 now.

About the author

John Hartmann is an award-winning chess book reviewer, a chess teacher, organizer, and tournament director in Omaha, Nebraska. Currently he writes for Chess Life, the British Chess Magazine, and his own website, Chess Book Reviews, where the above article originally appeared.

John is a coffee aficionado, a weekend baseball player, a Cardiff City supporter, and a soon-to-be father. He also tries, from time to time, to fit in some work on his dissertation in philosophy.

Feedback and mail to our news service Please use this account if you want to contribute to or comment on our news page service



Discuss

Rules for reader comments

 
 

Not registered yet? Register

NYTed NYTed 9/29/2014 12:53
Nice read, I pretty much agree with this assessment.
Werewolf Werewolf 9/29/2014 08:58
The next stage is in GUI development to properly use these engines. So far we have only seen limited ability by the Fritz GUI to manage engines for automated analysis. "Deep Position Analysis" is the best Chessbase have come up with so far, but they could go much further.

Edwin Bakker Edwin Bakker 9/29/2014 09:38
As you mentioned, Komodo 8 needs more time to really shine, but this can also be translated in having a faster machine.
If you have a consumer 8-core Core i7 5960x, which is 4~5 times faster than the old Q8300, a certain test position will take you <10 seconds to solve instead of 30 seconds.
Also compare the CCRL 40/4 and 40/40 ELO ratings and you'll see that Komodo 8 overtakes Stockfish in 40 minutes for 40 moves. It really gains strength.

In my opinion, Komodo 8 on a fast machine is the way to go (for now)!
PurpleUnicorns PurpleUnicorns 9/29/2014 09:55
For the most part I agree on the authors assessment. I have a friend that I recommended Komodo and he is a big fan now. I personally use Stockfish for the most part.

As for GUIs I used to use chessbase products, however due to their lack of linux support they have lost me as a customer over the years. Nowadays I mostly use Scid or xboard which work great for my needs.
dansafee dansafee 9/29/2014 12:38
I think that in general, the reviews here need to be taken with a grain of salt. At the end of the supposedly unbiased article, there is a link to buy the engine from the chessbase website. That doesn't mean the engines are bad or that the review is a shill, but it doesn't feel right.
schack schack 9/29/2014 01:44
I'm breaking the first rule of web authorship here by reading comments on my piece, but here we go...

ChessBase asked if they could reprint (in an truncated form) a piece that originally appeared on my blog. They added the links to their webshop. I suspect that were the piece truly a shill, they'd have cut all the references to Stockfish.
genem genem 9/30/2014 07:26
To @Werewolf (9/29/2014 08:58)...
What other "analysis features" would you like to see the Fritz GUI add?
------
I would like to see the following features added - Option to have the full game analysis (Compare Analysis, or Blunder Check)...

A. Calculate and report the best move in the position for BOTH colors, not just for the color that is on-turn. I understand an annotated game better when I know the threats vs when I do not.

B. Tag with the "Only Move" symbol (a square) whenever the best move has a 'significantly' better evaluation than the second best move. The definition of 'significantly' should be a numeric threshold that is tuneable by the user.
rkjulian rkjulian 10/1/2014 03:11
I have a perhaps larger question about chess engines: An FM suggest that maybe chess engines are being tuned to beat each other more than they are being improved to reach the truth of a position. A higher rating is almost the only basis people have for choosing an engine. In a world where human-computer teams are more and more important in many fields (including medicine), shouldn't engines be evaluated by how much assistance they provide humans in finding the "truth"? Are engines evolving in the right direction?
Edwin Bakker Edwin Bakker 10/1/2014 01:25
Quote: "An FM suggest that maybe chess engines are being tuned to beat each other more than they are being improved to reach the truth of a position"
That can only be reached by playing the game in a better way.
Nakamura tried to defeat Stockfish 5 with pawn odds and with the aid of Rybka! He was defeated:
http://schaken.chess.com/news/stockfish-outlasts-nakamura-3634
Stockfish was not even allowed to access either an opening book or an endgame tablebase.
Moreover, Stockfish was running on an old Macbook from 2008.
This was all to compensate, but to no avail!

Hard for people to admit, but modern chess engines are superior to GMs.
Archie Shipp Archie Shipp 1/3/2015 01:11
I agree with John's assessment of the capabilities of Houdini 4, Komodo 8, and Stockfish 5. I have and use all three for single move analysis and see very little difference even when running them in tournaments against each other. The one thing that I would add goes toward his second point about " the quality of their analysis – the ‘humanness’," In this respect, when I am working on full game analysis of my games for possible publication, I prefer to go all the way back to the last free version of Rybka 2.3.2. Anything more advanced tends to look into the position much deeper than my feeble mind can follow, and would bore the heck out of all but the most advanced grandmaster readers.
maxath maxath 4/24/2016 04:13
How to properly load a chess engine on Fritz, as Komodo, StockFish or Houdini:

https://www.youtube.com/watch?v=Sohmt3X8ezs
Sergiy Matis Sergiy Matis 5/27/2016 08:45
Yes that is really cool work.And where is your collective works on blue-ray disc?
1