John Hartmann: And then there were two

by John Hartmann
6/9/2015 – Currently there are two programs vying for the title of strongest chess engine in the world, and both have seen new releases in recent months. Komodo and Stockfish currently top all computer lists, in that order. But they have different personalities, as John Hartmann shows in his engine review. "Shows" includes both engines simultaneously analyzing in an interesting realtime video.

ChessBase 14 Download ChessBase 14 Download

Everyone uses ChessBase, from the World Champion to the amateur next door. Start your personal success story with ChessBase 14 and enjoy your chess even more!


Along with the ChessBase 14 program you can access the Live Database of 8 million games, and receive three months of free ChesssBase Account Premium membership and all of our online apps! Have a look today!

More...

And Then There Were Two

Review by John Hartmann

Now that Houdini seems to have gone gentle into that good night, there are two engines vying for the title of strongest chess engine in the world. Those two engines – Stockfish and Komodo – have each seen new releases in recent months. Stockfish 6 was released at the end of January, while Komodo 9 became available at the end of April from komodochess.com and the end of May from ChessBase.

Komodo 9, written by Don Dailey, Larry Kaufman and Mark Lefler. Available

  1. with Fritz GUI from Amazon ($80ish as of 5/28),

  2. for download with Fritz GUI from ChessBase.com ($73.50 w/o VAT as of 5/28) and

  3. directly from the Komodo website without GUI for $59.98.

Also available as part of a one year subscription package for $99.97.

Stockfish 6, written by the Stockfish Collective. Open-source and available at the Stockfish website.

Last year I wrote a review of Komodo 8 and Stockfish 5 that was republished at ChessBase.com, and much of what I wrote there applies here as well. Fear not, frazzled reader: you don’t need to go back and read that review, as most of the key points will be reiterated here.

First things first: any top engine (Komodo, Stockfish, Houdini, Rybka, Fritz, Hiarcs, Junior, Chiron, Critter, Equinox, Gull, Fire, Crafty, among many others) is plenty strong to beat any human player alive. This is not because each of these engines are equally strong. While they don’t always play the absolute best moves, none of the aforementioned engines ever make big mistakes. Against fallible humans, that’s a recipe for domination. It’s nearly useless – not to mention soul-crushing! – to play full games against the top engines, although I do recommend using weaker engines (Clueless 1.4, Monarch, Piranha) as sparring partners for playing out positions or endgames.

Even if all the major engines can beat us, they’re not all created equal. Three major testing outfits – CCRL, CEGT, and IPON – engage in ongoing and extensive testing of all the best engines, and they do so by having the engines play thousands of games against one another at various time controls. In my previous review I noted that Komodo, Stockfish and Houdini were the top three engines on the lists, and in that order. This remains the case after the release of Komodo 9 and Stockfish 6:

CCRL (TC 40 moves/40 min, 4-cpu computers):

  1. Komodo 9, 3325 (Komodo 8 was rated 3301)
  2. Stockfish 6, 3310 (Stockfish 5 was rated 3285)
  3. Houdini 4, 3269

CEGT

  • 40/4: 1. Komodo 9, 2. Stockfish 6, 3. Houdini 4
  • G/5’+3”: 1. Komodo 9, 2. Stockfish 6, 3. Houdini 4
  • 40/20: 1. Komodo 9, 2. Stockfish 6, 3. Houdini 4 (NB: list includes multiple versions of each engine)
  • 40/120: 1. Stockfish 6, 2. Komodo 8 (does not yet include version 9), 3. Houdini 4 (list includes multiple versions of each engine)

IPON

  1. Komodo 9, 3190 (Komodo 8 was 3142)
  2. Stockfish 6, 3174 (Stockfish 4 was 3142)
  3. Houdini 4, 3118

The results are fairly clear. Komodo 9 is ever so slightly stronger than Stockfish 6 when it comes to engine-engine play, and this advantage seems to grow when longer time controls are used.

For my purposes, though, what’s important is an engine’s analytical strength. This strength is indicated by engine-engine matches, in part, but it is also assessed through test suites and – perhaps most importantly – by experience. Some engines might be more trustworthy in specific types of positions than others or exhibit other misunderstandings. Erik Kislik, for instance, reports in his April 2015 Chess Life article on the TCEC Finals – some of which appeared in his earlier Chessdom piece on TCEC Season 6 – that only Komodo properly understood the imbalance of three minor pieces against a queen. There are undoubtedly other quirks known to strong players who use engines on a daily basis.

In my previous review I ran Komodo, Stockfish and Houdini (among others) through two test suites on my old Q8300. Since then I’ve upgraded my hardware, and now I’m using an i7-4790 with 12 GB of RAM and an SSD for the important five and six-man Syzygy tablebases included with ChessBase’s Endgame Turbo 4. (Note: if you have an old-fashioned hard drive, only use the five-man tbs in your search; if you use the six-man, it will slow the engine analysis down dramatically.) Because I have faster hardware I thought that a more difficult test suite would be in order, and – lucky me! – just such a suite was recently made available in the TalkChess forums. I gave Komodo 9 and Stockfish 6 one minute per problem to solve the 112 problems in the suite, and the results were as follows:

  • Komodo 9 solved 37 out of 110 problems (33.6%) with an average time/depth of 20.04 seconds and 24.24 ply.
  • Stockfish 6 solved 30/110 (27.2%) with an average time/depth of 20.90 seconds and 29.70 ply.

Note that while there are 112 problems in the suite, two of them were rejected by both engines because they had incomplete data.) A CBV archive of the test suite database can be found here.

I have also been using both Komodo 9 and Stockfish 6 in my analytical work and study. So that you might also get a feeling for how each evaluates typical positions, I recorded a video of the two at work.  Each engine ran simultaneously (2 cpus, 2gb of RAM) as I looked at a few games of interest, most of which came from Alexander Baburin’s outstanding e-magazine Chess Today. The video is 14 minutes long.

Komodo 9 and Stockfish 6 in comparative analysis – here are the games to replay and experiment with

Even a brief glance at the above video will make clear just how good top engines are becoming in their ability to correctly assess positions, but it also shows (in Gusev-Averbakh) that they are far from perfect. They rarely agree fully in positions that are not clear wins or draws, and this is due to the differences in evaluation and search between the two. Broadly speaking, we can say that evaluation is the criteria or heuristics used by each engine to ‘understand’ a position, while search is the way that the engine ‘prunes’ the tree of analysis. While many engines might carry similar traits in their evaluation or search, none are identical, and this produces the differences in play and analysis between them.

Stockfish 6 is a rather deep searcher. It achieves these depths through aggressive pruning of the tree of analysis. While there are real advantages to this strategy, not the least of which is quick analytical sight and tactical ingenuity, there are some drawbacks. Stockfish can miss some resources hidden very deep in the position. I find it to be a particularly strong endgame analyst, in part because it now reads Syzygy tablebases and refers to them in its search. Stockfish is an open-source program, meaning that it is free to download and that anyone can contribute a patch, but all changes to evaluation or search are tested on a distributed network of computers (“Fishtest”) to determine their value.

Komodo 9 is slightly more aggressive in its pruning than is Komodo 8, and it is slightly faster in its search as well. (Both changes seem to have been made, to some degree, with the goal of more closely matching Stockfish’s speed – an interesting commercial decision.) While Komodo’s evaluation is, in part, automatically tuned through automated testing, it is also hand-tuned (to what degree I cannot say) by GM Larry Kaufman.

The result is an engine that feels – I know this sounds funny, but it’s true – smart. It seems slightly more attuned to positional nuances than its competitors, and as all the top engines are tactical monsters, even a slight positional superiority can be important.  I have noticed that Komodo is particularly good at evaluating positions where material imbalances exist, although I cannot say exactly why this is the case!

As more users possess multi-core systems, the question of scaling – how well an engine is able to make use of those multiple cores – becomes increasingly important. Because it requires some CPU cycles to hand out different tasks to the processors in use, and because some analysis will inevitably be duplicated on multiple CPUs, there is not a linear relation between number of CPUs and analytical speed.

Komodo 8 was reputedly much better than Stockfish 5 in its implementation of parallel search, but recent tests published on the Talkchess forum suggest that the gap is narrowing. While Stockfish 6 sees an effective speedup of 3.6x as it goes from 1 to 8 cores, Komodo 9’s speedup is about 4.5x. And the gap is further narrowed if we consider the developmental versions of Stockfish, where the speedup is now around 4x.

Hardcore engine enthusiasts have, as the above suggests, become accustomed to downloading developmental versions of Stockfish. In an effort to serve some of the same market share, the authors of Komodo have created a subscription service that provides developmental versions of Komodo to users. This subscription, which costs $99.97, entitles users to all official versions of Komodo released in the following year along with developmental versions on a schedule to be determined. Only those who order Komodo directly from the authors are currently able to choose this subscription option.

The inevitable question remains: which engine should you choose? My answer is the same now as it was in my previous review. You should choose both – and perhaps more.

Both Komodo and Stockfish are insanely strong engines. There remain some positions, however, where one engine will get ‘stuck’ or otherwise prove unable to discern realistic (i.e. human) looking moves for both sides. In that case it is useful to query another engine to get a second (or perhaps even third) opinion. I find myself using Komodo 9 more than Stockfish 6 in my day-to-day work, but your mileage may well vary. Serious analysts, no matter their preference, will want to have both Komodo 9 and Stockfish 6 as part of their ‘teams.’

About the author

John Hartmann is an award-winning chess book reviewer, a chess teacher, organizer, and tournament director in Omaha, Nebraska. Currently he writes for Chess Life and his own website, Chess Book Reviews, where the above article originally appeared.

John is a coffee aficionado (like Gale?), a weekend baseball player, a Cardiff City supporter, and father of an infant daughter who has finally figured out how to sleep through the night. He also tries, from time to time, to fit in some work on his dissertation in philosophy.

Komodo Chess 9 by ChessBase includes:

  • The Komodo 9.01 engine, which supports up to 64 processor cores and 16 GB of hash memory;

  • The Deep Fritz 64-bit program interface (+ 32 bit program interface);

  • A ChessBase PREMIUM Account: six months online access to Playchess.com, ChessBase live database, Let’s Check, Engine Cloud, Tactics Training.

Price: €79.90. Update from Version 8: €39,90 including six months ChessBase Premium.

System requirements

Minimum: Pentium III 1 GHz, 2 GB RAM, Windows Vista, XP (Service Pack 3), 7/8, DirectX9, 256 MB graphics card, DVD-ROM drive, Windows Media Player 9 and Internet access for program activation, access to Playchess.com, Let’s Check and program updates.

Recommended: PC Intel i7 (Quadcore), 4 GB RAM, Windows 8.1, DirectX10, 512 MB graphics card, 100% DirectX10-compatible sound card, Windows Media Player 11, DVD-ROM drive and Internet access for program activation, access to Playchess.com, Let's Check and program updates.

Price: Komodo Chess 9 including six months ChessBase Premium account: €79.90. Update from Version 8: €39,90 including six months ChessBase Premium.

Order Komodo Chess 9 in the ChessBase shop now!



... is an award-winning chess book reviewer, a chess teacher, organizer, and tournament director in Omaha, Nebraska. Currently he writes for Chess Life, the British Chess Magazine, and his own website Chess Book Reviews. John is is a coffee aficionado, a weekend baseball player, a Cardiff City supporter, a father-to-be currently working on his dissertation in philosophy.
Discussion and Feedback Join the public discussion or submit your feedback to the editors


Discuss

Rules for reader comments

 
 

Not registered yet? Register

Jd1985 Jd1985 6/12/2015 07:30
LucieM : Let me assure you, my friend, that this thing is a monster. It's like a turbo charged Houdini 5 on steroids. While a lot of things out there are free, this is far from a bad bargain. Once you buy a program, you get subsequent upgrades for 1/3rd of the price (should a Komodo 10 come along).

By the way, do let me know how 80 Euros can go anywhere towards a new CPU, maybe I can assemble a super-computer.
ivan3ivanovich ivan3ivanovich 6/10/2015 03:17
@LucieM
You missed the whole point of the article.

What (artificial) ELO-rating the engines achieve in games against other engines tells us nothing about how well they can analyse a position on the board.

Choosing an engine to work with for analysis and preparation requires that you look at how well that engine handles the kind of positions that you're likely to encounter and that cannot in any way be assessed by looking at a number that measures something completely different.
Werewolf Werewolf 6/10/2015 02:57
The improvement is around 40 elo, not 15.
algorithmy algorithmy 6/10/2015 09:20
The first time Fritz9 was published we thought that's it!! it was just too strong for human, then Rybka came to put shame on Fritz, and we thought that's it!! Then Houdini came in and we thought what better could be and we thought that's it, and now they say that Komodo can beat Houdini(they say that, I didn't see it!), so the question is when does it end?
LucieM LucieM 6/10/2015 06:21
80 euros to gain 15 pts elo? I prefer to invest in a new CPU...
Karbuncle Karbuncle 6/9/2015 11:15
Being where I am on ICCF, I often get questions like "Which engine do you prefer?". I tell them the reality is I prefer to use both K9 and SF6 and cross-reference. The difference in strength between them is microscopic, and lets not forget that this difference is typically measured on rating lists where the engine is only taking minutes per move. When I use these engines, it's DAYS per move with a lot of forward-sliding, back-sliding, subtractive analysis, and of course cross-referencing. Often times they agree on the best move, but some times they can have blind spots and miss something the other engine catches.
1