Komodo 8: the smartphone vs desktop challenge

by Albert Silver
9/24/2014 – After hearing two strong players argue that the only real progress in chess engines in the last ten years was due to faster computers a special match was played to challenge this idea. Komodo 8 ran on a smartphone while a top engine of 2006 used a modern i7 computer that runs 50 times faster. This is the difference between Usain Bolt and the Concorde. Guess what happened?

ChessBase 18 - Mega package ChessBase 18 - Mega package

Winning starts with what you know
The new version 18 offers completely new possibilities for chess training and analysis: playing style analysis, search for strategic themes, access to 6 billion Lichess games, player preparation by matching Lichess games, download Chess.com games with built-in API, built-in cloud engine and much more.

More...

Over the last 20 years, and even more so the last decade, we have seen technology advancing by leaps and bounds in ways that would shock someone from only 1990. Hardware has advanced so much that our smartphones of today (the first iPhone dates back only seven years) are far more computer than phone, but software has also advanced hand-in-hand with this process. The strength of chess engines has also taken off, but it is sometimes hard for the layman to differentiate how much of the progress is due to hardware and how much to software. The answer is both of course. 

Still, it was a bit of a shock to listen to a conversation between two strong players, one of whom is a grandmaster, claim that engines had really not progressed at all in the past ten years, and that the Elo leaps the programmers were taking credit for, were entirely due to the faster hardware. Needless to say, for someone who knows several of these programmers, and the enormous time and effort expended, to see them classified as con artists did not sit well with me.

I realized the statements were not meant maliciously, and were sincere in their utter ignorance. After some thought I realized the vast majority of users are probably unclear on just how much progress has been made in pure programming terms. There are of course ratings lists of engines, but the numbers are so outrageously high over human ratings that it becomes hard to fit them into any normal perspective. Consider one of the top ratings lists over the past years: CCRL (Computer Chess Ratings List).

Komodo 8 tops the ratings lists for computer chess

At the top on a single core is Komodo 8 with a mere 3232 Elo. For most of us, the near 2900 rating of Magnus Carlsen is already approaching science fiction, so where does 3232 fit in the picture? That is today though. The best engine 10 years ago was in fact Shredder 8, but since the list does not go back that far, we will make do with Shredder 9, rated…. 2725.

Although a top engine in its day, the lists suggests that in pure software there is a 500 Elo gap

That suggests that software has advanced over 500 Elo in ten years, without factoring in the faster computers themselves. Can this be true?

I decided to run a small experiment. Since Komodo 8 exists on both the desktop as well as the Android smartphone, I decided to play a small match between it on a smartphone, facing Shredder on a modern, top of the line quad-core i7 processor. I ran a small test to compare the speeds of the platforms, by running Komodo on the start position and seeing how long it took to reach 20 plies. The result was that it ran about 50 times slower on a smartphone. Not 50%, no, 50 times.

To put that in perspective, if Komodo were racing at the speed of Usain Bolt at his fastest (roughly 45km/h) then Shredder would be racing in a Concorde at 2250 km/h. Therefore if Komodo’s edge over the best of yesteryear was in fact only 150-200 Elo, then the fantastic hardware advantage by Shredder should allow it to crush it. Although I actually own Shredder 8, and could dig it up, it is unable to make use of more than one core (2004 after all), so instead I used Deep Shredder 10, from 2006, which is not only much stronger but able to make full use of the quad-core i7.

A view of Shedder 10 using an i7, facing Komodo 8 running on a smartphone

Three positions were chosen to be played with reverse colors, each engine playing it both as white and black, at a time control of fifteen minutes with a ten second increment. Why only six games? Quite simply because I had to do this by hand, and moreover, was forced to manually tell Komodo when to play. The reason is that it was not designed to play against an opponent on a smartphone, just as an analysis engine, so I would enter Shredder’s move, let Komodo analyze, and then tell it to play its move.

 

I took the opportunity of recording the sixth and last game in such a way that readers might
follow the analysis of Shredder 10, but also Komodo 8 on the smartphone at the same time.

The final result after six games was a merciless crush by Komodo 8 with four wins and two draw, winning by 5-1. Had the match been closer, I might have considered extending it to see how it progressed, but 5-1 is pretty clear. As a reference, the performance over the six games was a 280 Elo lead, therefore if you accept that Deep Shredder 10 on an i7 is at least 2800 Elo (it was rated 2770 on a slower single-core system), then Komodo 8 is well over 3000 Elo on an Android smartphone. For the record, the smartphone in question was an LG Optimus G Pro from 2013.

The games of the match

[Event "Rapid 15m+10s"] [Site "Rio de Janeiro, Brazil"] [Date "2014.09.22"] [Round "1"] [White "Deep Shredder 10 UCI"] [Black "Komodo 8, smartphone"] [Result "1/2-1/2"] [ECO "D51"] [PlyCount "125"] 1. d4 Nf6 2. c4 e6 {This was the starting position. After this, the engines were on their own.} 3. Nc3 d5 4. Bg5 Nbd7 5. e3 h6 6. Bxf6 Nxf6 7. Nf3 c6 8. Be2 dxc4 9. Bxc4 Bd6 10. e4 Be7 11. O-O O-O 12. Qd3 a5 13. e5 Nd5 14. Bb3 b6 15. Nxd5 cxd5 16. Bc2 g6 17. Ba4 Bd7 18. Bb3 a4 19. Bc2 Rc8 20. Rac1 Kg7 21. Qe3 Bb5 22. Rfe1 Qd7 23. a3 Rc7 24. Bb1 Rfc8 25. Rxc7 Rxc7 26. h4 Bc4 27. Qd2 Ba6 28. g3 Qb5 29. Kg2 Rc6 30. Nh2 h5 31. Nf3 Rc7 32. Rc1 Rxc1 33. Qxc1 Qe2 34. Bc2 b5 35. Qd2 b4 36. Qxe2 Bxe2 37. axb4 Bb5 38. Nd2 Bxb4 39. Nb1 Ba5 40. Kf3 f6 41. exf6+ Kxf6 42. Ke3 e5 43. dxe5+ Kxe5 44. Bxg6 Bb6+ 45. Kd2 Bxf2 46. Nc3 Bd7 47. Ne2 d4 48. Bxh5 Be3+ 49. Kc2 Bf5+ 50. Kd1 Bf2 51. Kd2 d3 52. g4 Bh7 53. Nc1 Kd4 54. Nxd3 Bxd3 55. Be8 Bxh4 56. Bxa4 Bc4 57. Bc2 Bg5+ 58. Kd1 Bb5 59. Bf5 Ba4+ 60. Bc2 Be8 61. Bf5 Bc6 62. Be6 Ba4+ 63. Ke2 1/2-1/2 [Event "Rapid 15m+10s"] [Site "Rio de Janeiro, Brazil"] [Date "2014.09.22"] [Round "2"] [White "Komodo 8, smartphone"] [Black "Deep Shredder 10 UCI"] [Result "1-0"] [ECO "E46"] [PlyCount "141"] 1. d4 Nf6 2. c4 e6 {This was the starting position. After this, the engines were on their own.} 3. Nc3 Bb4 4. e3 O-O 5. Bd2 b6 6. Bd3 Ba6 7. a3 Bxc3 8. Bxc3 d5 9. b3 Nbd7 10. Nf3 Ne4 11. Bb2 f5 12. cxd5 Bxd3 13. Qxd3 exd5 14. O-O c6 15. Ne5 Rc8 16. Rfe1 Nxe5 17. dxe5 Qh4 18. Qc2 c5 19. f3 Ng5 20. Rad1 Rc6 21. Rf1 Rh6 22. g3 Qh5 23. Qg2 Rd8 24. Rd2 Rg6 25. Kh1 Ne6 26. f4 Nc7 27. Bc3 a6 28. b4 c4 29. Kg1 Rh6 30. a4 Re8 31. Re1 Kh8 32. Bd4 Rb8 33. a5 bxa5 34. bxa5 Rb5 35. Bb6 Ne8 36. Rdd1 Rb3 37. Rf1 Rb4 38. Rb1 Rxb1 39. Rxb1 Qf7 40. Rd1 Rc6 41. Rxd5 Rc8 42. Rd1 Kg8 43. e4 fxe4 44. Qxe4 c3 45. Qd3 c2 46. Rc1 Qc4 47. Qxc4+ Rxc4 48. Kf2 Nc7 49. Bxc7 Rxc7 50. Ke3 Kf7 51. Kd3 Rd7+ 52. Kxc2 Rd5 53. Kb3 Rxa5 54. Rc7+ Kf8 55. e6 Rf5 56. Kc4 a5 57. Kd4 Rb5 58. Rf7+ Kg8 59. Ra7 Kf8 60. Ke4 Rb2 61. h4 a4 62. Rxa4 Ke7 63. f5 Rb7 64. g4 g6 65. Rd4 gxf5+ 66. gxf5 Kf6 67. Rd8 Ra7 68. Rf8+ Kg7 69. Rc8 Kf6 70. h5 Ra4+ 71. Kd5 1-0 [Event "Rapid 15m+10s"] [Site "Rio de Janeiro, Brazil"] [Date "2014.09.22"] [Round "3"] [White "Deep Shredder 10 UCI"] [Black "Komodo 8, smartphone"] [Result "0-1"] [ECO "A28"] [PlyCount "256"] 1. c4 e5 {This was the starting position. After this, the engines were on their own.} 2. Nc3 Nf6 3. Nf3 Nc6 4. e4 Bb4 5. d3 Bc5 6. Nxe5 Nxe5 7. d4 Bb4 8. dxe5 Nxe4 9. Qd4 Nxc3 10. bxc3 Ba5 11. Ba3 b6 12. Rc1 c5 13. Qd5 Rb8 14. Qd6 Bb7 15. Be2 Rc8 16. O-O Qg5 17. g3 Qf5 18. Rcd1 Qh3 19. f3 Bxc3 20. Rd3 Ba5 21. g4 Bc6 22. Re3 h5 23. g5 Qe6 24. Bb2 Kd8 25. f4 g6 26. Qd1 Kc7 27. Qc2 Kb8 28. Rd1 Rhe8 29. Ra3 Qe7 30. Bf1 Rc7 31. Bh3 Bb4 32. Re3 Rcc8 33. Qb3 Rc7 34. Red3 Qd8 35. Rd6 Re7 36. R1d3 Qc8 37. a3 Ba5 38. Rd1 Qb7 39. Bc3 Bf3 40. Rf1 Bxc3 41. Qxc3 Bc6 42. Ra1 Re8 43. a4 Ka8 44. Bf1 Re6 45. Rd2 a5 46. Bh3 Re7 47. Rb1 Be4 48. Rb3 Qa6 49. Rd6 Bc6 50. Rb2 Qb7 51. Qb3 Ka7 52. Rb1 Qb8 53. Rd2 Qb7 54. Rdb2 Be4 55. Rd1 Bf3 56. Rd6 Bc6 57. Qa3 Re8 58. Rb1 Re7 59. Qb3 Qb8 60. Rbd1 Qb7 61. Rf1 Be4 62. Rf2 Rc6 63. Rdd2 Rc8 64. Rd1 Rc7 65. Rdd2 Rc8 66. Rd6 Rc6 67. Rd1 Rc7 68. Rfd2 Bf3 69. Rb1 Bc6 70. Rbb2 Be4 71. Qc3 Bc6 72. Qc2 Be4 73. Qb3 Bf3 74. Rd3 Be4 75. Re3 Bc6 76. Rd2 Kb8 77. Re1 Ka7 78. Rd6 Rc8 79. Rb1 Rc7 80. Rd2 Re8 81. Rd6 Re7 82. Rf1 Be4 83. Qa3 Rc6 84. Rd2 Rc7 85. Qb3 Kb8 86. Rfd1 Bf3 87. Ra1 Bc6 88. Rd6 Ka7 89. Qb2 Bf3 90. Qb3 Bc6 91. Rad1 Be4 92. Qb2 Bc6 93. Qc3 Bxa4 94. Ra1 Bc6 95. Rd2 Re8 96. Rd6 Rcc8 97. Rad1 Re7 98. Rb1 Rb8 99. Rbd1 Ra8 100. Rb1 a4 101. Qc2 Kb8 102. Qc3 a3 103. Rd2 Ra4 104. Ra2 Qa6 105. Bf1 Re6 106. Rb3 Qa5 107. Qc1 d6 108. Qd1 dxe5 109. Bh3 Kb7 110. Bxe6 fxe6 111. fxe5 Rxc4 112. Rbxa3 Rd4 113. Qa1 Qb4 114. Ra7+ Kb8 115. Rf7 Rg4+ 116. Kf1 Rf4+ 117. Rxf4 Qxf4+ 118. Rf2 Bb5+ 119. Kg1 Qxg5+ 120. Rg2 Qe3+ 121. Rf2 Bd3 122. Kg2 Bf5 123. Kg1 c4 124. Qd1 Bd3 125. Qd2 Qd4 126. Qf4 Qxf4 127. Rxf4 c3 128. Rf8+ Kb7 0-1 [Event "Rapid 15m+10s"] [Site "Rio de Janeiro, Brazil"] [Date "2014.09.23"] [Round "4"] [White "Komodo 8, smartphone"] [Black "Deep Shredder 10 UCI"] [Result "1/2-1/2"] [ECO "A28"] [PlyCount "162"] 1. c4 e5 {This was the starting position. After this, the engines were on their own.} 2. Nc3 Nf6 3. Nf3 Nc6 4. e4 Bc5 5. Nxe5 Nxe5 6. d4 Bb4 7. dxe5 Nxe4 8. Qd4 Nxc3 9. bxc3 c5 10. Qe3 Ba5 11. Be2 O-O 12. Ba3 b6 13. Bf3 Rb8 14. O-O Ba6 15. Bd5 d6 16. Rad1 dxe5 17. Qxe5 Qd7 18. Bc1 Rbe8 19. Qg3 Qf5 20. Rd3 Qg6 21. Bf4 Rd8 22. Re1 Qxg3 23. hxg3 Rfe8 24. Rde3 Rxe3 25. Rxe3 Kf8 26. Bc7 Rd7 27. Bc6 Rd1+ 28. Kh2 g5 29. Bd5 Rd2 30. Rf3 Bb7 31. Bxb7 Rd7 32. Be5 Rxb7 33. Bf6 h6 34. Rd3 Rb8 35. Kh3 b5 36. cxb5 Rxb5 37. Rd7 Bb6 38. Rb7 Ke8 39. Re7+ Kf8 40. a4 Ra5 41. Re4 Bc7 42. f4 gxf4 43. gxf4 Bd6 44. Kg4 Ra6 45. Kh5 Rb6 46. g3 Ra6 47. f5 Rc6 48. c4 Rb6 49. g4 a5 50. Bd8 Rb8 51. Bxa5 Rb3 52. g5 hxg5 53. Bd8 Ra3 54. a5 Bf4 55. f6 Ra2 56. Bb6 Kg8 57. Re7 Ra1 58. Re8+ Kh7 59. Rf8 Rh1+ 60. Kg4 Kg6 61. a6 Ra1 62. Rg8+ Kxf6 63. a7 Ra3 64. Bd8+ Ke6 65. Bxg5 Bxg5 66. a8=Q Rxa8 67. Rxa8 Bf6 68. Ra3 Ke5 69. Rf3 Bh8 70. Rh3 f5+ 71. Kf3 Bf6 72. Rh6 Bg5 73. Rg6 Bd2 74. Rg7 Kd4 75. Rd7+ Kc3 76. Rd5 Bc1 77. Rxc5 Ba3 78. Ra5 Bb4 79. Rxf5 Kxc4 80. Ke4 Bc5 81. Rxc5+ Kxc5 1/2-1/2 [Event "Rapid 15m+10s"] [Site "Rio de Janeiro, Brazil"] [Date "2014.09.23"] [Round "5"] [White "Komodo 8, smartphone"] [Black "Deep Shredder 10 UCI"] [Result "1-0"] [ECO "C51"] [PlyCount "139"] 1. e4 e5 2. Nf3 Nc6 3. Bc4 Bc5 4. b4 Bxb4 5. c3 {This was the starting position. After this, the engines were on their own.} Bd6 6. d4 Nf6 7. O-O O-O 8. Bd3 a6 9. Nbd2 b5 10. Nb3 Bb7 11. a4 h6 12. h3 Qb8 13. Nc5 Bxc5 14. dxc5 bxa4 15. Rb1 Qa7 16. Nh2 Rae8 17. Ng4 Nxg4 18. Qxg4 Re6 19. Bc4 Rf6 20. Be3 Bc8 21. Qe2 Na5 22. Ba2 Qa8 23. Rb4 a3 24. Ra4 Nc6 25. Rxa3 Ne7 26. f3 Ng6 27. g3 Rd8 28. Ra5 Ne7 29. Rb1 Rf8 30. Bc4 Bb7 31. h4 Nc6 32. Ra4 a5 33. Rba1 Qa7 34. Rd1 Qa8 35. Rxd7 Rd8 36. Rd5 Bc8 37. Rxd8+ Nxd8 38. Bd5 c6 39. Bb3 Be6 40. Bc2 Rg6 41. Kh2 h5 42. Ra1 Nb7 43. Rb1 a4 44. Rb4 a3 45. Ra4 Qe8 46. Rxa3 Qf8 47. Qf2 Bc4 48. Ra7 Qc8 49. f4 Qb8 50. Ra4 Bb5 51. Rb4 exf4 52. Bxf4 Qa8 53. e5 Re6 54. Bb3 Rg6 55. Be3 Nd8 56. Qf3 Qc8 57. Qxh5 Bf1 58. g4 Ne6 59. Bc2 Qc7 60. Kg1 Bb5 61. Bxg6 fxg6 62. Qxg6 Qxe5 63. Qe8+ Kh7 64. Qh5+ Qxh5 65. gxh5 Bd3 66. Kf2 Bf5 67. Rb7 Bc2 68. h6 Kg8 69. hxg7 Nxg7 70. Kf3 1-0 [Event "Rapid 15m+10s"] [Site "Rio de Janeiro, Brazil"] [Date "2014.09.23"] [Round "6"] [White "Deep Shredder 10 UCI"] [Black "Komodo 8, smartphone"] [Result "0-1"] [ECO "C51"] [PlyCount "108"] 1. e4 e5 2. Nf3 Nc6 3. Bc4 Bc5 4. b4 Bxb4 5. c3 {This was the starting position. After this, the engines were on their own.} Bd6 6. d4 Nf6 7. O-O O-O 8. Re1 h6 9. Nbd2 Re8 10. Bb2 Rb8 11. Qb3 Re7 12. Bb5 Na5 13. Qc2 b6 14. c4 Bb4 15. dxe5 Nh5 16. a3 Bxd2 17. Nxd2 Nf4 18. Qc3 Ne6 19. Qg3 Nc5 20. Bd4 Nab7 21. Nb3 a6 22. Bxc5 Nxc5 23. Nxc5 axb5 24. Nb3 bxc4 25. Nd4 Qf8 26. Rac1 Ba6 27. Red1 Rbe8 28. f4 d6 29. Nc6 Re6 30. f5 Rxe5 31. Nxe5 Rxe5 32. f6 b5 33. Re1 g6 34. Qe3 Kh7 35. Qd4 Qe8 36. Re3 Qc6 37. Rce1 Bb7 38. h4 Rh5 39. g3 Re5 40. Kf2 Qd7 41. a4 Qe6 42. axb5 Rxb5 43. Rc1 Rb4 44. Rd1 Rb3 45. Rxb3 cxb3 46. Rd3 Bxe4 47. Re3 Qf5+ 48. Ke1 c5 49. Qc3 Bd5 50. Kd2 c4 51. Re2 b2 52. Qxb2 Qd3+ 53. Kc1 c3 54. Qb4 Qxe2 0-1

If anyone had any doubt as to the genuine progress of chess engines, and the merit of buying new versions, they can lay these concerns to rest. Chess engines have not only progressed over the last ten years, but there is reason to believe they have outpaced the hardware even. If you want the latest and greatest, Komodo 8 is the king of the hill.

Komodo 8 can be purchased in the ChessBase Shop

Born in the US, he grew up in Paris, France, where he completed his Baccalaureat, and after college moved to Rio de Janeiro, Brazil. He had a peak rating of 2240 FIDE, and was a key designer of Chess Assistant 6. In 2010 he joined the ChessBase family as an editor and writer at ChessBase News. He is also a passionate photographer with work appearing in numerous publications, and the content creator of the YouTube channel, Chess & Tech.

Discuss

Rules for reader comments

 
 

Not registered yet? Register

axiom axiom 1/2/2015 08:33
Albert Silver: Komodo 8 on a smartphone reaches superior depth, being more than 5 years younger than Shredder, and thus 4 cpu on Shredder really doesn't matter. You could have the processing power of Deep Blue against Komodo on a phone, and it would still lose because of this.

It doesn't matter how many processors and positions it searches per minute, if the other engine is a generation ahead on depth. So your study did not prove anything:)

yurikvelo yurikvelo 11/24/2014 08:34
People claiming
1) big hardware improvement
2) no software improvement
and particularly Peter Swidler (commenting WCC-2014) do not know what they talk about.
1) Hardware became ridiculously slow (but very cheap) both in terms of MIPS and N/sec.
2) Despite 10..100-fold dropdown in HW speed, software made huge leap in playing strength.
20-30 kn/sec DroidFish or Komodo wouldn't leave a chance to mighty Deep Blue-2 with its 200 MN/sec or Hydra with 40-50 MN/sec
EddieFreak EddieFreak 9/26/2014 12:11
I agree with MrL2014. I would have liked to have seen this tested using older hardware. example PIII 600mhz. Take away all those little instructions sets that the older engine can't use.
PurpleUnicorns PurpleUnicorns 9/25/2014 10:52
Chessbase needs to add the feature to be able to edit posts, I made so many grammar mistakes just now =(
PurpleUnicorns PurpleUnicorns 9/25/2014 10:50
To those criticizing the openings keep in mind the openings were symmetric in the sense each engine played each side once so any claim of inferiority (which is ridiculous anyhow as the openings are just normal and fine choices) would if at all only bias the result towards a 50% score.

Secondly no, I'm pretty sure the engines are not optimized towards using there own opening books. If anything the optimization would be to make a book to an engines strength. Either way a more modern engine would if anything again have an edge here, though I don't believe it would make any difference at all.

Finally I can point you all to the TCEC, the Thorensen Chess Engine Competition which is probably the most respected chess modern chess engine competition nowadays. One of the stages this season will be played without book and previously there was a chess960 competition. The chess960 competition only further showed that with or without book, the top engines dominate the competition.

Speaking of that chess960 competition, Stockfish scored 25.0/28 which is ridiculous. For comparison, the newest version of Rybka scored 14.5/28 and the newest version of shredder scored 7.5/28

The engines of 10 years ago are so much weaker then today's beasts its not even funny.
Rama Rama 9/25/2014 08:00
Komodo's place on the CCRL rating list was based upon opponents rated lower than Stockfish's were and on hundreds of fewer games. Why don't they just do a quadruple round robin between only the top three engines to settle this thing once and for all?
joe99 joe99 9/25/2014 07:17
it is about the quality of the chess engine and not the quantity , two heads are not better than one and speed calculation shouldn't make a big factor either. if you have an Engine that knows how to play with high quality moves it will easily win against an engine that is calculating at high speed but is making mistakes deep in its calculation that cannot come to the right conclusion of a position . you could be have the fastest car in the world but if the driver can't drive then it is as useful as a bicycle with one wheel.
Bertman Bertman 9/25/2014 05:34
@ISeeThis Yes, same, but I had never really thought about it until you brought it up. In case you are curious, here are the computer logs of Deep Blue showing its analysis for each move.

https://www.research.ibm.com/deepblue/watch/html/c.shtml
iSeeThis iSeeThis 9/25/2014 03:48
@Bertman @jhoravi
Deep Blue was my Sci-fi fantasy. I then always dreamed of having Deep Blue in my palm. Just didn't dream that it would happen before I die (or even old :-D)
MrL2014 MrL2014 9/25/2014 02:57
The question is if a 2006 software can make use of a 2014 processor in an efficient way. Not to say that there was not a tremendous progress on the engine itself.
NeoNietzschean NeoNietzschean 9/25/2014 02:32
@Badir: No, the speedup cited in the article is accurate.

What matters here is how quickly the program performs a given task on the phone relative to performing the same task on the desktop.

For chess, that difference is very, very large. For example, I've run multiple tests on my systems (phone is Samsung Mega 6.3, desktop uses an i5-3550), and it takes on average 41x as much time to reach the same depth on my phone as on my desktop.

I'm sure there are specific benchmarks where phone processors are much closer to desktop processors, but it doesn't matter for this article whether video rendering, flash processing, or compression only take 8-10 times longer on phone processors.

What matters is how much faster chess engines are, and the number cited in the article is consistent with what I've seen on my own hardware.

Just my $.02
Badir Badir 9/25/2014 02:22
First a modern cell phone cpu is not 50 times slower than an i7. You can run several test to compare and most put the speed difference from 8 to 10 times faster and NOT 50. They are getting closer in terms of speed every year also.

Secondly you can't force openings on the programs as they are optimized for their library they have. With an active library they would probably dump both programs into roughly equal 20 move midgames instead of what they were forced into.

Thirdly the only test of interest would be tournament style length game say something like 40 moves per 2hours.

The article seems to be about pushing software on people instead of giving any real information.
jhoravi jhoravi 9/25/2014 01:19
Komodo 8 in a Smartphone will destory IMB Deep Blue 2 more convincingly.
Bertman Bertman 9/24/2014 10:43
@ISeeThis Does that really surprise you? 30 x 120 MHz is... 3.6GHz, which is already slower than my desktop. Plus it was split up in 30 CPUs, which means a huge loss in efficiency. Between the hardware advances and the huge software advances, yes, Komodo on a smartphone would mop the room with 1997's Deep Blue. Besides, DB did not go nearly as deeply as you might think. The average depth of the middlegame was 12 plies or so.
MCris MCris 9/24/2014 10:25
I don't know if anybody saw all the games -there is a drop-down menu- but when Shredder lost with White it was put to play inferior openings like the English (1.c4 e5) and an Italian variation where one pawn is thrown away in the opening (!)
PurpleUnicorns PurpleUnicorns 9/24/2014 09:23
Perhaps an easy and interesting test would be to have both engines on the same hardware, but give handicap Komodo's time by factor 50 or something similar. This would make it easier to compare as well as remove the automate move transmission.
NeoNietzschean NeoNietzschean 9/24/2014 08:33
Yeah, I too have heard people claim that engines are only better now because of hardware. It's obviously false to anyone remotely familiar with engines, but it's nice to have a highly publicized test like this.

It is a shame that it couldn't be automated, as the 6 game limit is probably what people who want to dismiss the result will point to, since 6 games is such a small sample.

Of course the result would hold over a large number of games, but the doubters will continue to doubt until such a test is made public.

Also, the numbers for rating gain per doubling are not fixed at any value. The longer the time control/the faster the processors, the less gain per doubling.

With a given branching factor, a doubling in speed will buy you a fixed increase in average search depth per move, and the data bear out the intuition that buying an extra ply of depth at 10 ply is worth a lot more rating than buying an extra ply of depth at 30 ply.

Several tests at talkchess.com have indicated that at classical time controls, the gain could be as low as 30 points for a doubling in speed, even (but don't seem to ever flatten out much beyond that, interestingly.)
GalacticKing GalacticKing 9/24/2014 06:31
Nice work, Albert! I love running engine matches, so kudos for taking the time to do this. So far, Komodo 8 has been the only worthy challenger to Stockfish 5 on my android phone!
iSeeThis iSeeThis 9/24/2014 06:30
It should be noted that Deep Blue was one of the most powerful supercomputers in the world in 1997, running a massively parallel RISC System/6000 with 30 nodes each containing a 120 MHz Power2 Super Chip microprocessor. Someone estimated that Deep Blue was around 2800 then (well, that's enough to beat the then #1 chess player on this planet; Kasparov!) Roughly, Shredder10, also 2800, is supposed to beat Deep Blue 50-50. Guys, it means that an app on smartphone today can easily beat Deep Blue like father beats his child!
alfilferoz alfilferoz 9/24/2014 06:20
It would be interesting to repeat the experiment with Komodo 8 in the same i7 desktop computer against Komodo 8 smartphone.....
Bertman Bertman 9/24/2014 05:23
@thlai80 Most of your questions are answered in the article and video. Bear in mind that the generation of i7 used, the model of the smartphone, are all irrelevant, since what really matters ultimately is to know how much of a speed difference there was between the two during the test. That said, for the sake of disclosure, it was an Ivy Bridge i7, and the phone is an LG Optimus G Pro. Regarding NPS, the answer is there to see. Note that you cannot compare the NPS of two completely different engines.
thlai80 thlai80 9/24/2014 05:06
A lot of details were left out however. Which smartphone with what processor, and what is the depth when the move was forced for Komodo on the phone? Then, there's the setting of Shredder on desktop, the contempt value, the memory used, is it tournament book, etc. Finally what exactly is top line quad core i7 processor (1st gen, 2nd gen or 3rd gen) what is the amount of ram, not to mention OS (eg. windows version since it is Shredder from 2006) etc? In terms of calculation speed, what exactly is the position or nodes per sec for both setups? For pure computer strength comparison, I would take out opening book altogether!
MarriedRhombus MarriedRhombus 9/24/2014 04:13
These engines are the main tools of the cheaters.
Bertman Bertman 9/24/2014 03:55
@Deltamax2020 Actually, engines gain about 50-60 Elo per doubling in speed. This is easily verifiable in engine ratings lists that test single-core, dual core, and more, and has been the reality since the advent of multi-core engines.

@Gurcan The 50x comparison between Komodo on a smartphone and desktop is valid since the point is to show that Komodo is giving up a 50x speed edge. If it also ran on a desktop against Shredder its speedup would be.... 50x.
PurpleUnicorns PurpleUnicorns 9/24/2014 03:01
Opening book isn't really that important. Top engines usually don't make large mistakes with or without book anyhow.

I've read from credible sources that a doubling in CPU power is around 50-70 elo, however it is less when the increased power comes from an increased number of cores.

The improvement of engine software is ridiculously massive over the passed few years. Significantly larger then any CPU improvements in the same time-frame. Some people probably might think Rybka, Fritz or shredder are top engines nowadays, but those days are long passed. Their newest versions are only the #7, #14 and #20 respectively according to CCRL. Top engines today are Komodo, Stockfish and Houdini in that order. Those three engines will tear any older engine to pieces with their bare hands.

The problem is people have trouble imagining just how ridiculously powerful these engines are, which makes it even harder for people to comprehend that the engines are continuing in leaps and bounds. This to a large degree is probably due to the #2 engine Stockfish which is open source and can get downloaded for free after a trivial google search.

That being said, I believe many will prefer Komodo 8 style when analysing over Stockfishs. In any case I think it's completely legitimate to support these brilliant programmers.
Gurcan Uckardes Gurcan Uckardes 9/24/2014 12:50
Well done for better pushing Komodo sales. I'm not against that as a mobile chess freak. According to the article, it's fine but there's a detail i don't find accurate. This 50X ratio of power doesn't make sense unless the progs are the same. If Komodo mobile plays vs Komodo PC it's OK but here in this experiment, no! However it's good to see the proof that modile CPU's are not toys anymore and some of them are really challenging. Regarding the calc power vs ELO, exact versions today can reach ++50 to 100 ELO with double time but it does not apply to other engines directly. Another factor which breaks this rule is the depth vs time. Every engine has its own curve and is optimized for a range of hardware power. We can't assume Komodo would reach 3600 ELO just by using 64 cores (though it's not possible today). The ELO gain is less and less per extra depth beyond a threshold.
DELTAMAX2020 DELTAMAX2020 9/24/2014 11:14
ELO calculation is based on theaverage ELO of oppenents.For example if you have 100 diferent engines on the ELO list like 1st 2800 2nd 2700 ...etc it does not matter when you run the same test on a 1 millon times faster computers. ELO ratings in this case stays nearly same for all the engines. 1st engine 2800 2nd 2700 etc.But when you make comparison outside the same list you can see that on a one million times faster computers, power of the engines are skyrocketed compare to human ELO or "real" ELO.This is because of the ELO formula.
Werewolf Werewolf 9/24/2014 10:50
I ran a match recently between Stockfish 5 on the iPhone 5s and Deep Fritz 10.1 running on a 16 core workstation. The result was similar - Fritz lost almost every game.
DELTAMAX2020 DELTAMAX2020 9/24/2014 10:40
According to my own calculations when the CPU power doubles it has 100 elo plus for every engine.
DELTAMAX2020 DELTAMAX2020 9/24/2014 10:37
Openin book is also an important factor.
1