TCEC 11: Premier Division starts
2/14/2018 – The "Top Chess Engine Championship", TCEC for short, is considered a kind of unofficial computer world championship. Currently the 11th season is played, now in the form of a league with promotion and relegation. After the preliminaries, the "Premier League" started on Tuesday with the eight top programs. S.O. Platz gives an overview of the history of this unofficial chess computer world championship.
Fritz 16 - He just wants to play!
Fritz 16 is looking forward to playing with you, and you're certain to have a great deal of fun with him too. Tense games and even well-fought victories await you with "Easy play" and "Assisted analysis" modes.
Newcomer "Andscacs" scores first
The eleventh season at TCEC has been running since Janurary 3rd, 2018, but on Tuesday, February 13th, 19:00 CET, the "Premier Division" kicked off with the eight best chess programs. The two first place finishers will contest a Super Final — a 100 game match like last season — later this year. In 2017, Houdini beat Komodo 53 : 47 in December.
TCEC 11 adopts a league format
For the first time, the competition of the world's best chess programs takes the form of a league system, with both promotion and relegation. The participating programs were divided into five groups: "Division" 1 to 4 and a "Premier Division". Games are played with powerful hardware (2 x Intel Xeon E5 2699 v4 @ 2.8 GHz, 44 CPU) — each program may use up to 43 processors — that means computer chess at the highest level.
Results so far
The time control for Division 4 and Division 3 was 30 minutes plus 10 seconds per move for the game.
TCEC Season 11, Division 4
- Defenchess 10 / 14
- Senpai 9 / 14
- Pedone 8 / 14
- Ethereal 7½ / 14
- ChessBrainVB 7½ / 14
- Toga 7 / 14
- The Baron 4 / 14
- Scorpio 3 / 14
In addition to the winner Defenchess, Senpai [which means "mentor" in Japanese -Ed.], Pedone, Ethereal and ChessBrain VB also qualified for the next higher league (Division 3).
TCEC Season 11, Division 3
- Fritz 16 18½ / 28
- Laser 17½
- Nemorino 14½
- Pedone 14
- Defenchess 14
- Senpai 12½
- ChessBrainVB 11
- Ethereal 10
Fritz 16 celebrated a successful debut in TCEC. The new program version of star programmer Vasik Rajlich won 10 games, lost only one and played 17 draws. This meant first in the final ranking with 18½ out of 28 in front of Laser (17½) and Nemorino (14½). Thus, Fritz 16 and Laser rose to the next higher league, while Ethereal and ChessBrainVB have to compete again in Division 4 next time.
Fritz 16 - He just wants to play!
Fritz 16 is looking forward to playing with you, and you're certain to have a great deal of fun with him too. Tense games and even well-fought victories await you with "Easy play" and "Assisted analysis" modes.
TCEC Season 11, Division 2
The time control in Division 2 was 45 minutes plus 10 seconds per move for the game.
- Jonny 20 / 28
- Laser 17½
- Texel 16½
- Arasan 14
- Fritz 16 14
- Vajolet 13
- Bobcat 10
- Wasp 7
The Jonny program by the German developer Johannes Zwanzger won clearly (+12, -0, =16). In addition, Laser made the leap to the next higher league, while Bobcat and Wasp descended. Fritz 16 defended his place in Division 2 in shared 4th/5th place.
TCEC Season 11, Division 1
In this division programs had 60 minutes plus 10 seconds per move, and played 56 games.
- Andscacs 37 / 56
- Fizbo 31½
- Booot 31
- Jonny 30
- Gull 26½
- Laser 24½
- Hannibal 23
- Nirvana 20½
The leap into the "Premier Divison" was achieved by the top program Andscacs, developed by Daniel José Queraltó from Andorra, and Fizbo by Youri Matiounine (USA). Hannibal and Nirvana were relegated.
Tension guaranteed in the "Premier Division"
Houdini 6 continues where its predecessor left off, and adds solid 60 Elo points to this formidable engine, once again making Houdini the strongest chess program currently available on the market.
The participants in the "Premier Division" include several of the usual suspects: Houdini, Komodo, Stockfish, but also Fire, Ginkgo, Chiron, Andscacs and Fizbo.
The time control is a full 90 minutes plus 10 seconds per move for the game.
This competition is likely to be of particular interest, as Stockfish 9, the latest version of this successful open source engine was released earlier this month. Last year, Stockfish missed out on the Super Final, but this time it could be different, because the programmers have especially improved the engine's "Contempt" attitude. This ensures that draws are avoided against weaker programs, even when the position is slightly worse. We look forward to seeing what this change will do in practice.
In addition to Stockfish, the other favorites are of course last year's winner Houdini and last year's finalist Komodo. The computer chess fans can look forward to an exciting tournament.
You can watch the games live at TCEC.chessdom.com. In the first round, there was one decisive game which saw the newcomers Andscacs and Fizbo face off.
The multiple computer chess world champion comes in a new and yet more powerful version. Thanks to co-author US Grandmaster Larry Kaufman, Komodo is the strategist among the top chess programs!
Links
Discussion and Feedback
Join the public discussion or submit your feedback to the editors
Tord was just an example. Anyone saying formal things is not allowed to speculate. But then their polite words have zero effect.
I know what you think led to the 1GB hash.
I was just admitting that it's hard to come up with another explanation.
For other bad things the Alpha Zero team did, other explanations can be thought up, though they don't excuse the Alpha Zero team.
Speculation was that DeepMind used the totally retarded fixed time of 1 min/move because they couldn't set up Alpha Zero to do normal time controls.
And yeah, the website article you quoted is embarrassing. Esp. since the author is a computer scientist.
I know what you think led to the 1GB hash.
I was just admitting that it's hard to come up with another explanation.
For other bad things the Alpha Zero team did, other explanations can be thought up, though they don't excuse the Alpha Zero team.
Speculation was that DeepMind used the totally retarded fixed time of 1 min/move because they couldn't set up Alpha Zero to do normal time controls.
And yeah, the website article you quoted is embarrassing. Esp. since the author is a computer scientist.
But Tord has already said what was needed: the results are meaningless, and said so with adequate justification and good manners. No need for him to add speculation. No need to prove anything either. Tord merely adhered to available facts.
The 1 Gb hash decision seems pretty obvious to me. There is a clear purpose behind it which I believe I explained with some irony. It is also sad to see that AlphaZero team is not willing to take a real challenge. A 10 second per move time control looks like a blitz game so they should have used like 4Gb hash at least. However, they should have also used a standard time control, more like a 3+2 to make a blitz game. Even a bullet time control of 2m + 1s or 1+1 would have been better, though not ideal.
Of course with such a short time control you can reduce hash. However I would have given Stockfish 8Gb in hash and 32 threads and not 64. However, if you want to use as baseline a powerful but home equipment, then you could instead use a 16 core processor like the Ryzen Threadripper, give 16 threads to Stockfish, and provide as much hash as you can with a recommended amount of 8Gb (more if possible). Also give a dedicated SSD drive just for the endgame tablebases, completely separate of the operating system hard drive. You should use opening books if regular chess is going to be played. If instead of regular chess, Chess960 would be played, only then you can get rid of opening books.
In fact, in the last TCEC Season 10, in the blitz portion, 8Gb has were used, with 43 threads for the engines when possible. For the Superfinal, which had an standard long time control of 120m + 15s engines were given 16 Gb hash and were running on 43 threads.
Back on Season 6, engines were given less threads, 16 to be precise, because there were not so much cores available back then. Engines were given 16 Gb has back then as well. I don't know why aren´t they giving more hash space now compared to earlier seasons. Probably because some engines in the competition have other hash limitations.
The 1 Gb hash decision seems pretty obvious to me. There is a clear purpose behind it which I believe I explained with some irony. It is also sad to see that AlphaZero team is not willing to take a real challenge. A 10 second per move time control looks like a blitz game so they should have used like 4Gb hash at least. However, they should have also used a standard time control, more like a 3+2 to make a blitz game. Even a bullet time control of 2m + 1s or 1+1 would have been better, though not ideal.
Of course with such a short time control you can reduce hash. However I would have given Stockfish 8Gb in hash and 32 threads and not 64. However, if you want to use as baseline a powerful but home equipment, then you could instead use a 16 core processor like the Ryzen Threadripper, give 16 threads to Stockfish, and provide as much hash as you can with a recommended amount of 8Gb (more if possible). Also give a dedicated SSD drive just for the endgame tablebases, completely separate of the operating system hard drive. You should use opening books if regular chess is going to be played. If instead of regular chess, Chess960 would be played, only then you can get rid of opening books.
In fact, in the last TCEC Season 10, in the blitz portion, 8Gb has were used, with 43 threads for the engines when possible. For the Superfinal, which had an standard long time control of 120m + 15s engines were given 16 Gb hash and were running on 43 threads.
Back on Season 6, engines were given less threads, 16 to be precise, because there were not so much cores available back then. Engines were given 16 Gb has back then as well. I don't know why aren´t they giving more hash space now compared to earlier seasons. Probably because some engines in the competition have other hash limitations.
@ChessPizzas:
Yeah, but if you're e.g. Tord, you cannot say anything without proving it & knowing what happened, or else as you said before you'll be seen as the villain of the story.
I admit, it's hard to think up ANY reason for using only 1GB hash. If I'm a Deepmind guy who really knows nothing about anything, I'll assume that BIGGER IS BETTER. So then I'll give Stockfish as much hash as I can. $200 laptops now come with 4GB memory. So how can 1GB be the choice??
TCEC only gives 16GB hash. Should probably be 32GB or 64GB. Was it even less before?
Yeah, but if you're e.g. Tord, you cannot say anything without proving it & knowing what happened, or else as you said before you'll be seen as the villain of the story.
I admit, it's hard to think up ANY reason for using only 1GB hash. If I'm a Deepmind guy who really knows nothing about anything, I'll assume that BIGGER IS BETTER. So then I'll give Stockfish as much hash as I can. $200 laptops now come with 4GB memory. So how can 1GB be the choice??
TCEC only gives 16GB hash. Should probably be 32GB or 64GB. Was it even less before?
We don't need to prove it, and also we don't need to know exactly what happened. We instead can infer what happened. For me it is fairly safe to exclude the possibility that an incompetent guy actually came to configure stockfish, all without knowing how he did it.
Have they hired Homer Simpson?
https://www.youtube.com/watch?v=v5MOatu5-DE
Instead it is easier to infer that this was deliberate and not product of incompetence. It makes far more sense. You don't make science randomly. You don't make a publicity stunt randomly either.
On the writers making the articles on several websites, I can only say that some of them are easy to impress and fairly clueless. Just look at this quote:
------------------------
Stockfish used 64 threads, which suggests it was running on a very powerful PC, but only used a modest hash size of 1 GB. As against that, AlphaZero had 5,064 TPUs at its disposal, but only used four of them in the match.
Many people have proposed how to make a fair match between the two, but this is not really possible, as they rely on radically different hardware. A race between a person and a horse would not be made “fair” by only permitting the use of two legs.
(...)
If you’re still thinking about the size of Stockfish’s hash table, you’re really missing the point of what’s happened. Put it this way: AlphaZero’s achievement would have been only a shade less amazing had it instead lost to Stockfish by a similar score.
End quote.
Source: https://www.chess.com/article/view/whats-inside-alphazeros-brain
-----------------------
Look at the first paragraph in the quote. Seems like the author is trying to mislead the reader into the belief that even though Stockfish was using a very small hash, it was compensated by the fact that AlphaZero was not using 5,064 TPUs but only four. This of course is not providing any compensation at all. I find this disingenuous.
Then follow the second paragraph. He now makes use of a false analogy to support the point that a fair match between AlphaZero and Stockfish (or any other engine) is impossible. This is also false. A fair match is of course possible and there is no need for both to run on the same hardware or configuration as I mentioned in previous comments. Saying otherwise is like saying that previous encounters between humans and machines were impossible because they are so different.
-But machines are made of silicon and other metals while humans are made of protein! Besides humans have no circuits! An encounter between them is impossible!
I mean, seriously, this poor guy can't hide his bias. But now let's go to the third paragraph (fourth in the original). Do you see something wrong with it? Hint: not just a shade less amazing. In reality it would have gone to the other side of the spectrum. It would have been more like:
- Hey dude, you are facing Stockfish, what were you expecting? Oh! You really were expecting to win! In that case just tune Stockfish's parameters a little. ;-)
Have they hired Homer Simpson?
https://www.youtube.com/watch?v=v5MOatu5-DE
Instead it is easier to infer that this was deliberate and not product of incompetence. It makes far more sense. You don't make science randomly. You don't make a publicity stunt randomly either.
On the writers making the articles on several websites, I can only say that some of them are easy to impress and fairly clueless. Just look at this quote:
------------------------
Stockfish used 64 threads, which suggests it was running on a very powerful PC, but only used a modest hash size of 1 GB. As against that, AlphaZero had 5,064 TPUs at its disposal, but only used four of them in the match.
Many people have proposed how to make a fair match between the two, but this is not really possible, as they rely on radically different hardware. A race between a person and a horse would not be made “fair” by only permitting the use of two legs.
(...)
If you’re still thinking about the size of Stockfish’s hash table, you’re really missing the point of what’s happened. Put it this way: AlphaZero’s achievement would have been only a shade less amazing had it instead lost to Stockfish by a similar score.
End quote.
Source: https://www.chess.com/article/view/whats-inside-alphazeros-brain
-----------------------
Look at the first paragraph in the quote. Seems like the author is trying to mislead the reader into the belief that even though Stockfish was using a very small hash, it was compensated by the fact that AlphaZero was not using 5,064 TPUs but only four. This of course is not providing any compensation at all. I find this disingenuous.
Then follow the second paragraph. He now makes use of a false analogy to support the point that a fair match between AlphaZero and Stockfish (or any other engine) is impossible. This is also false. A fair match is of course possible and there is no need for both to run on the same hardware or configuration as I mentioned in previous comments. Saying otherwise is like saying that previous encounters between humans and machines were impossible because they are so different.
-But machines are made of silicon and other metals while humans are made of protein! Besides humans have no circuits! An encounter between them is impossible!
I mean, seriously, this poor guy can't hide his bias. But now let's go to the third paragraph (fourth in the original). Do you see something wrong with it? Hint: not just a shade less amazing. In reality it would have gone to the other side of the spectrum. It would have been more like:
- Hey dude, you are facing Stockfish, what were you expecting? Oh! You really were expecting to win! In that case just tune Stockfish's parameters a little. ;-)
@ ChessPizzas:
I cannot deny that DeepMind's stuff is very, very bad. Not trying to make excuses for them. Just saying it's very hard to prove that it was deliberate and not just incompetent.
I don't mean to blame medic/GM for everything. Could be any author of Alpha Zero paper at fault or all authors. Medic/GM just got attention from people because he used to play chess at a high level.
Yeah, the whole Alpha Zero team has to take the blame. But in reality a person cannot police everything everyone else on his team does. So we just don't know what happened.
Journalists/media are also partly to blame, including one or two who write on this website. They keep reporting stuff that's not true about Alpha Zero.
I cannot deny that DeepMind's stuff is very, very bad. Not trying to make excuses for them. Just saying it's very hard to prove that it was deliberate and not just incompetent.
I don't mean to blame medic/GM for everything. Could be any author of Alpha Zero paper at fault or all authors. Medic/GM just got attention from people because he used to play chess at a high level.
Yeah, the whole Alpha Zero team has to take the blame. But in reality a person cannot police everything everyone else on his team does. So we just don't know what happened.
Journalists/media are also partly to blame, including one or two who write on this website. They keep reporting stuff that's not true about Alpha Zero.
Of course is possible as you put it, but still, what is its relevance? If this sloppy Medic/GM with a PhD in Whateverness made all the configuration, it is still Deepmind's team fault as a whole. You cannot publicly put the blame in this hypothetically incompetent guy. This lack of attention to the detail puts AlphaZero "achievement" under question.
The configuration of Stockfish or any other engine is not some obscure secret difficult to find out. Deepmind knows well of the existence of the TCEC, otherwise how could they know Stockfish was champion not so long ago?! They could actually see how the engines were configured there. Also recently a Computer Chess Championship was organized by another prominent chess website. They could also see and compare configurations there.
The only thing repeated here was misconfiguration. They did it not only with Stockfish, but also with Elmo, the Shogi engine. Elmo was not only misconfigured but it is also known that Elmo is not the strongest Shogi engine out there. Someone has to tell Deepmind to learn to read the bloody manuals. Take a look at the following link:
http://www.uuunuuun.com/single-post/2017/12/07/Some-concerns-on-the-matching-conditions-between-AlphaZero-and-Shogi-engine
The thing is if you are going to publish your work like a scientific breakthrough, then you need to put attention to detail. You cannot just start doing random stuff and expect to be excused because someone in your team didn't do his homework.
Let me quote a small fragment from an article written by José Camacho Collados that is right on spot:
"We should scientifically scrutinize alleged breakthroughs carefully, especially in the period of AI hype we live now. It is actually responsibility of researchers in this area to accurately describe and advertise our achievements, and try not to contribute to the growing (often self-interested) misinformation and mystification of the field. In fact, this early December in NIPS, arguably the most prestigious AI conference, some researchers showed important concerns about the lack of rigour of this scientific community in recent years [15]."
https://medium.com/@josecamachocollados/is-alphazero-really-a-scientific-breakthrough-in-ai-bf66ae1c84f2
Also from the comments in that article you can find this:
"2) Fairness of results vs. Stockfish: I would ask your question differently. Hypothetically, what would be the point of evaluating a system against another system which is not used at its best and under a questionable setting? Proper evaluation settings and solid baselines help providing better scientific insights and to draw more reliable conclusions."
The configuration of Stockfish or any other engine is not some obscure secret difficult to find out. Deepmind knows well of the existence of the TCEC, otherwise how could they know Stockfish was champion not so long ago?! They could actually see how the engines were configured there. Also recently a Computer Chess Championship was organized by another prominent chess website. They could also see and compare configurations there.
The only thing repeated here was misconfiguration. They did it not only with Stockfish, but also with Elmo, the Shogi engine. Elmo was not only misconfigured but it is also known that Elmo is not the strongest Shogi engine out there. Someone has to tell Deepmind to learn to read the bloody manuals. Take a look at the following link:
http://www.uuunuuun.com/single-post/2017/12/07/Some-concerns-on-the-matching-conditions-between-AlphaZero-and-Shogi-engine
The thing is if you are going to publish your work like a scientific breakthrough, then you need to put attention to detail. You cannot just start doing random stuff and expect to be excused because someone in your team didn't do his homework.
Let me quote a small fragment from an article written by José Camacho Collados that is right on spot:
"We should scientifically scrutinize alleged breakthroughs carefully, especially in the period of AI hype we live now. It is actually responsibility of researchers in this area to accurately describe and advertise our achievements, and try not to contribute to the growing (often self-interested) misinformation and mystification of the field. In fact, this early December in NIPS, arguably the most prestigious AI conference, some researchers showed important concerns about the lack of rigour of this scientific community in recent years [15]."
https://medium.com/@josecamachocollados/is-alphazero-really-a-scientific-breakthrough-in-ai-bf66ae1c84f2
Also from the comments in that article you can find this:
"2) Fairness of results vs. Stockfish: I would ask your question differently. Hypothetically, what would be the point of evaluating a system against another system which is not used at its best and under a questionable setting? Proper evaluation settings and solid baselines help providing better scientific insights and to draw more reliable conclusions."
Yeah, absolutely. The importance of opening books is obvious. The important of endgame tablebases should be obvious too. Because engines go to depth 35, 40, so in the opening they are computing endgames.
ChessPizzas:"This is why I don't give Deepmind the benefit of the doubt. I don't care if one guy or two within the team do not know how to properly configure an engine. *That is no excuse*. We would be talking about and entire team or even a full organization who doesn't know how to do something relatively simple."
Yeah, we know 100% DeepMind stuffed it up big time. But we don't know 100% how it happened. What about this possibility?
Other Alpha Zero guys: "We're comp sci guys. Leave the programming to us."
Medic/GM: "Ok."
Other Alpha Zero guys: "You used to play chess. Why don't you set up the match with Stockfish?"
Medic/GM: "Ok"
And then medic makes huge mistakes with Stockfish.
ChessPizzas:"This is why I don't give Deepmind the benefit of the doubt. I don't care if one guy or two within the team do not know how to properly configure an engine. *That is no excuse*. We would be talking about and entire team or even a full organization who doesn't know how to do something relatively simple."
Yeah, we know 100% DeepMind stuffed it up big time. But we don't know 100% how it happened. What about this possibility?
Other Alpha Zero guys: "We're comp sci guys. Leave the programming to us."
Medic/GM: "Ok."
Other Alpha Zero guys: "You used to play chess. Why don't you set up the match with Stockfish?"
Medic/GM: "Ok"
And then medic makes huge mistakes with Stockfish.
Absolutely. I was discussing the importance of opening books and endgame tablebase the other day with a friend. The engines are designed to work with them. For AlphaZero fans this is just another ugly issue that they would be happy to dismiss.
This is why I don't give Deepmind the benefit of the doubt. I don't care if one guy or two within the team do not know how to properly configure an engine. *That is no excuse*. We would be talking about and entire team or even a full organization who doesn't know how to do something relatively simple.
Configuring Stockfish is not rocket science, and even what they did with the whole AlphaZero research looks way more complex than rocket science. How come they could not configure Stockfish? 1 Gb hash? Really? My smartphone (being a modest one) has that much memory. Still an irrelevant issue AlphaZero fans/team?
This is why I don't give Deepmind the benefit of the doubt. I don't care if one guy or two within the team do not know how to properly configure an engine. *That is no excuse*. We would be talking about and entire team or even a full organization who doesn't know how to do something relatively simple.
Configuring Stockfish is not rocket science, and even what they did with the whole AlphaZero research looks way more complex than rocket science. How come they could not configure Stockfish? 1 Gb hash? Really? My smartphone (being a modest one) has that much memory. Still an irrelevant issue AlphaZero fans/team?
Had a look at the TCEC today. Didn't know before how important tablebases are. Can be 10000 tablebase hits at move 12 and 1 million tablebase hits at move 18.
Before I thought Deepmind not configuring Stockfish with an opening book was terrible but tablebases less important.
Now I think Deepmind not configuring Stockfish with tablebases was also terrible. It's a big deal even at move 12.
Alpha Zero fans say: "Tablebases don't matter because that's just at the end." Wrong, wrong, wrong.
Before I thought Deepmind not configuring Stockfish with an opening book was terrible but tablebases less important.
Now I think Deepmind not configuring Stockfish with tablebases was also terrible. It's a big deal even at move 12.
Alpha Zero fans say: "Tablebases don't matter because that's just at the end." Wrong, wrong, wrong.
@ChessPizzas:
It's good for Tord to be polite. But trying too hard to be polite can be bad. Tord kinda said AZ's "match" is completely unfair as a sporting contest but it's okay as science. That insults science.
DeepMind is not Google but owned by it. Yeah, agree Google don't hire incompetent guys. Probably even more true for DeepMind. But they're very competent just at what they're hired for. Not at everything. The TCEC chat won't tell much unless there were DeepMind guys in it.
One of the DeepMind team guys is a GM and medical doctor - neurologist. Probably does not even know how to turn on and off MRI machines. A radiographer does MRIs for him. A radiologist reads MRIs for him. So even in his field he only knows some things. So maybe should not expect him to know how to download and set up Stockfish software. We don't know.
It's good for Tord to be polite. But trying too hard to be polite can be bad. Tord kinda said AZ's "match" is completely unfair as a sporting contest but it's okay as science. That insults science.
DeepMind is not Google but owned by it. Yeah, agree Google don't hire incompetent guys. Probably even more true for DeepMind. But they're very competent just at what they're hired for. Not at everything. The TCEC chat won't tell much unless there were DeepMind guys in it.
One of the DeepMind team guys is a GM and medical doctor - neurologist. Probably does not even know how to turn on and off MRI machines. A radiographer does MRIs for him. A radiologist reads MRIs for him. So even in his field he only knows some things. So maybe should not expect him to know how to download and set up Stockfish software. We don't know.
Yeah, and he should keep it polite. He is being sensible and moderate. If he were not to be, he would be seen as the villain of the story.
For me, the attitude from the Deepmind team shows lack of sportsmanship. I have nothing to admire from this. I actually believe it was unethical and completely unnecesary for selling their own business. They actually managed to defeat the best players in Go and totally deserve recognition for that. But later when they published the paper on the Stockfish match, claiming "superiority" over it, while seems impressive at first glance, I find it unacceptable from a sports perpective given Stockfish configuration and also from a scientific perspective. They are not using a good baseline for their tests. They might as well could have claimed that they could defeat PlayMagnus App instead of Stockfish.
I also think that you are being a bit generous with them when you say that this could be a sign of incompetence from those guys on Deepmind. Trust me, to work at Google, you can be everything but incompetent. They actually know *way much* better. And I say this not just by myself. I remember this from a conversation in the TCEC chat where this was pointed out.
Now some may argue after all I said, that I am angry because AlphaZero "won" and I am not accepting it. Here is the thing. If AlphaZero had actually won a *real* match, I would not be angry at all (and yes, I am a Stockfish fan). I probably would be excited and happy to see a new milestone in chess. And a say it sincerely because *I initially was*. It was later when I discovered the conditions of the "match" (which was rather an exhibition) that I changed my admiration to disappointment.
For me, the attitude from the Deepmind team shows lack of sportsmanship. I have nothing to admire from this. I actually believe it was unethical and completely unnecesary for selling their own business. They actually managed to defeat the best players in Go and totally deserve recognition for that. But later when they published the paper on the Stockfish match, claiming "superiority" over it, while seems impressive at first glance, I find it unacceptable from a sports perpective given Stockfish configuration and also from a scientific perspective. They are not using a good baseline for their tests. They might as well could have claimed that they could defeat PlayMagnus App instead of Stockfish.
I also think that you are being a bit generous with them when you say that this could be a sign of incompetence from those guys on Deepmind. Trust me, to work at Google, you can be everything but incompetent. They actually know *way much* better. And I say this not just by myself. I remember this from a conversation in the TCEC chat where this was pointed out.
Now some may argue after all I said, that I am angry because AlphaZero "won" and I am not accepting it. Here is the thing. If AlphaZero had actually won a *real* match, I would not be angry at all (and yes, I am a Stockfish fan). I probably would be excited and happy to see a new milestone in chess. And a say it sincerely because *I initially was*. It was later when I discovered the conditions of the "match" (which was rather an exhibition) that I changed my admiration to disappointment.
Yeah, you're right, DeepMind declining to answer questions because it's in review is a bad look. It looks like they're running and hiding. They shouldn't do that. Esp. when they're happy to market AZ. They can't have it both ways. Not good.
Tord sounded like he was trying too hard to be polite. He could have criticized DeepMind much more than that. It sounded like he didn't want to seem like he was whining.
Tord sounded like he was trying too hard to be polite. He could have criticized DeepMind much more than that. It sounded like he didn't want to seem like he was whining.
Wow, so this is actually funny. I said tha I was a bit sarcastic in an effort to make things clear but that only served to do things more confusing.
Well, AlphaZero is not a public project so it is hard to say what misconfiguring would mean for them. Probably putting AlphaZero on suboptimal hardware could actually be considered misconfiguring. By suboptimal I mean not running in TPUs for AlphaZero.
It would actually be interesting, not the match itself, but the protests that could possibly come from AlphaZero fans. Suddenly AlphaZero would probably start losing game after game after game against every opponent because it would be running with its performance diminished by running on suboptimal hardware.
Then they will start to say: Oh, it is not fair, it should run on TPUs, not in normal hardware, that is why it is losing.
Then someone else would say: so what? it is running on equal conditions.
- But it was misconfigured!
- Let's call it even then. ;-)
Those guys on the AlphaZero camp usually try to dimiss configuration thing as if it was irrelevant.
They say: If you are talking about the hash size, you are missing the point.
I say, quite the contrary, we are pointing a critical detail, and an ugly one which they don't want to see. Simple and pure denial or ignorance.
Of course this hypothetical scenario is not going to happen because AlphaZero is not going to be executed in TCEC hardware. As I said before, it is technically unfeasible and also highly undesirable. Seriously, what would be the point?
However, even ignoring the technical imposibility of running AlphaZero on TCEC hardware, you should not expect a match like it to happen, simply because the Deepmind team is not interested. They just wanted to make a publicity stunt and that's it. To put it very clear, when the Deepmind team was questioned about the configuration issues of Stockfish (and also Elmo, let's remind ourselves), and the comments of criticism that came from the community, particularly the comments from Tord Romstadt, one of the Stockfish creators, correctly stating that under those circumstances those apparently astonishing results are actually meaningless, they declined to comment by saying that AlphaZero research was not complete. That actually tells a lot. Of course Romstadt was only one of many who pointed out this issue.
You should expect, if such a match ever happens (which, I insist, is very unlikey), to have AlphaZero running on TPU architecture and the rest of the engines in a normal but powerful hardware like the TCEC one. Then they should relay the moves to each other.
So actually a match is possible (again, extremely unlikely to happen) and requires, if it is going to be part of TCEC, a change in the tournament format.
Well, AlphaZero is not a public project so it is hard to say what misconfiguring would mean for them. Probably putting AlphaZero on suboptimal hardware could actually be considered misconfiguring. By suboptimal I mean not running in TPUs for AlphaZero.
It would actually be interesting, not the match itself, but the protests that could possibly come from AlphaZero fans. Suddenly AlphaZero would probably start losing game after game after game against every opponent because it would be running with its performance diminished by running on suboptimal hardware.
Then they will start to say: Oh, it is not fair, it should run on TPUs, not in normal hardware, that is why it is losing.
Then someone else would say: so what? it is running on equal conditions.
- But it was misconfigured!
- Let's call it even then. ;-)
Those guys on the AlphaZero camp usually try to dimiss configuration thing as if it was irrelevant.
They say: If you are talking about the hash size, you are missing the point.
I say, quite the contrary, we are pointing a critical detail, and an ugly one which they don't want to see. Simple and pure denial or ignorance.
Of course this hypothetical scenario is not going to happen because AlphaZero is not going to be executed in TCEC hardware. As I said before, it is technically unfeasible and also highly undesirable. Seriously, what would be the point?
However, even ignoring the technical imposibility of running AlphaZero on TCEC hardware, you should not expect a match like it to happen, simply because the Deepmind team is not interested. They just wanted to make a publicity stunt and that's it. To put it very clear, when the Deepmind team was questioned about the configuration issues of Stockfish (and also Elmo, let's remind ourselves), and the comments of criticism that came from the community, particularly the comments from Tord Romstadt, one of the Stockfish creators, correctly stating that under those circumstances those apparently astonishing results are actually meaningless, they declined to comment by saying that AlphaZero research was not complete. That actually tells a lot. Of course Romstadt was only one of many who pointed out this issue.
You should expect, if such a match ever happens (which, I insist, is very unlikey), to have AlphaZero running on TPU architecture and the rest of the engines in a normal but powerful hardware like the TCEC one. Then they should relay the moves to each other.
So actually a match is possible (again, extremely unlikely to happen) and requires, if it is going to be part of TCEC, a change in the tournament format.
@ChessPizzas:
I actually misunderstood you. Let me explain. My first reading of your first comment was closer to what you did mean. Your second comment, "I was being sarcastic", made me re-read your first comment. I then thought your sarcastic point was "It doesn't matter that DeepMind gave Stockfish small hash, no opening book, etc. It's okay if they were sloppy. Everyone who points those things out is making a fuss about nothing." But your latest comment shows that's not what you meant by "being sarcastic" at all.
So now I know your point was:
"What I was saying from the beginning is that I don't want to see AlphaZero opponents deliberately crippled or blinded by the sloppy configuration that the Deepmind team used for Stockfish and Elmo in their "matches"."
But you also don't want to see AZ not on TPUs because "it is not feasible unless you also want to see it severely diminished in its performance (just like the AlphaZero team misconfigured Stockfish to diminish its strength)."
Okay, I get you now. Thanks for clearing that up.
But there's a difference between weaker hardware and "sloppy configuration Deepmind used".
Putting AZ on the best normal hardware TCEC or normal people can get is not misconfiguring. AZ will perform worse just because normal hardware has much, much less computing power, not because AZ has the wrong settings. If you care about that, AZ can be given more time to make up.
Deepmind gave Stockfish 64 threads, so probably 32 cores, so probably a PC. Good one but still just regular single PC. That's weak, but maybe they say that's the best they had available for Stockfish, though it's much much less powerful than TPUs for AZ.
Misconfiguring Stockfish with little hash, no book, etc. is different. It's not a lack of computer power, but more like incompetence, like what the pointy haired boss from Dilbert would do.
I actually misunderstood you. Let me explain. My first reading of your first comment was closer to what you did mean. Your second comment, "I was being sarcastic", made me re-read your first comment. I then thought your sarcastic point was "It doesn't matter that DeepMind gave Stockfish small hash, no opening book, etc. It's okay if they were sloppy. Everyone who points those things out is making a fuss about nothing." But your latest comment shows that's not what you meant by "being sarcastic" at all.
So now I know your point was:
"What I was saying from the beginning is that I don't want to see AlphaZero opponents deliberately crippled or blinded by the sloppy configuration that the Deepmind team used for Stockfish and Elmo in their "matches"."
But you also don't want to see AZ not on TPUs because "it is not feasible unless you also want to see it severely diminished in its performance (just like the AlphaZero team misconfigured Stockfish to diminish its strength)."
Okay, I get you now. Thanks for clearing that up.
But there's a difference between weaker hardware and "sloppy configuration Deepmind used".
Putting AZ on the best normal hardware TCEC or normal people can get is not misconfiguring. AZ will perform worse just because normal hardware has much, much less computing power, not because AZ has the wrong settings. If you care about that, AZ can be given more time to make up.
Deepmind gave Stockfish 64 threads, so probably 32 cores, so probably a PC. Good one but still just regular single PC. That's weak, but maybe they say that's the best they had available for Stockfish, though it's much much less powerful than TPUs for AZ.
Misconfiguring Stockfish with little hash, no book, etc. is different. It's not a lack of computer power, but more like incompetence, like what the pointy haired boss from Dilbert would do.
Perhaps I should have added the "practically" adverb to my statement to make it entirely true to your satisfaction so let me rephrase it: AlphaZero and traditional engines cannot *practically* run on the same hardware.
Now don't tell me it is untrue now. You said it yourself, on traditional hardware it would run badly.
"You are entitled not to advocate or to advocate whatever you want. Doesn't mean others aren't also entitled to advocate whatever they want, which might be something different".
I am not saying otherwise.
"Some people might like to see what moves AZ makes on traditional hardware."
Like whom and for what purpose? At the time of writing this it is not feasible unless you also want to see it severely diminished in its performance (just like the AlphaZero team misconfigured Stockfish to diminish its strength). I want to see AlphaZero working at its best and that is not going to happen on traditional hardware right now. AlphaZero should run in its TPUs just like it was designed to work with.
What I was saying from the beginning is that I don't want to see AlphaZero opponents deliberately crippled or blinded by the sloppy configuration that the Deepmind team used for Stockfish and Elmo in their "matches".
Now don't tell me it is untrue now. You said it yourself, on traditional hardware it would run badly.
"You are entitled not to advocate or to advocate whatever you want. Doesn't mean others aren't also entitled to advocate whatever they want, which might be something different".
I am not saying otherwise.
"Some people might like to see what moves AZ makes on traditional hardware."
Like whom and for what purpose? At the time of writing this it is not feasible unless you also want to see it severely diminished in its performance (just like the AlphaZero team misconfigured Stockfish to diminish its strength). I want to see AlphaZero working at its best and that is not going to happen on traditional hardware right now. AlphaZero should run in its TPUs just like it was designed to work with.
What I was saying from the beginning is that I don't want to see AlphaZero opponents deliberately crippled or blinded by the sloppy configuration that the Deepmind team used for Stockfish and Elmo in their "matches".
@ChessPizzas:
I know your first comment was sarcastic because your second comment said so. Your second comment had an untrue statement in it. You can make your point without making untrue statements.
You are entitled not to advocate or to advocate whatever you want. Doesn't mean others aren't also entitled to advocate whatever they want, which might be something different. Some people might like to see what moves AZ makes on traditional hardware.
I know your first comment was sarcastic because your second comment said so. Your second comment had an untrue statement in it. You can make your point without making untrue statements.
You are entitled not to advocate or to advocate whatever you want. Doesn't mean others aren't also entitled to advocate whatever they want, which might be something different. Some people might like to see what moves AZ makes on traditional hardware.
Again, it is not the point. Wether it can run or not, or if it does badly when run on normal hardware is not the point I was making. Also I am not advocating that AlphaZero should be run on traditional hardware just like the other engines.
@ChessPizzas:
Okay, it's hard to tell with these comments.
But that's not correct. AlphaZero could be run on normal hardware. It'd just do really badly.
Okay, it's hard to tell with these comments.
But that's not correct. AlphaZero could be run on normal hardware. It'd just do really badly.
celeje, I was being sarcastic. AlphaZero and traditional engines cannot run on the same hardware but that is not the point.
@ChessPizzas:
If it's played on exactly the same normal hardware, AZ would probably need opponents to have more like 1 second per move to AZ's 1 minute per move.
10s per move for opponents would probably be too much, based on AZ's hardware superiority.
If it's played on exactly the same normal hardware, AZ would probably need opponents to have more like 1 second per move to AZ's 1 minute per move.
10s per move for opponents would probably be too much, based on AZ's hardware superiority.
AlphaZero will only accept matches if their opponents reduce their hashes to 1 Gb, and also get rid of endgame tablebases and play ten seconds per move. That way it can show it's "superiority" to the world!
I assume @ChessHulk's question is rhetorical & saying that on equal normal hardware Alpha Zero would do really badly. That's true. But it would be interesting to see if Alpha Zero can play okay moves on normal hardware even if it's beaten in every game.
I assume it's been mentioned somewhere, but where is Alpha Zero? This competition would seem to be the *real* place to show the AI self-learning capabilities?
1
