TCEC 11: Premier Division starts

Advertising

Books, boards, sets: Chess Niggemann

Stephan Oliver Platz Stephan is a passionate collector of chess books and for years he has been successfully playing as an amateur for his German club. The former musician and comedian works as a freelance journalist and author in Berlin and in the Franconian village Hiltpoltstein.

Discuss

Rules for reader comments

celeje 3/1/2018 02:11

Tord was just an example. Anyone saying formal things is not allowed to speculate. But then their polite words have zero effect.

I know what you think led to the 1GB hash.
I was just admitting that it's hard to come up with another explanation.
For other bad things the Alpha Zero team did, other explanations can be thought up, though they don't excuse the Alpha Zero team.

Speculation was that DeepMind used the totally retarded fixed time of 1 min/move because they couldn't set up Alpha Zero to do normal time controls.

And yeah, the website article you quoted is embarrassing. Esp. since the author is a computer scientist.

ChessPizzas 3/1/2018 08:12

But Tord has already said what was needed: the results are meaningless, and said so with adequate justification and good manners. No need for him to add speculation. No need to prove anything either. Tord merely adhered to available facts.

The 1 Gb hash decision seems pretty obvious to me. There is a clear purpose behind it which I believe I explained with some irony. It is also sad to see that AlphaZero team is not willing to take a real challenge. A 10 second per move time control looks like a blitz game so they should have used like 4Gb hash at least. However, they should have also used a standard time control, more like a 3+2 to make a blitz game. Even a bullet time control of 2m + 1s or 1+1 would have been better, though not ideal.

Of course with such a short time control you can reduce hash. However I would have given Stockfish 8Gb in hash and 32 threads and not 64. However, if you want to use as baseline a powerful but home equipment, then you could instead use a 16 core processor like the Ryzen Threadripper, give 16 threads to Stockfish, and provide as much hash as you can with a recommended amount of 8Gb (more if possible). Also give a dedicated SSD drive just for the endgame tablebases, completely separate of the operating system hard drive. You should use opening books if regular chess is going to be played. If instead of regular chess, Chess960 would be played, only then you can get rid of opening books.

In fact, in the last TCEC Season 10, in the blitz portion, 8Gb has were used, with 43 threads for the engines when possible. For the Superfinal, which had an standard long time control of 120m + 15s engines were given 16 Gb hash and were running on 43 threads.

Back on Season 6, engines were given less threads, 16 to be precise, because there were not so much cores available back then. Engines were given 16 Gb has back then as well. I don't know why aren´t they giving more hash space now compared to earlier seasons. Probably because some engines in the competition have other hash limitations.

celeje 3/1/2018 07:02

@ChessPizzas:

Yeah, but if you're e.g. Tord, you cannot say anything without proving it & knowing what happened, or else as you said before you'll be seen as the villain of the story.

I admit, it's hard to think up ANY reason for using only 1GB hash. If I'm a Deepmind guy who really knows nothing about anything, I'll assume that BIGGER IS BETTER. So then I'll give Stockfish as much hash as I can. $200 laptops now come with 4GB memory. So how can 1GB be the choice??

TCEC only gives 16GB hash. Should probably be 32GB or 64GB. Was it even less before?

ChessPizzas 2/28/2018 08:34

We don't need to prove it, and also we don't need to know exactly what happened. We instead can infer what happened. For me it is fairly safe to exclude the possibility that an incompetent guy actually came to configure stockfish, all without knowing how he did it.

Have they hired Homer Simpson?
https://www.youtube.com/watch?v=v5MOatu5-DE

Instead it is easier to infer that this was deliberate and not product of incompetence. It makes far more sense. You don't make science randomly. You don't make a publicity stunt randomly either.

On the writers making the articles on several websites, I can only say that some of them are easy to impress and fairly clueless. Just look at this quote:

------------------------
Stockfish used 64 threads, which suggests it was running on a very powerful PC, but only used a modest hash size of 1 GB. As against that, AlphaZero had 5,064 TPUs at its disposal, but only used four of them in the match.

Many people have proposed how to make a fair match between the two, but this is not really possible, as they rely on radically different hardware. A race between a person and a horse would not be made “fair” by only permitting the use of two legs.

(...)

If you’re still thinking about the size of Stockfish’s hash table, you’re really missing the point of what’s happened. Put it this way: AlphaZero’s achievement would have been only a shade less amazing had it instead lost to Stockfish by a similar score.

End quote.
Source: https://www.chess.com/article/view/whats-inside-alphazeros-brain
-----------------------

Look at the first paragraph in the quote. Seems like the author is trying to mislead the reader into the belief that even though Stockfish was using a very small hash, it was compensated by the fact that AlphaZero was not using 5,064 TPUs but only four. This of course is not providing any compensation at all. I find this disingenuous.

Then follow the second paragraph. He now makes use of a false analogy to support the point that a fair match between AlphaZero and Stockfish (or any other engine) is impossible. This is also false. A fair match is of course possible and there is no need for both to run on the same hardware or configuration as I mentioned in previous comments. Saying otherwise is like saying that previous encounters between humans and machines were impossible because they are so different.

-But machines are made of silicon and other metals while humans are made of protein! Besides humans have no circuits! An encounter between them is impossible!

I mean, seriously, this poor guy can't hide his bias. But now let's go to the third paragraph (fourth in the original). Do you see something wrong with it? Hint: not just a shade less amazing. In reality it would have gone to the other side of the spectrum. It would have been more like:

- Hey dude, you are facing Stockfish, what were you expecting? Oh! You really were expecting to win! In that case just tune Stockfish's parameters a little. ;-)

celeje 2/28/2018 01:56

@ ChessPizzas:

I cannot deny that DeepMind's stuff is very, very bad. Not trying to make excuses for them. Just saying it's very hard to prove that it was deliberate and not just incompetent.

I don't mean to blame medic/GM for everything. Could be any author of Alpha Zero paper at fault or all authors. Medic/GM just got attention from people because he used to play chess at a high level.
Yeah, the whole Alpha Zero team has to take the blame. But in reality a person cannot police everything everyone else on his team does. So we just don't know what happened.

Journalists/media are also partly to blame, including one or two who write on this website. They keep reporting stuff that's not true about Alpha Zero.

ChessPizzas 2/28/2018 04:24

Of course is possible as you put it, but still, what is its relevance? If this sloppy Medic/GM with a PhD in Whateverness made all the configuration, it is still Deepmind's team fault as a whole. You cannot publicly put the blame in this hypothetically incompetent guy. This lack of attention to the detail puts AlphaZero "achievement" under question.

The configuration of Stockfish or any other engine is not some obscure secret difficult to find out. Deepmind knows well of the existence of the TCEC, otherwise how could they know Stockfish was champion not so long ago?! They could actually see how the engines were configured there. Also recently a Computer Chess Championship was organized by another prominent chess website. They could also see and compare configurations there.

The only thing repeated here was misconfiguration. They did it not only with Stockfish, but also with Elmo, the Shogi engine. Elmo was not only misconfigured but it is also known that Elmo is not the strongest Shogi engine out there. Someone has to tell Deepmind to learn to read the bloody manuals. Take a look at the following link:

http://www.uuunuuun.com/single-post/2017/12/07/Some-concerns-on-the-matching-conditions-between-AlphaZero-and-Shogi-engine

The thing is if you are going to publish your work like a scientific breakthrough, then you need to put attention to detail. You cannot just start doing random stuff and expect to be excused because someone in your team didn't do his homework.

Let me quote a small fragment from an article written by José Camacho Collados that is right on spot:

"We should scientifically scrutinize alleged breakthroughs carefully, especially in the period of AI hype we live now. It is actually responsibility of researchers in this area to accurately describe and advertise our achievements, and try not to contribute to the growing (often self-interested) misinformation and mystification of the field. In fact, this early December in NIPS, arguably the most prestigious AI conference, some researchers showed important concerns about the lack of rigour of this scientific community in recent years [15]."

https://medium.com/@josecamachocollados/is-alphazero-really-a-scientific-breakthrough-in-ai-bf66ae1c84f2

Also from the comments in that article you can find this:
"2) Fairness of results vs. Stockfish: I would ask your question differently. Hypothetically, what would be the point of evaluating a system against another system which is not used at its best and under a questionable setting? Proper evaluation settings and solid baselines help providing better scientific insights and to draw more reliable conclusions."

celeje 2/27/2018 04:03

Yeah, absolutely. The importance of opening books is obvious. The important of endgame tablebases should be obvious too. Because engines go to depth 35, 40, so in the opening they are computing endgames.

ChessPizzas:"This is why I don't give Deepmind the benefit of the doubt. I don't care if one guy or two within the team do not know how to properly configure an engine. *That is no excuse*. We would be talking about and entire team or even a full organization who doesn't know how to do something relatively simple."

Yeah, we know 100% DeepMind stuffed it up big time. But we don't know 100% how it happened. What about this possibility?

Other Alpha Zero guys: "We're comp sci guys. Leave the programming to us."
Medic/GM: "Ok."
Other Alpha Zero guys: "You used to play chess. Why don't you set up the match with Stockfish?"
Medic/GM: "Ok"

And then medic makes huge mistakes with Stockfish.

ChessPizzas 2/25/2018 09:32

Absolutely. I was discussing the importance of opening books and endgame tablebase the other day with a friend. The engines are designed to work with them. For AlphaZero fans this is just another ugly issue that they would be happy to dismiss.

This is why I don't give Deepmind the benefit of the doubt. I don't care if one guy or two within the team do not know how to properly configure an engine. *That is no excuse*. We would be talking about and entire team or even a full organization who doesn't know how to do something relatively simple.

Configuring Stockfish is not rocket science, and even what they did with the whole AlphaZero research looks way more complex than rocket science. How come they could not configure Stockfish? 1 Gb hash? Really? My smartphone (being a modest one) has that much memory. Still an irrelevant issue AlphaZero fans/team?

celeje 2/24/2018 01:17

Had a look at the TCEC today. Didn't know before how important tablebases are. Can be 10000 tablebase hits at move 12 and 1 million tablebase hits at move 18.

Before I thought Deepmind not configuring Stockfish with an opening book was terrible but tablebases less important.
Now I think Deepmind not configuring Stockfish with tablebases was also terrible. It's a big deal even at move 12.

Alpha Zero fans say: "Tablebases don't matter because that's just at the end." Wrong, wrong, wrong.

celeje 2/22/2018 12:11

@ChessPizzas:

It's good for Tord to be polite. But trying too hard to be polite can be bad. Tord kinda said AZ's "match" is completely unfair as a sporting contest but it's okay as science. That insults science.

DeepMind is not Google but owned by it. Yeah, agree Google don't hire incompetent guys. Probably even more true for DeepMind. But they're very competent just at what they're hired for. Not at everything. The TCEC chat won't tell much unless there were DeepMind guys in it.

One of the DeepMind team guys is a GM and medical doctor - neurologist. Probably does not even know how to turn on and off MRI machines. A radiographer does MRIs for him. A radiologist reads MRIs for him. So even in his field he only knows some things. So maybe should not expect him to know how to download and set up Stockfish software. We don't know.

ChessPizzas 2/20/2018 05:59

Yeah, and he should keep it polite. He is being sensible and moderate. If he were not to be, he would be seen as the villain of the story.

For me, the attitude from the Deepmind team shows lack of sportsmanship. I have nothing to admire from this. I actually believe it was unethical and completely unnecesary for selling their own business. They actually managed to defeat the best players in Go and totally deserve recognition for that. But later when they published the paper on the Stockfish match, claiming "superiority" over it, while seems impressive at first glance, I find it unacceptable from a sports perpective given Stockfish configuration and also from a scientific perspective. They are not using a good baseline for their tests. They might as well could have claimed that they could defeat PlayMagnus App instead of Stockfish.

I also think that you are being a bit generous with them when you say that this could be a sign of incompetence from those guys on Deepmind. Trust me, to work at Google, you can be everything but incompetent. They actually know *way much* better. And I say this not just by myself. I remember this from a conversation in the TCEC chat where this was pointed out.

Now some may argue after all I said, that I am angry because AlphaZero "won" and I am not accepting it. Here is the thing. If AlphaZero had actually won a *real* match, I would not be angry at all (and yes, I am a Stockfish fan). I probably would be excited and happy to see a new milestone in chess. And a say it sincerely because *I initially was*. It was later when I discovered the conditions of the "match" (which was rather an exhibition) that I changed my admiration to disappointment.

celeje 2/20/2018 07:01

Yeah, you're right, DeepMind declining to answer questions because it's in review is a bad look. It looks like they're running and hiding. They shouldn't do that. Esp. when they're happy to market AZ. They can't have it both ways. Not good.

Tord sounded like he was trying too hard to be polite. He could have criticized DeepMind much more than that. It sounded like he didn't want to seem like he was whining.

ChessPizzas 2/19/2018 06:28

Wow, so this is actually funny. I said tha I was a bit sarcastic in an effort to make things clear but that only served to do things more confusing.

Well, AlphaZero is not a public project so it is hard to say what misconfiguring would mean for them. Probably putting AlphaZero on suboptimal hardware could actually be considered misconfiguring. By suboptimal I mean not running in TPUs for AlphaZero.

It would actually be interesting, not the match itself, but the protests that could possibly come from AlphaZero fans. Suddenly AlphaZero would probably start losing game after game after game against every opponent because it would be running with its performance diminished by running on suboptimal hardware.

Then they will start to say: Oh, it is not fair, it should run on TPUs, not in normal hardware, that is why it is losing.
Then someone else would say: so what? it is running on equal conditions.
- But it was misconfigured!
- Let's call it even then. ;-)

Those guys on the AlphaZero camp usually try to dimiss configuration thing as if it was irrelevant.
They say: If you are talking about the hash size, you are missing the point.
I say, quite the contrary, we are pointing a critical detail, and an ugly one which they don't want to see. Simple and pure denial or ignorance.

Of course this hypothetical scenario is not going to happen because AlphaZero is not going to be executed in TCEC hardware. As I said before, it is technically unfeasible and also highly undesirable. Seriously, what would be the point?

However, even ignoring the technical imposibility of running AlphaZero on TCEC hardware, you should not expect a match like it to happen, simply because the Deepmind team is not interested. They just wanted to make a publicity stunt and that's it. To put it very clear, when the Deepmind team was questioned about the configuration issues of Stockfish (and also Elmo, let's remind ourselves), and the comments of criticism that came from the community, particularly the comments from Tord Romstadt, one of the Stockfish creators, correctly stating that under those circumstances those apparently astonishing results are actually meaningless, they declined to comment by saying that AlphaZero research was not complete. That actually tells a lot. Of course Romstadt was only one of many who pointed out this issue.

You should expect, if such a match ever happens (which, I insist, is very unlikey), to have AlphaZero running on TPU architecture and the rest of the engines in a normal but powerful hardware like the TCEC one. Then they should relay the moves to each other.

So actually a match is possible (again, extremely unlikely to happen) and requires, if it is going to be part of TCEC, a change in the tournament format.

celeje 2/18/2018 04:44

@ChessPizzas:

I actually misunderstood you. Let me explain. My first reading of your first comment was closer to what you did mean. Your second comment, "I was being sarcastic", made me re-read your first comment. I then thought your sarcastic point was "It doesn't matter that DeepMind gave Stockfish small hash, no opening book, etc. It's okay if they were sloppy. Everyone who points those things out is making a fuss about nothing." But your latest comment shows that's not what you meant by "being sarcastic" at all.

So now I know your point was:
"What I was saying from the beginning is that I don't want to see AlphaZero opponents deliberately crippled or blinded by the sloppy configuration that the Deepmind team used for Stockfish and Elmo in their "matches"."

But you also don't want to see AZ not on TPUs because "it is not feasible unless you also want to see it severely diminished in its performance (just like the AlphaZero team misconfigured Stockfish to diminish its strength)."

Okay, I get you now. Thanks for clearing that up.

But there's a difference between weaker hardware and "sloppy configuration Deepmind used".
Putting AZ on the best normal hardware TCEC or normal people can get is not misconfiguring. AZ will perform worse just because normal hardware has much, much less computing power, not because AZ has the wrong settings. If you care about that, AZ can be given more time to make up.

Deepmind gave Stockfish 64 threads, so probably 32 cores, so probably a PC. Good one but still just regular single PC. That's weak, but maybe they say that's the best they had available for Stockfish, though it's much much less powerful than TPUs for AZ.

Misconfiguring Stockfish with little hash, no book, etc. is different. It's not a lack of computer power, but more like incompetence, like what the pointy haired boss from Dilbert would do.

ChessPizzas 2/17/2018 06:30

Perhaps I should have added the "practically" adverb to my statement to make it entirely true to your satisfaction so let me rephrase it: AlphaZero and traditional engines cannot *practically* run on the same hardware.

Now don't tell me it is untrue now. You said it yourself, on traditional hardware it would run badly.

"You are entitled not to advocate or to advocate whatever you want. Doesn't mean others aren't also entitled to advocate whatever they want, which might be something different".

I am not saying otherwise.

"Some people might like to see what moves AZ makes on traditional hardware."

Like whom and for what purpose? At the time of writing this it is not feasible unless you also want to see it severely diminished in its performance (just like the AlphaZero team misconfigured Stockfish to diminish its strength). I want to see AlphaZero working at its best and that is not going to happen on traditional hardware right now. AlphaZero should run in its TPUs just like it was designed to work with.

What I was saying from the beginning is that I don't want to see AlphaZero opponents deliberately crippled or blinded by the sloppy configuration that the Deepmind team used for Stockfish and Elmo in their "matches".

celeje 2/17/2018 05:37

@ChessPizzas:

I know your first comment was sarcastic because your second comment said so. Your second comment had an untrue statement in it. You can make your point without making untrue statements.

You are entitled not to advocate or to advocate whatever you want. Doesn't mean others aren't also entitled to advocate whatever they want, which might be something different. Some people might like to see what moves AZ makes on traditional hardware.

ChessPizzas 2/17/2018 05:05

Again, it is not the point. Wether it can run or not, or if it does badly when run on normal hardware is not the point I was making. Also I am not advocating that AlphaZero should be run on traditional hardware just like the other engines.

celeje 2/17/2018 06:46

@ChessPizzas:
Okay, it's hard to tell with these comments.
But that's not correct. AlphaZero could be run on normal hardware. It'd just do really badly.

ChessPizzas 2/17/2018 06:31

celeje, I was being sarcastic. AlphaZero and traditional engines cannot run on the same hardware but that is not the point.

celeje 2/17/2018 05:21

@ChessPizzas:
If it's played on exactly the same normal hardware, AZ would probably need opponents to have more like 1 second per move to AZ's 1 minute per move.
10s per move for opponents would probably be too much, based on AZ's hardware superiority.

ChessPizzas 2/17/2018 03:53

AlphaZero will only accept matches if their opponents reduce their hashes to 1 Gb, and also get rid of endgame tablebases and play ten seconds per move. That way it can show it's "superiority" to the world!

celeje 2/16/2018 02:59

I assume @ChessHulk's question is rhetorical & saying that on equal normal hardware Alpha Zero would do really badly. That's true. But it would be interesting to see if Alpha Zero can play okay moves on normal hardware even if it's beaten in every game.

ChessHulk 2/15/2018 02:28

I assume it's been mentioned somewhere, but where is Alpha Zero? This competition would seem to be the *real* place to show the AI self-learning capabilities?

News

Komodo Dragon 2

Komodo Dragon 2 has both excellent positional play - thanks to the neural network - and tremendous tactical power and computing speed, thanks to the classic Alpha Beta engine.

€99.90

Surprising Your Opponents with the Lazy Sicilian

In just 60 minutes, you will be well-equipped with a surprise weapon, which will take most of your opponents out of book right from the start.

€9.90

ChessBase Magazine Extra 203

Videos: Jan Werle takes a close look at Carlsen's 4.c4 in the Caro-Kann Advance Variation. Robert Ris shows a powerful attacking idea against the Taimanov Variation. "Lucky bag" with analyses by Edouard, Krasenkow, Kapnisis, Shah and Srinath.

€12.99

Master Class Vol.8 - Magnus Carlsen 2nd Edition

Let our authors show you how Carlsen tailored his openings to be able to outplay his opponents strategically in the middlegame or to obtain an enduring advantage into the endgame.

€29.90

Calculation Training in Attack & Defence Vol.1

The two DVDs offer you the chance to solve 66 exercises with multiple questions. These exercises are presented in the interactive format, which makes them accessible for players of different strengths as we will go through the thought process step by step

€29.90

Calculation Training in Attack & Defence Vol.2

€29.90

Calculation Training in Attack & Defence Vol.1 & Vol.2

€54.90

SHOP

SHOP

TCEC 11: Premier Division starts

ONLINE SHOP

A Complete Black Repertoire versus the English, 1...e5

Newcomer "Andscacs" scores first

TCEC 11 adopts a league format

Results so far

TCEC Season 11, Division 4

TCEC Season 11, Division 3

TCEC Season 11, Division 2

TCEC Season 11, Division 1

Tension guaranteed in the "Premier Division"

Links

Discuss

Komodo Dragon 2

Surprising Your Opponents with the Lazy Sicilian

ChessBase Magazine Extra 203

Master Class Vol.8 - Magnus Carlsen 2nd Edition

Calculation Training in Attack & Defence Vol.1

Calculation Training in Attack & Defence Vol.2

Calculation Training in Attack & Defence Vol.1 & Vol.2

Pop-up for detailed settings