ICGA/Rybka controversy: An interview with David Levy (1)

2/6/2012 – He is a star chess programmer with an unbroken record of supremacy in computer chess during the past five years. In June 2011 the International Computer Games Association found Vasik Rajlich guilty of "plagiarism" in early program versions and banned him for life. We published a vigorous defence of Rajlich and a rebuttal by ICGA President David Levy, who promised us an indepth interview.

ChessBase 14 Download ChessBase 14 Download

Everyone uses ChessBase, from the World Champion to the amateur next door. Start your personal success story with ChessBase 14 and enjoy your chess even more!


Along with the ChessBase 14 program you can access the Live Database of 8 million games, and receive three months of free ChesssBase Account Premium membership and all of our online apps! Have a look today!

More...

Introduction

In June 2011 it was widely reported in the global media that the International Computer Games Association (ICGA), headed by Dr. David Levy, had found chess programmer Vasik Rajlich in breach of the ICGA’s annual World Computer Chess Championship (WCCC) tournament rule related to program originality. In the ICGA’s accompanying report it was asserted that Rajlich’s chess program Rybka contained “plagiarized” code from Fruit, a program authored by Fabien Letouzey of France.

Rajlich, who cruised to victory in four consecutive WCCC tournaments in 2007, 2008, 2009 and 2010, was retroactively stripped of all titles he had won in competition and was slapped with a lifetime ban. He claimed complete innocence and made it clear that he found the ICGA’s investigatory process and conclusions to be biased and unprofessional, and the charges baseless and unworthy. Rajlich refused to be drawn into a protracted dispute with his accusers or mount a comprehensive defence. This was done by Dr Søren Riis, a computer scientist from London, in a paper we published in four parts. There was vigorous feedback from the chess community, including a rebuttal by the ICGA. Links to the relevant articles are given at the bottom of this page.

In his rebuttal David Levy agreed to do a full one-on-one exclusive interview with ChessBase on the entire ICGA/Rybka scandal. It was conducted by sequential emails, and took a fair amount of time, since David was consulting with experts who live in different time zones around the globe.


Interview with David Levy on the ICGA/Rybka scandal

ChessBase: David, do you believe that the ICGA has conclusively proved that Vasik Rajlich copied actual code (as opposed to ideas and algorithmic techniques) from the Fruit program?

David Levy: At the outset I wish to make it clear that this is not an issue about verbatim copying. We have proven that Rybka is a derivative, but due to different internal board representations the Fruit code required modification when being copied. But a conversion such as mailbox to bitboard still leaves it a derivative, as would changes such as refactoring the code and tuning the numbers.


IM David Levy, the President of the ICGA

The forums continually mention code copying instead of Rajlich failing to report a derivative in his ICGA entry submissions. It is important to remember that the offence for which Rajlich was convicted and banned by the ICGA was failing to comply with our Tournament Rule 2. The various reports (here and here) show that Rybka 1.0 beta and Rybka 2.3.2a were clearly derivatives of Fruit 2.1. Postings on the forums and Riis stating that merely retyping or converting a program to another format are clearly wrong. Merely retyping and/or converting code from one format to another is still making a derivative, as too are some changes that affect game play (such as evaluation tuning). So yes, the Rybka entries were derivative of the Fruit code in a manner that broke Rule 2. And Rajilch says that "Rybka is original at the level of source code" meaning "the author either typed his own code or typed the code which generated his own code", going so far as to say that Rybka derivatives which he had previously censured are "original at the source code level" if they were typed by hand. If this were the case then, if instead of cutting and pasting a programmer just re-types by hand what he has stolen, then according to Rajlich it is original and does not violate ICGA or copyright rules! Really?

In which versions of his program did this occur?

Rybka versions from 1.0 Beta through 2.3.2a were shown to be clear derivatives of Fruit 2.1.

Did any of these versions (for which there is definite proof that they contain derivatives Fruit code) play in an ICCA or ICGA event?

Yes, Rybka 2.3.2a definitely competed in ICGA events. The evaluation of Rybka 2.3.2a clearly was derived from Fruit 2.1. Note that the report states "Rybka 2.3.2 was released Jun 13, 2007, and 2.3.2a about five days later. Rybka competed in and won the ICGA Chess Tournament (Jun 11-18 2007)". ICGA analysis of several versions of Rybka between 1.0 beta and 2.3.2a show they also contained a derivative of the Fruit 2.1 evaluation function. During this period a version of Rybka 2.0 competed in the 2006 ICGA World Computer Chess Championship tournament (May 2006) under the name "Rajlich". The closest commercial release was the MP-only Rybka 2.0 Beta 4 with a release date of May 17 (Beta 4).

During the investigation process one member of the ICGA investigation secretariat asked Rajlich "1. Are you able to provide us with the exact .exes that played in all ICGA events particularly those from 2006 and 2007? If not, is it a fair approximation to use Rybka 2.3.2a for the 2007 event?" On the Rybka Forum, Rajlich said "Yes" to the second question, confirming that this version was used in an ICGA event.

Were any sold commercially?

Yes, see above. Rybka 2.0 Beta 4 and Rybka 2.3.2a were sold commercially.

What do the ICGA experts think about the current versions of Rybka (i.e. versions 3 and 4)? Do they also contain code taken from Fruit?

The ICGA has not done extensive analysis of newer versions of Rybka. It was sufficient for the ICGA to determine that Rajlich had broken Tournament Rule 2, which we did. When the ICGA chose to ban Rybka for life due to violating the rules in ICGA events in 2006 and 2007, this required all of Rajlich's programs be removed, since the newer programs would never have been allowed to compete if the violation of earlier versions was discovered sooner and the lifetime ban issued earlier. So while it is possible that newer versions of Rybka also contained Fruit-derived code, it was not essential to prove this once we had proved the case for the 2006 and 2007 ICGA tournaments.

If Rybka was, as you put it, a derivative of Fruit 2.1, how do you explain that it was massively stronger? Normally by tuning an existing program, even your own, you gain 30 point per year. We know that from all our engine programmers. 150 points is unheard of.

The panel did not do much examination as to what made Rybka stronger, so I would suggest you ask Rajlich directly. But experiments with Strelka, which Vas has said is a clone/derivative of Rybka, show gains due to bitboard speedup, 64 bit code being more efficient than 32 bit code, the addition of a form of futility using verification by a qsearch, and tuning the Fruit evaluation. Harder to test is effect of the the Rybka material imbalance tables, but they could have caused some gain as well.

If you compare the 32 bit version of Rybka 1.0 (Elo 2885) to Fruit 2.1 (Elo 2793), the Elo difference is only 92. The speedup going to 64 bit code (which benefits bitboard programs more than mailbox) is about 34 Elo. In versions of Rybka higher than 1.0, there is speculation that more strength advancement came through additional evaluation tuning, and search changes.

The use of Fruit gave Rajlich a huge head start, as he only had to improve a 2793 Elo program instead of writing one from scratch. To get some idea of what small changes to a strong program can do, consider what an amateur programmer did with Fruit in creating Toga II (an admitted Fruit derivative). He was able to add 135 Elo in strength to the Fruit code base, bringing it to an Elo of 2928. The ratings we quote are from are here.

It would appear that Rajlich was punished not for taking code from Fruit 2.1 verbatim, or "deriving" his program from it, but for failing to comply with our Tournament Rule 2. If for argument's sake we assume that Rybka 1.0 Beta through 2.3.2a were clear derivatives of Fruit 2.1, what exactly would he have had to do to comply with Tournament Rule 2?

As we have intimated on previous occasions he should have revealed, when submitting his entries, that Rybka was a derivative of Fruit, in order to allow the ICGA to decide whether it should be allowed entry.

What is the ICCA definition of "derivative"? If a programmer takes six lines of code from one of the many open source programs (Crafty, Fruit, Strelka, Stockfish, etc.) must this be specified in the entry form to comply with Rule 2? What if he takes twelve lines? 24? 200? Or can it be given in percentage of the entire playing code? Where is this defined in the ICCA rules or charter?

One doesn't normally define things so precisely, for the same reason that the United States Constitution is open for interpretation in various places. If you paint yourself into a corner, you are stuck there, but reasonable people can come to a reasonable decision about "when is it too much to allow." And this is really an issue of functionality, rather than counting characters or lines. For example, if you copy a pawn evaluation procedure, that is copying too much. If you find a line of code that does something in a clever way you did not think of yourself, that probably would not even be noticed.

Your question implies that Rajlich might only have copied a small and insignificant amount of Fruit, while the conclusion of the ICGA investigation was that a lot of code was copied. When considering the extent of copying no one on the panel thought that what Rajlich had done was "borderline" in a quantitative sense. The panel actually discussed how much evidence and which of the evidence points would be necessary and sufficient to demonstrate Rajlich's guilt, and looked closely at evaluation feature overlap because this was seen as the most notable and unarguable from the standpoint of Rule 2 (on program originality).

The essence of the case against Rajlich, as we repeatedly explain, is that when entering the World Computer Chess Championship a programming team is obliged to reveal, in their entry submission, if they have used any derived code. The ICGA can then assess, possibly after questioning the programming team and/or after examining the entrant's source code, whether or not the entry should be accepted. That will depend on the nature of what was copied, permissions received from the author of the original program, and whether another program in the same family has applied to enter the tournament.

I would also like to stress, as a separate issue, that a programmer should always comply with the licence of the engine from which he copies the code, as well as declaring it on the ICGA entry form.

One should also mention here that, even if one were to ignore his copying of Crafty code, the most serious allegations of infringements by Rajlich do not start with his 2006 WCCC entry but with his December 2005 Rybka release and its alleged copyright infringement of Fruit. That release is not, in itself, part of the ICGA's grievance against Rajlich, since copyright infringement does not become an ICGA matter unless and until the infringing software is submitted for entry into the WCCC.

Is it fair to say that Vasik Rajlich ruined his stellar career and his livelihood in computer chess, and got himself condemned in the world press (including CNN, Wired) as a liar, cheat, thief, plagiarist, etc. because he failed to fill out one line in the ICGA entry form correctly? [Note that filling out that line correctly would have legitimised his participation and prevented the situation that we currently have]

That "one line" would not necessarily have solved the problem. Letouzey would also have had to give his permission as well, in order to be listed as an author. If Rajlich had declared Rybka's Fruit origins on the entry form and complied with the Fruit GPL he might well have been allowed to enter with Letouzey's permission. In that case only one Fruit-based engine would have been allowed – the one which had Letouzey's permission and his name in the entry submission.

For whatever has happened in Rajlich's life due to the ICGA's investigation and verdict, he has only himself to blame. If people do not follow rules or laws then they themselves are responsible for the consequences.

Let us now consider what might have followed if Rajlich's entry form had been filled out correctly in the first place. His entry would certainly have been rejected unless he had also obtained Letouzey's permission and put Letouzey's name in his entry submission. What might have happened then? Perhaps Rajlich would have become frustrated trying to write his own evaluation function, in which case he might have approached Letouzey and joined forces with him in some way, or he might have given up trying to create a world beating program and someone else might have become the world beater for the next four years (either starting from Fruit code, with Letouzey's permission, or otherwise). Or Rajlich might have decided after all to develop his own evaluation function, in which case who knows how strong or weak Rybka might have turned out to be?

Out of curiosity: how many other programs have listed anything or even filled out that line in the entry form in the past?

Some programmers who enter the WCCC use the Nalimov endgame tablebases and declare as such in their entry submissions. Some programmers use openings books developed by others and similarly declare this in their entry submissions. I do not recall any entry submission that admitted to copying an evaluation function as Rajlich did (particularly in his early versions).

How many members were there in the panel that investigated Vasik Rajlich? How many voted? Who were the members who voted and how did they vote?

At the time of voting the panel was made up as follows: There were three members of the investigation Secretariat; four members of the ICGA board (who did not vote prior to the completion of the report), and the WCCC Tournament Director (who also did not vote prior to the completion of the report), and 34 members of the panel who were invited to vote if they wished. Here is exactly what we asked the panel members under the topic of "Opinion".

"Panel members are kindly invited to review the Rybka-Fruit Controversy pages and render opinions. Please answer these questions:

(a) Have you read all of the evidence linked to the Rybka-Fruit Controversy pages?

(b) In your opinion, did Vasik Rajlich's chess program Rybka violate the ICGA Tournament Rules, especially rule 2:

"2. Each program must be the original work of the entering developers. Programming teams whose code is derived from or including game-playing code written by others must name all other authors, or the source of such code, in the details of their submission form. Programs which are discovered to be close derivatives of others (e.g., by playing nearly all moves the same), may be declared invalid by the Tournament Director after seeking expert advice. For this purpose a listing of all game-related code running on the system must be available on demand to the Tournament Director."

Opinions should be presented within two weeks so that the Summary can be written"

So each panel member who responded to the invitation to vote answered two questions. Question (a) was to determine if they had read all the important evidence, so we could judge if the panel member was qualified to answer (b). Question (b) was to get the opinions of panel members as to whether or not Rajlich violated Rule 2. It was not really handled like a jury trial, and the final determination of guilt was up to the ICGA Executive Committee.

Sixteen panel members voted, all of whom answered "yes" to both questions. The ICGA would not divulge the names of those who voted (other than Ken Thompson) without first asking their permission and we do not intend to do so.

Why must the names of the panel members (except one) be kept a secret? Who were the three members of the "Investigation Secretariat", and did they lead the investigation?

We decided during the investigation process on confidentiality with respect to the names of those who voted. The Secretariat comprised: Mark Lefler, Robert Hyatt and Harvey Williamson. They administered the investigation and they wrote the report. Incidentally, I must re-iterate that the report was written after the panel members voted, whereas Søren Riis incorrectly states on Chessbase.com that "It really goes without saying that the panel members voted on the basis of the findings of the ICGA report." He shouldn't have said that, given that it is false.

How many votes were required to pass a verdict? Was this defined anywhere before the investigation was undertaken?

The purpose of the voting by the panel was to assess the weight of opinion of those panel members who wanted to make their opinions known, so that the secretariat could take that into account when preparing their report on the investigation and so that the ICGA office bearers could take it into account when deciding on their verdict. After the ICGA office bearers studied the report and the evidence submitted as part of the report, we voted on whether to find Rajlich innocent or guilty of breaking the ICGA's Tournament Rule 2. There was unanimous agreement on a guilty decision.

After the guilty vote how was the punishment determined? Where was the degree of punishment or the procedure to decide on it defined?

The punishment was determined by the Executive Committee of the ICGA (excluding the Vice-President for Asia): David Levy, Yngvi Bjornsson, Hiroyuki Iida, Rémi Coulom and Jaap van den Herik, who was the Tournament Director of all the tournaments under discussion. From the outset there was no doubt about disqualification and cancellation of World Champion titles, or on the fact that a ban from participating in future ICGA events was called for. The only matter on which there was any debate was the duration of the ban. Some ICGA office bearers felt that ten years was the appropriate period, while others favoured a life ban. By a majority decision the ICGA decided to make it a life ban.

As to the procedure to decide the degree of punishment – this comes under Article III Section 1 of the ICGA Constitution, which is always published in the ICGA Journal prior to the tri-ennial meeting at which ICGA office bearers are elected. Section 1 states that the Executive Committee of the ICGA is charged with the administrative affairs of the association.

When the investigation process against Rajlich was initiated and when the guilty verdict plus sentence was passed, the ICGA issued press releases and distributed them freely. The banning of Rajlich for life was reported in chess blogs and in the international media (including the New York Times and CNN). We cannot understand why the voting members of the jury who found Rajlich guilty need to be kept a strict secret (except for one, who is proudly listed). This creates the impression that some of them may have had a vested interest in the outcome of the investigation. How many panel members that voted were rival programmers who had been beaten by Rybka in computer chess tournaments? How many voters received World Championship titles "posthumously" when Rybka was stripped of its ICGA titles?

We decided at the start of the process that we would not reveal the names of those who voted, so we did not know at that time who would vote and which way they would vote.

The suggestion that some of those who voted may have had a vested interest in a "guilty" verdict overlooks one very important and indisputable fact. The world of elite chess programmers is a very small one, and these are the people who are best equipped to judge the issues involved in the investigation. So Rajlich's actions were being assessed by his peers. If the ICGA had excluded from the panel anyone who is an elite chess programmer we would have been significantly diminishing the overall quality of the investigation. Remember also that I, as President of the ICGA, and the other Board members who I have listed above, were able to see not only all the evidence and all the votes of the panel, but also we were able to follow the discussion on the closed Wiki that was set up for the benefit of panel members. If any of us had seen any comments that led us to believe that any unfair bias existed within the panel's deliberations, we would have had the right and the duty to say so at the time.

Finally, on this point, I would like to add that I have always enjoyed a courteous relationship with Vasik Raljich, and until the evidence leading to this investigation was presented to me I had absolutely nothing against him. In fact, when I was first told the rumour (as it then was) that he had used code derived from Fruit, I did not believe it, largely because I do not regard the person who told me as a credible source. Furthermore, if I had had a choice of the verdict (in the absence of the evidence) I would very much have preferred it to have been a "Not guilty" one, as the investigation and its aftermath could do absolutely no good for the image of computer chess. The only good that has come out of the whole affair so far as I am concerned, is that programmers who are tempted in the future to break the ICGA's tournament rules in a serious way will now be less likely to do so.

– Part two will follow shortly –
Please do not write in commenting on this interview until part two has been published!


Previous articles on the ICGA/Rybka scandal

Feedback on the ICGA/Rybka disqualification scandal
13.01.2012 – It's a long read, but many of the letters we received in reaction to the defence of the Rybka program by Dr Søren Riis are quite passionate and well thought out in their content. We start with the summary of a long rebuttal of the Riis paper that was sent to us by the ICGA and circulated on the Internet – with links to the full version and ancillary documents. Take a deep breath.
A Gross Miscarriage of Justice in Computer Chess (part four)
05.01.2012 – As proof that Vas Rajlich had copied program code the ICGA presented pages and pages of Fruit and Rybka code side by side. But, according to Dr Søren Riis, what was labeled as Rybka code was actually fabricated to look like Fruit. He criticizes the ICGA's process failures and ruminates on the reasons behind the unprecedented vendetta launched against the star chess programmer.
A Gross Miscarriage of Justice in Computer Chess (part three)
04.01.2012 – A core accusation against Vas Rajlich is that Rybka and Fruit have very similar positional evaluations, and the use of floating point numbers in Rybka’s time management code had to be copied from Fruit. Søren Riis enumerates the ten substantive evaluation differences and shows how the second accusation boils down to a single misplaced keystroke with zero impact on Rybka's play.
A Gross Miscarriage of Justice in Computer Chess (part two)
03.01.2012 – In this part Dr Søren Riis of Queen Mary University in London shows how most programs (legally) profited from Fruit, and subsequently much more so from the (illegally) reverse engineered Rybka. Yet it is Vasik Rajlich who was investigated, found guilty of plagiarism, banned for life, stripped of his titles, and vilified in the international press – for a five-year-old alleged tournament rule violation. Ironic.
A Gross Miscarriage of Justice in Computer Chess (part one)
02.01.2012 – "Biggest Sporting Scandal since Ben Johnson" and "Czech Mate, Mr. Cheat" – these were headlines in newspapers around the world six months ago. The International Computer Games Association had disqualified star programmer Vasik Rajlich for plagiarism, retroactively stripped him of all titles, and banned him for life. Søren Riis, a computer scientist from London, has investigated the scandal.

Discussion and Feedback Join the public discussion or submit your feedback to the editors


Discuss

Rules for reader comments

 
 

Not registered yet? Register