Winning starts with what you know
The new version 18 offers completely new possibilities for chess training and analysis: playing style analysis, search for strategic themes, access to 6 billion Lichess games, player preparation by matching Lichess games, download Chess.com games with built-in API, built-in cloud engine and much more.
It has been a while since I have written exclusively about Chesthetica, a computer program I started developing at the beginning of my doctoral studies back in 2006. It was designed initially to evaluate aesthetics in chess and I later enhanced it to actually compose chess problems as well. This was achieved using a variety of improving methods and not really using the aesthetics model in the way one might expect. Just as an art critic may be good at critiquing, that particular knowledge or ability does not lend itself easily to actually creating good or interesting art. Perhaps this is partly why beauty contest judges are seldom predominantly beauty queens themselves. Put simply, one may recognize aesthetics and even be able to judge or evaluate it, but being able to create it is quite another matter.
In the last installment, I explained how Chesthetica was able to compose longer mates from originally being able to compose only three-movers. I had intended to update ChessBase readers about Chesthetica composing study-like constructs as well, which has been taking place since late 2015 but several other article ideas came up that got in the way. So as an official update to readers, Chesthetica is indeed able to compose study-like constructs now. It uses the same DSNS approach I have written about before which does not rely on endgame ‘tablebases’ or traditional artificial intelligence (AI) approaches that focus on positions with relatively few pieces. It does not even try to ‘improve upon’ or modify existing chess problems. It composes essentially from scratch and to this day, I am not able to explain why it works. I can only tell you how it works, as I have done earlier on ChessBase and more thoroughly in the DSNS book linked to above.
As I have explained before, the DSNS approach serves as the ‘spark’ of creativity in the machine, and the rest are basically parameters of requirements, e.g. three-mover, four-mover, study. This is why I believe it can be applied in virtually any domain and can almost certainly play a part in the composition of any type of chess problem as well. So the study-like constructs composed are basically puzzles that resemble traditional composed studies in the sense that the solver is invited to figure out the key or first move that leads to the most decisive advantage, i.e. the ‘best’ move. The difference between a study-like construct, puzzle or problem compared to a traditional composed study is that it appears to be more likely reachable in a real game. Compared to a ‘tactical study’ which is basically a position taken from a critical point in a real game (usually between master players), study-like constructs are more economically-designed. In other words, there are not as many superfluous pieces, if any.
The DSNS approach was actually developed to further illustrate the creative abilities of computers and one of the best and most amenable domains to test creativity, in my view, is the composition of chess problems or puzzles. It was also convenient that computational creativity, to my knowledge, had never actually been applied in this domain before, and certainly not using the DSNS approach which was only recently developed. Here is an example of a relatively complex study composed autonomously by Chesthetica.
Chesthetica is also able to compose the rarer “White to Play and Draw” type of study, like the one here.
The complete study collection (in two parts) which is always being updated is available here. Readers are invited to try the problems which are of varying difficulty. I chose them from the much larger ‘raw’ collection of compositions by Chesthetica simply because they looked interesting to me. However, I did not alter them (or the chosen solution) in any way whatsoever. What you see is exactly what Chesthetica created and chose and this, in itself, is a marvel of computational creativity. Why should a computer be able to create even things like this at all? Are they not supposed to be ‘mindless’ automatons doing precisely what they were programmed to do every single time and nothing more? Are they not supposed to ‘repeat themselves’ after a while? Are they not supposed to be predictable? All these notions are clearly being challenged in one go.
Given the ‘success’ of Chesthetica applying the DSNS approach in the domain of chess problem composition, the next logical step was to see if it could be applied to more ‘important’ fields and none commands more respect and urgency than medicine, for obvious reasons. Readers may not find it at all surprising that my parents wanted me to study medicine instead of computer science but I really had no interest in it. To this day, even if I could rewind the clock 20 years, I would still choose to study computer science over medicine. My extended family is full of physicians, by the way. Anyway, I had actually applied for a sizeable research grant to test the DSNS technology in the field of protein folding (as I had earlier written about on ChessBase). This would theoretically have applications in drug design to treat diabetes, cancer and Alzheimer’s, to name a few diseases.
The result of that application came back several weeks ago after 15 months of deliberation and two defense sessions; first with an AI panel of assessors and then transferred to the ‘biotech’ panel which was deemed more appropriate. The answer was no. The project had been rejected and no official reason was given.
A Protein Structure [Huntingtin and Huntington's Disease]
Here is my take on what probably happened. I had two research partners, by the way. A wealthy and independent researcher overseas and a highly talented (easily borderline genius-level) young woman with a Ph.D. in chemical engineering. My associate overseas has been studying protein structures for years and is also a very talented computer programmer working in the finance sector. The young woman, a Malaysian just like me, I found online while looking for suitable candidates to join the project, and was very impressed by her CV. I asked her to join me and she said yes. So I was probably the least competent of the group, yet in charge of the project. To make a long story short, I had virtually no expertise with regard to protein folding so my young associate had to handle that part.
She understood the DSNS approach after I explained it to her and found it fascinating. She too saw the unlimited potential and even necessity in the domain of protein folding. Our relationship developed over the months and she also followed me to the defense sessions, doing her best to explain the worthiness of our project proposal to over half a dozen grey-haired men, all of whom were probably experts in their fields as well. To be honest, the impression I first got during the biotech defense session was that they perhaps did not take her seriously enough for some reason but later I realized that they simply could not wrap their heads around how the DSNS approach would be able to ‘create’ or ‘propose’ new protein structures that might hold the key to curing some of the worst diseases that have plagued humanity for centuries.
This is something we are told the best medical minds have been struggling with to no avail yet here were two ‘nobodies’ (one of whom is not even a ‘biotech person’) proposing they might have a solution. Regardless, I do think they were interested but there were so many technical hurdles (especially on the medical side) with regard to how each proposed protein structure needed to be tested and the timeframe required in order to do so that the 18-month project proposal did not seem feasible. Also, what would be needed is not the paltry 6 figures in funding I was asking for but more like 8 figures and closer to 60 months with no guarantee of a positive result, which was way beyond their scope and risk threshold.
Having worked in AI for many years now, I am used to being able to test and do things relatively quickly but in medicine, it is a whole different ball game, so to speak. Some aspects of medical research are downright ‘primitive’ in comparison and understandably so because biology and silicon are two different things. Two totally different worlds, in fact. Needless to say, I was not surprised by their final decision to reject the project (with no reason given or even an explanation of how to revise or improve the proposal) because there was probably no reason and no way to improve it that would make a difference. The project was simply ahead of its time and not within the purview of what the grant is designed to facilitate. I was glad, however, that with a decision finally made on this project, I am now free to apply for a different proposal (on another topic) if I choose, as two simultaneous applications are not permitted.
My associate overseas was disappointed and begged me not to give up. He had mentioned earlier that protein structures hold the key to immortality and all we needed to do was find the right structures in the 20^200 universe of possibilities. No amount of brute-force computing in finite time could ever suffice in such a vast search space. Creativity (human or machine) was the only way to leapfrog to meaningful or useful areas of that space and I believe we all understood this. Alas, I do not think it is worth trying again on this topic as the same issues are likely to be brought up. Also, my other project member has plans to possibly move overseas so we are essentially disbanded. I have therefore decided to look into applying the DSNS into perhaps general artwork instead.
Demonstrating its efficacy in another domain (apart from chess problems) would help establish the approach as viable enough to be useful to other researchers and perhaps included in AI textbooks someday; not unlike artificial neural networks and genetic algorithms. However, such things can take a lot of time after they are first introduced; decades, even. In any case, I hope to soon document the specifics of what I had in mind in adapting the DSNS approach to protein folding. I doubt any science journal or magazine would even be willing to publish it without experimental data (no grant = no funding = no experiments) so it will either have to be self-published or filed away in one of my drawers for a time. I might even just pass it on to one of my research assistants. It is important, I think, to write these things down quickly because memory fades and inspiration does not seem to have a predictable schedule or even repeat itself.
To be honest, the whole DSNS idea simply came to me years ago over a few days in precise detail which I decided to write down and sketch almost immediately. As the years went by, whenever I sat down in front of my computer trying to solve a particular problem related to the DSNS in code, very often my fingers would seemingly start moving on their own and before I even realized what was really going on, the problem was solved and the program actually compiled and ran without errors, which seldom ever happened working on other programs or other aspects of Chesthetica. In fact, debugging programs is usually a very tedious experience. Only a few days ago in debugging and enhancing Chesthetica from v10.12 to v10.13, I managed to accomplish everything in under an hour, when I had originally suspected would take at least a week of torture and sleepless nights (and therefore put off the upgrade for over a month!).
After a decade of working on Chesthetica for various purposes, it is now stable and functioning quite well in composing three-movers, four-movers, five-movers and studies. It can also evaluate beauty in these problems using the aesthetics model I developed earlier on (for my Ph.D.). So thousands upon thousands of chess problems can be ranked aesthetically and also seemingly endlessly composed from scratch which is something that would require countless man-hours if domain-competent humans were to attempt it. There may be minor enhancements to the program every now and then but for the most part, the AI work (with respect to the aim of the DSNS) appears done.
There was never really any intention (without say, millions of dollars of support from a company like IBM) to have Chesthetica compose only world-class traditional chess problems because perhaps only 1% of all chess players can fully appreciate those whereas 99% of players can probably find something interesting in the constructs Chesthetica creates. Is what they perceive not considered beauty or aesthetics in the game as well? Analogously, are works only by Picasso and Van Gogh worthy of being considered as having ‘creative value’ in the art world? What about all the art artists we have never heard of create that also sell (albeit for not as much)? I think my point should be clear.
I have kept up the initiative to upload a personal selection of Chesthetica’s creations to YouTube even though at an absolute and significant financial loss in terms of the hours and effort involved but I feel it is important work that may pay off one day. If nothing else, there is at least some historical value with regard to the history of computer chess and AI. I will keep doing so for as long as it interests me. Chesthetica could, in theory, go on forever but I cannot. I have even added a link to a periodically-updated PGN of the compositions (in the “About” > “Links” section of the YouTube channel) for those who find the videos too cumbersome or inconvenient and prefer to use a PGN reader of some kind.
You will probably not find any of these positions having occurred in real games or composed by anyone as they are quite literally the products of the ‘mind’ of an AI. Had some of these constructs occurred in actual tournament games between human masters, one can almost imagine Danny King commenting on the key as, “a beautiful move here” with inflection. There are no plans at the moment to make Chesthetica available to the public even though that would be an interesting prospect. I sometimes wonder how it might run on thousands of other machines and what kind of compositions it might be able to create with such a multitude of instances running and thousands of human minds judging its creations.
ChessBase has sort of approached me a couple of times with regard to the potential of applying Chesthetica in certain aspects of its business but nothing materialized. Despite all the publicity via ChessBase, trade exhibitions and others, I have not received any interest from any other commercial software developers either so I suppose it is unlikely to be available, even for purchase, any time soon. This does not bother me as I am not really in need of the money anyway. Overall, the reception from the public has been quite positive, however. People have contacted me wanting to download and even purchase the software and the feedback I have received with regard to the constructs created is generally quite positive as well (80% positive, at least).
So for now let me end this article and the Chesthetica saga by thanking ChessBase and readers for their support all these years. It has been an interesting ride but it is now perhaps time to move on to other pastures. Just as I moved on from computational aesthetics to computational creativity, it is now time to move the DSNS from chess problems to some other domain. I would have liked to apply it in medicine and protein folding but alas, that was not meant to be. Regardless, there are plenty of other areas of research to look into. Life in academia never gets boring as long as the ideas keep coming. It’s what we do.
5/31/2015 – Celebrating 300 machine generated problems
As we reported before, Chesthetica, a program by Azlan Iqbal, is autonomously generating mate in three problems by the hundreds, and the author is posting his selections in a very pleasing format on YouTube. The technology behind the program’s creativity is a new AI approach and Dr. Iqbal is looking for a substantial research grant for applications in other fields.
2/6/2015 – Computer generated chess problems for everyone
Now they are composing problems that fulfil basic aesthetic criteria! Chesthetica, a program written by Azlan Iqbal, is churning out mate in three constructs by the hundreds, and the author is posting them in a very pleasing format on Youtube. How long will Chesthetica theoretically be able to generate new three-movers? Quite possibly for tens of thousands of years.
11/7/2014 – A machine that composes chess problems
Chess problems are an art – positions and solutions, pleasing to the mind and satisfying high aesthetic standards. Only humans can compose real chess problems; computers will never understand true beauty. Really? Dr Azlan Iqbal, an expert on automatic aesthetic evaluation, imbued his software with enough creativity to generate problems indefinitely. The results are quite startling.
7/26/2014 – Best ‘Chess Constructs’ by ChessBase readers
Chess constructs are basically an intermediate form of composition or chess problem, lying somewhere between brilliancies from chess history – and artistic chess problems, between real game sequences and traditional award-winning compositions. A month ago Dr Azlan Iqbal explained the concept asked our readers to submit compositions of their own. Here are the winners.
6/29/2014 – Azlan Iqbal: Introducing ‘Chess Constructs’
People love brilliancies from chess history – and artistic chess problems. But there is a big gap between the two. Positions from games demonstrate the natural beauty of actual play, while chess problems are highly technical, with little practical relevance. The author of this interesting article suggest an intermediate form, one you can try your hand at – and win a prize in the process.
9/2/2009 – Can computers be made to appreciate beauty?
Or at least to identify and retrieve positions that human beings consider beautiful? While computers may be able to play at top GM level, they are not able to tell a beautiful combination from a bland one. This has left a research gap which Dr Mohammed Azlan Mohamed Iqbal, working at Universiti Tenaga Nasional, Malaysia, has tried to close. Here's his delightfully interesting PhD thesis.
12/15/2012 – A computer program to identify beauty in problems and studies
Computers today can play chess at the grandmaster level, but cannot tell a beautiful combination from a bland one. In this research, which has been on-going for seven years, the authors of this remarkable article show that a computer can indeed be programmed to recognize and evaluate beauty or aesthetics, at least in three-move mate problems and more recently endgame studies. Fascinating.
2/2/2014 – A new, challenging chess variant
Ever since desktop computers can play at its highest levels and beat practically all humans, the interest of the Artificial Intelligence community in this game has been sagging. That concerns Dr Azlan Iqbal, a senior lecturer with a PhD in AI, who has created a variant of the game that is designed to rekindle the interest of computer scientists – and be enjoyable to humans as well: Switch-Side Chain-Chess.
