Emotion detection in chess

Emotion detection in chess

Do you often encounter unnecessary mishaps in your tournament games? Do you sometimes panic during the game, are you are sometimes too relaxed, over-confident in an endgame that needs special attention? There are many companies and research organizations developing AI systems to track and record human emotions. Recently we came across one such initiative that is applying this to chess. Initial experiments with 2700 GMs are showing great promise. Maybe this will soon be one of your standard training tools? 

Fritz 17 - The giant PC chess program, now with Fat Fritz

The most popular chess program offers you everything you will need as a dedicated chess enthusiast, with innovative training methods for amateurs and professionals alike.

As many of you know, for the last year and a half I have been involved in a neural network project, based on pioneering work by Google's Deep Mind. It involved developing a chess playing program that basically generated itself. Fat Fritz, as we deemed to call it, could well be the strongest chess playing entity that has ever existed.

While we were developing Fat Fritz I was contacted, in January last year, by an Artificial Intelligence lab that was working on a project to use the webcam of any notebook, and currently one external sensor, to monitor players (of chess or any other game) and track their emotional status. They knew about my involvement in AI and in the area of emotion tracking during chess games.

Boris Spassky and Ulf Andersen in 1979

In 1979, there was a grand international GM tournament in Munich, at which GM Dr Helmut Pfleger carried out medical tests on the players (of which he was one). The point was to monitor their heart rates during a game. Two volunteers were Boris Spassky and Ulf Andersson, who both wore pulse monitors. I was Helmut's assistant. We also monitored him during the games, and in addition used a beta blocker to test its effect.

In the graph (click to enlarge) you can see that the pulse rate during a normal game (top line) can double from its normal rate. This can be avoided by using a beta blocker (bottom line). Helmut was disappointed with his performance when playing "calmly" — apparently the pulse rate highs are required for acute tactical play.

This was a fairly simple experiment — sophisticated monitoring systems were not available at the time. But in the forty years since then there have been enormous advances. I have tested it myself: the above pulse rate detection can be done with the simple webcam on my notebook, just watching my face. Apparently it uses very subtle colour changes in my skin to measure my pulse rate — and it does so very accurately, as I determined using a traditional external sensor.

But there are also techniques available today to detect emotions that are reflected in our behaviour and physiology, such as anxiety, calmness, stress, optimism, etc. While emotions emanating from behavioural expressions such as face and voice can be controlled and altered, the same is not true for physiological sources. For example, one cannot control the way one's heart beats or palms sweat.

The AI group mentioned above (which currently wishes to remain unnamed) has conducted experiments, including two sessions with leading grandmasters, who played over 150 rated online games each, against other GMs and IMs across the world. These were in blitz and bullet formats and were played over a duration of four days. A webcam was used to measure the heart rate and then derive “excitement” level from value measured. Another small external sensor was used to precisely measure the inter-beat intervals of the heart, from which cognitive states can be derived. The typical duration of a game varied from 1 to 3 minutes, which allowed the triggering of different emotional and cognitive states at different stages of games in a relatively short time.

Key findings & observations

• Since the participants were not visually particularly expressive, the observations analyzed from physiological sensing seemed more useful to yield insights.

• In situations where participants made mistakes or lost consecutive games, the heart rate variability index (HRV) was lower than the baseline, hinting frustration.

• The excitement level of participant (inferred from inter-heartbeat intervals) went momentarily high when the opponent blundered. The facial expression of the participant did not change significantly in this case.

• The openings of the games were inferred as “relaxed”. In the scenario where a participant was very familiar with the openings, or was playing with little effort, or with high confidence of winning, the HRV index stayed closer to their baseline.

• In the scenario where participants began problem solving, thinking for relatively longer durations (middle of the game), the HRV index increased well above baseline, indicating higher cognitive efforts.

One of the test subjects, a 2700+ GM, played for some hours during the first day, with average results. The AI group showed him his emotional data during the games and undertook a special feedback training — breathing and positive visualization. In the next session he started killing anyone playing against him. Clearly the training had helped dramatically. "This is absolutely superb," he told me, "truly amazing. I need to have it. Put it into ChessBase, Fred. For me!"

Emotion detection and interpretation

Let us take a closer look at some of the graphs [click all graphs to enlarge];

Heart rate and HRV trend

The first graph shows the variations in heart rate and the heart rate variability from the opening to the end of one of the games played by the 2700+ GM. This graph should be interpreted along with the next graph of inter-beat interval.

Heart interbeat interval trend

The inter-beat interval is the time span between successive heart beats measured in milliseconds with at least a 500Hz sampling rate. As you may know, the human heart does not beat like a metronome. The interval between successive heart beats always varies, in the order of milliseconds. It is the manner of variability within inter-heart beat intervals that is the key to gaining insights into the state of the autonomous nervous system. This insight requires inter-beat interval to be measured with high precision.

Emotion trend

The third graph [click to enlarge] is a measure of the arousal state (level of excitement) of the player, ranging from relaxed (almost sleepy) to highly excited (sudden surprise/fear). This is computed from heart rate. In the current version of the AI software the overall excitement range is split into five zones. However, the 2700+ GM was rarely in a state of high excitement. So, for ease, the graph during his game only shows three zones. The lowest excitement level is "Relaxed," the next higher excitement state is "Calm," and next is "Activated" (on verge of getting excited). In this game, you will notice that the GM starts in a Calm state and then switches between Calm and Activated. He has good concentration (indicated by a high HRV index in middle game), combined with moments of calmness. Like in the opening, his excitement level is back to a Calm State towards the finish of the game.

Initial conclusions of the AI group

Emotion and Chess: While highly structured logical thinking plays a key role in chess, emotions are important as well. For example, a top player can make seemingly amateurish or trivial mistakes mainly based on his or her emotional state (e.g. too relaxed, nervous, etc., which affects their cognitive abilities such as think creatively or make decisions) during the game. And while, some players are expressive (facial reactions) others are not — for example they can make a blunder and act as if nothing has happened.

Current State: In chess tournaments, both over-the-board or online events, chess engines built into the live game streaming platforms can show what the best moves are and who is better. While the audience can see the best moves in a position (using chess engines that are more powerful than any human player), it is usually difficult for them to understand why a player chose a specific move or made a seemingly amateurish mistake. It is left to the commentator's best guess to derive the emotional state of the player. Automatic emotion detection could provide interesting additional insights — though it is unclear whether the players themselves would welcome this broadcast option.

Mental preparation: Chess coaches place a lot of emphasis on psychological preparation — players should maintain the right emotional state for optimum performance. There are books that specifically reference the subject, for instance Practical Chess Psychology by Amatzia Avni, in which the author talks about goal-setting, failure, underhanded psychological ploys, concentration, determination, etc.

Chess training: If emotional state data is captured during online play and baked into the game file, that would be incredibly useful for players during their postgame analysis. With this data collected during your games your emotional patterns can be deciphered (e.g. a tendency to make mistakes during the transition from a complex to a simple position, due to an overly relaxed state). Additionally, emotion data gathered from tournament games of top players (who agree to cooperate) can be included in chess databases, so that when the games are studied by players at lower levels, they not only learn chess strategy but also the key psychological aspects that influence the outcome of chess games.

HRV training: Also, structured emotional training (breathing, visualization etc.) with real-time feedback, as a regular practice, could strengthen quality of play by enabling the player to maximize cognitive abilities during game play.

Future intentions

Enabling heart rate and emotion detection during game play could become a potent training tool for tournament chess players. ChessBase intends to delve deeper into the opportunities this technology provides, in a pilot project. The webcam monitoring (basically overlaying heart rate information with moves) followed by more detailed emotions sensing (e.g. showing arousal levels such as “excited” or “relaxed” or “bored/sleepy” or “passive”) can be used in a 3-6 months trial to validate the benefits of a structured HRV training to improve quality of play. Chess can be used as a vehicle to quantitatively measure the players performance based on quality of moves as well as emotional regulation during games.