Observational Analysis of Mistakes in Chess Initiation, Using Decision Trees

A consensus exists in the scientific community in attributing beneficial aspects to child chess practice in the development of children’s executive functions (Aciego et al., 2012; Kazemi et al., 2012; Smith & Cage, 2000; Trinchero, 2013). Other work has shown improvement in the academic performance of students between the ages of 4 and 18 (Celone, 2001; Gumede et al., 2017; Sala & Gobet, 2016). These academic improvements, associated with chess, also include students with disabilities (Barrett & Fish, 2011; Scholz et al., 2008; Storey, 2000) or low academic achievement (Hong & Bart, 2007). The benefits of chess practice are also linked to everyday activities (Bilalić, et al., 2008), with chess players themselves able to perceive the benefits of chess in their daily performance (Chitiyo et al., 2021).

The process of initiation to chess demands considerable intellectual effort (Jerrim et al., 2016). The beginner is confronted with aspects such as the mobility of the pieces, the evaluation of the positions on the board or the understanding of the implications of each move. Knowledge of the game is acquired through a process of initiation, usually facilitated at school or in chess academies, in what amounts to an educational chess environment (Hall & Nash, 2021). In this environment, player errors are inevitable and are an essential part of the learning process (Linhares et al., 2012).

The concept of mistake in chess has undergone a profound conceptual transition (Kasparov, 2018), from subjective and qualitative personal assessment by an expert observer (De Groot, 1965; Dvoretsky, 2003; Grau, 1943; Saariluoma, 1992) to objective and quantitative measurement by chess analysis software. According to Guid, et al., (2008), while computer uses evaluations in a numerical form, human player usually has in mind descriptive evaluations, such as small advantage, decisive advantage, and unclear position.

In 1997 “Deep Blue” managed to surpass the level of human play, beating the world champion. Today there are analysis engines such as “Stockfish 16 NNUE” with an estimated ELO rating (Elo & Sloan, 1978) of 3500 points, which is significantly higher than any other player (Anderson et al., 2017). This software measures the quality of moves in “centipawns” (Tays, 2023) so that the closer the move made by the player is to the optimum set by the software, the fewer centipawns are lost. The value given to the centipawn measurement is near to 0 when the ideal move is made and increases its value as the quality of the move decreases. Therefore, for example the loss of 0.5 centipawns is the equivalent to losing half a pawn; or the loss of a whole centipawn unit equates to having lost a pawn, without involving the literal loss of a pawn following the move. According to McIlroy-Young (2020), there are three levels of error according to the degree of loss of centipawns per move: inaccuracy (loss of up to 0.5 centipawns), mistake (loss of between 0.7 and 1 centipawn) and blunder (loss of more than one centipawn).

Based on the above, this paper aims to analyse the definitive errors, those that if made cost the player the game, in beginner’s chess (U8 category). This is the youngest age group at which national competitions are held, using the decision tree analysis technique.

This research is framed within an observational methodology (Anguera, 1979), based on its justified consideration as mixed methods in itself (Anguera, et al., 2017), which has developed its own way of quantifying based on the “connect” option (Creswell and Plano Clark, 2017) allowing the alternation of QUAL-QUAN-QUAL stages.

According to Anguera et al. (2011), the observational design used is: nomothetic (73 participants), inter- and intra-sessional follow-up (what happened in each of the games of each round of the championship analysed) and multidimensional (the different criteria of the observation instrument).

Regarding the type of observation (direct, indirect), this study produces an exceptional enclave in which both are considered in an integrated way, given that direct observation (Sánchez-Algarra & Anguera, 2013) is required through visual perception and indirect observation (Anguera et al., 2018) from the reasoning and strategy that each player has cognitively developed, taking into account the rules that regulate chess.

Data analysis

The objective of the present work is met by means of the decision tree analysis technique. A decision tree is a non-parametric supervised learning algorithm (Alpaydin, 2014) that can be applied to both classification tasks, when the criterion dimension measure is qualitative, and regression tasks, in case the criterion dimension measure is quantitative (Yang & Zhou, 2020). In particular, this technique allows for three interconnected analyses: 1) it facilitates the search for the best associations of predictor variables with the criterion variable; 2) it makes it possible to discover which categories or values of a predictor variable are homogeneous in relation to the criterion variable; 3) it detects interactions between predictor variables (Escobar, 2007). The essential components of a decision tree model comprise nodes (root, internal and terminal) and branches, while crucial steps in model building are splitting, stopping and pruning (Song & Ying, 2015). The decision tree algorithm starts at the root node, which represents the entire sample. It then divides the sample into two or more homogeneous sets according to the most significant characteristic, represented by the branches of the tree. This procedure is repeated recursively for each secondary node until the terminal nodes, which represent the final predictions, are reached (Jindal et al., 2023).

In this study, classification trees have been chosen because the criterion dimension (with identifier no. 20) “impact of the error” is categorical in nature. The model called “cataloguing the error made” tree (Figure 1) incorporates information related to the context, integrating the predictor dimensions “seriousness of the error” and “real consequence” to the predictor variables of the first tree. On the other hand, the “disposition of the error made” tree (Figure 2) focuses on the error itself, through the predictor dimensions “number of different pieces”, “simultaneity of tactical aspects”, “consequence zone”, “range of movement” and “number of turns”.

Figure 1.

Decision tree of “cataloguing the error made”.

Figure 2.

Decision tree of “disposition of the error made”.

There are several statistical algorithms for the construction of decision trees (Song & Ying, 2015) such as: Classification and Regression Trees (CART), C4.5, Chi-Squared Automatic Interaction Detection (CHAID) and Quick, Unbiased, Efficient, Statistical Tree (QUEST). The choice of the CHAID algorithm in this study is based on its recognition as one of the most widely used supervised learning methods, noted for its efficiency in data analysis and accurate detection of significantly correlated factors (Yang et al., 2023). Moreover, it stands out as a more stable technique, and one that obtains more accurate results, when confronted with many categorical dimensions (van Diepen & Franses, 2006; Yang, et al., 2023). IBM SPSS Statistics 28.0 software was used to develop the algorithm.

In general terms, the CHAID algorithm uses the Chi-square test to guide its decision-making process on how to separate or group data. At each step, CHAID chooses the predictor dimension that has the strongest interaction with the criterion dimension so that each predictor is significantly different relative to the predicted one. In addition, during this process, the categories of each predictor are grouped together if they do not show significant differences with respect to the criterion dimension. This strategy seeks to avoid over-fitting and build a more efficient and easily interpretable model (Arabfard et al., 2023; Escobar, 2007). In this work, the filters used to regulate the expansion of the classification trees in terms of depth and number of nodes have been the following. Firstly, level filters aimed at restricting tree growth in terms of depth. In this regard, it is important to note that the automatic adjustment of the SPSS program limits the tree to three levels below the root node for the CHAID methods. Secondly, size filters to control the expansion of the trees by limiting the number of frequencies in the tree segments or nodes, both internal and terminal. In our study, the recommended rule of thumb of 100 cases for internal nodes and 50 cases for terminal nodes is followed (Escobar, 2007). And thirdly, CHAID employs the significance filter. Its criterion is to avoid segmentations that are not statistically significant based on the Chi-square statistic, with significance limits set at the 0.05 level. Finally, the conclusion of the analysis is achieved by assessing the predictive accuracy of the segmentation set, which represents the goodness of fit of the model, by estimating the risk of the classificatory ability (Escobar, 2007).

The two decision trees (“cataloguing the error made” and “disposition of the error made”) obtained using the CHAID procedure, taking “impact of the error” as the predicted dimension, are presented below.

Figure 1 shows the classification tree called “cataloguing the error made”. This tree is three levels deep and has four terminal nodes identified as 1, 3, 5 and 6. The first level indicates that the best classifier is the “real consequence” dimension, followed by the “seriousness of the error” dimension, and finally, at the third level of analysis is the “type of error” dimension. The following dimensions are discarded: “number of different pieces”, “simultaneity of tactical aspects”, “consequence zone”, “range of movement” and “number of turns”.

As for the first level, the “real consequence” dimension branches into two nodes: “Not exploited” (node 1) and “exploited” (node 2). Node 1 reveals that within the 65.3% of errors made where the consequence is not exploited, 98.7% of these errors do not turn out to be definitive or gamelosing error.

Following the vertical reading of the tree, the second level involves the branching of node 2 into nodes 3 and 4, as a result of the interaction of the “seriousness of the error” dimension. Terminal node 3 specifies that, among the total number of errors exploited, 10% are mistakes. 94% of these errors are not definitive.

Finally, at the third level of the tree, the subdivision of the “blunder” category of the “seriousness of the error” dimension, due to the interaction with the “type of error” dimension, leads to the appearance of two terminal nodes: node 5 includes the macrocategories “undefended pieces” and “subtle mistake”, while node 6 encompasses the macrocategories “tactical motifs” and “checkmate”. Specifically, node 5 predicts that when the “type of error”, either categorised as “undefended pieces” or as “subtle mistake”, in the antecedent conditions of “blunder” and being the consequence “exploited”, there is a 60.3% probability that these errors will not be losing blunders. Conversely, in those same previous situations, if the “type of error” is classified as either “checkmate” or “tactical motifs” there is a 60.9% chance that these errors will be definitive.

Regarding best-fit of the model, the results show that this CHAID tree correctly classifies 89.1% of the errors in chess. In particular, a higher percentage of correctness within the dimension “impact of the error” is shown for the category “non definitive” (95.6%) than for the category “definitive” (45.9%). In order to evaluate the generalizability of the model, a cross-validation procedure provided by SPSS was applied. The results indicate an error rate of 12.8% with crossvalidation, compared to an error rate of 10.9% obtained without applying cross-validation. This reflects a slight decrease in performance when the model is evaluated on unseen data partitions, but it confirms the robust predictive capability of the CHAID model.

Figure 2 depicts the “disposition of the error made” classification tree, which is three levels deep and has six terminal nodes labelled 2, 4, 5, 7, 8 and 9. At the first level, the best classifier is the dimension “number of turns”, followed by the dimensions “range of movement” and “type of error” at the second level. Finally, at the third level is the “consequence zone” dimension. The dimensions “number of pieces” and “simultaneity of tactical aspects” are excluded from the analysis.

At the first level, the dimension “number of turns” is divided into three nodes: node 1 includes the categories of “5, 3, 4 and 2 movements”; node 2 covers the categories of “more than 5 movements” and “wide open”; and node 3 corresponds to the category of “1 movement”. Node 2, which is the only terminal node at the first level, indicates that within the 22.7% of errors committed in the “number of turns” that have “more than 5 movements” and “wide open”, 98.1% of these errors do not turn out to be definite, which is the condition with the highest probability of not committing a definite error reflected in this tree.

The second level is divided into two branches. The first is the splitting of node 1 into terminal nodes 4 and 5, as a consequence of the interaction of the “range of movement” dimension. In both nodes, the percentage of non-definite errors is higher, although this percentage is higher when the “range of movement” is “open” or “semi-open” (96.7%) than in the “closed” category (86.0%) when the “number of turns” of play involves 2, 3, 4 or 5 movements. The second branch, the splitting of node 3, is due to the interaction with the dimension “type of error” generating nodes 6 and 7. This last node is terminal and indicates that 94.0% of the “subtle mistake” type of error when the game turn involves “1 movement” does not turn out to be a definite error.

On the other hand, the interaction of node 6 with the criterion “consequence zone” gives rise to the third level of the tree with terminal nodes 8 and 9. Node 8 encompasses the macrocategories “starts” zone and “centre” zone, while node 9 corresponds to the macrocategory “flanks”. In both situations, the percentage of non-definitive errors is higher, although it is higher in the “flanks” (85.3%) than in the macrocategories “start” and “centre” (68.4%), when the “type of error” includes the categories “undefended pieces”, “tactical motifs” or “checkmate” preceded by “1 movement” game turns.

In terms of predictive ability or goodness of fit, the results reveal that this model correctly classifies 86.9% of the final errors in chess. Specifically, a higher percentage of accuracy is observed within the “impact of the error” dimension for the “non definitive” category (100%) compared to the “definitive” category (0%). As with the first model, in this second analysis, we applied cross-validation using the CHAID method in SPSS. The results indicate that the error rate is identical in both resubstitution (0.131) and cross-validation (0.131), suggesting that the model’s generalization capability is not significantly affected by the application of crossvalidation. These results reinforce the conclusion that the CHAID model has a good generalization ability.

We now proceed to the discussion of the results that will allow us to achieve the objective of this work, namely to analyse the definitive errors (those that cost the player the game) in beginners’ chess -the youngest age at which national competitions are held (U8)- by means of the decision tree technique.

In this paper, decision trees have been used as a relevant segmentation technique in several research fields (Lee, et al., 2022), supported by their interpretability, prediction accuracy, speed of calculation and wide availability of software (Loh, 2014). In addition, the visual representation of decision trees facilitates their understanding, allowing the most relevant variables to be identified quickly, which is not always so easy with other algorithms (Higueras-Castillo et al., 2023). This combination of benefits positions decision trees as powerful and accessible tools in the scientific community.

To meet the pre-set objective, two different decision trees have been made in which the starting point (node 0) has been set as the “impact of the error” dimension, that differentiates between a “definite error”, meaning an error that inevitably leads to the loss of the game, regardless of the number of moves until the end of the game (Anderson et al., 2017), and “non-definite errors”, in which the game is not lost as a result of the error made.

We proceed to the discussion of the results of the “cataloguing the error made” tree (Figure 1). The decision tree shows us the dimension “real consequence” (nodes 1 and 2) as the best predictor of the dimension “impact of the error” (node 0); the results exposed by nodes 1 and 2 are consistent with the internal logic of “sum 0” chess (Balduzzi, et al., 2019), since: a) if the error was not exploited we could not speak of a definite error; b) when the error is exploited by the opponent, the probability of an error becoming definite increases.

On a second level (nodes 3 and 4), derived from node 2 “real consequence”, the decision tree reflects the interaction with the dimension “seriousness of the error”. Node 3 shows how “mistakes” that are exploited are rarely (6%) converted into final errors. Node 4, however, indicates that the “blunders” exploited become (48.4%) final errors. This circumstance also fits in with the reality of expert chess (Dvoretsky, 2003).

At the third level of analysis (nodes 5 and 6), derived from node 4 “blunder”, the decision tree incorporates the dimension “type of error”. Node 5 reflects that blunders of “undefended pieces” and “subtle mistake” become definite errors only in 39.7%. However (node 6) the blunders “tactical motifs” and “checkmate” become definite errors in 60.9%. This evidence deepens the separation established by expert chess players (Beim, 2022) of two distinct but complementary frameworks for analysing chess positions in order to make the most appropriate piece move: “tactics” and “strategy”. Tactics is understood as the analysis of a given position looking for specific defects in the opponent’s position in order to achieve a material advantage or checkmate in the short term and by means of very specific moves. Strategy, on the other hand, refers to general, long-term, more abstract plans, such as the domination of the centre of the board, the search for weak squares or aspects related to pawn structure (Dvoretsky, 2004). Ultimately, the strategy is the plan to be carried out – for example the search for weakness implied by moving the king to the middle of the board where it is easily attacked; the tactic, meanwhile, is how to carry out this plan – for example, how to attack that king through tactical issues, undefended pieces or checkmates. The present research shows that in introductory chess, errors related to “tactics” and “calculation” are more serious than those related to “strategy” in the game (Chandler & Burgess, 1998).

The “disposition of the error made” tree, shown in Figure 2, shows the dimension “number of turns” (nodes 1, 2 and 3) as the best predictor of the dimension “impact of the error” (node 0); the results shown by nodes 1, 2 and 3 are consistent with the relationship of chess with memory and the ability to find appropriate sequences within the diversity of moves available in a position (Aargaard, 2008). The probability of an error being final for the side making the error decreases as the number of turns players have to calculate increases (23.8% when players have to calculate a single turn; and between 1.9% and 10.8% when they have to evaluate more turns). The results show that elite U8 players have a better ability to calculate the next turn in a position and a lower ability to calculate subsequent turns. This differentiation between next turn and turns after the next turn was already established by Grau (1943) who postulated an immediate and a mediate vision of the game. Immediate vision is linked to a player’s ability to know all the possible moves available in a given position. Instead, the mediate vision is not based on the turn immediately afterwards; it refers to a player’s ability to evaluate more than one move in order to anticipate the opponent’s risks and devise solid plans of attack. There is a consensus on the importance of mastering first immediate vision and then mediate vision (Dvoretsky, 2003; Kotov, 2005), as there is no point in training one’s ability to analyse several turns if one does not even know all the possibilities offered by the position of the pieces.

In nodes 4 and 5, derived from node 1 (“2 movements”, “3 movements”, “4 movements” and “5 movements”), the decision tree shows a relationship with the dimension “range of movement”, establishing that in positions where there is more than one good move option (“semi-open” and “open” ranges) the players who make the error have a lower probability (3.3%) that the error made is definitive than when it occurs in situations of “closed” range (14.0%). These results are in line with the findings of: Guid et al. (2008) who postulated that positions where the analysis engines show a variability of options pose greater problems to chess players regardless of their level; and Anderson et al. (2017) who pointed out that there are moves whose analysis does not vary according to time, but that there are positions in which, due to their extreme difficulty, the analysis changes according to the time available for processing (also chess engines).

In nodes 6 and 7, derived from node 3 “1 movement”, the interaction occurs with the dimension “type of error”, reflecting similar information as in nodes 5 and 6 of the “cataloguing the error made” tree (Figure 1). The dimensions related to calculation and tactics (Kotov, 2005) are associated in node 6 and show a higher probability (27.2%) of becoming definitive errors leading to defeat in the game. The strategy dimension is located in node 7 and has a much lower percentage (6.0%) compared to node 6. Therefore, it is further concluded that, in players with the characteristics of the selected sample, tactical and calculation errors have more negative consequences for the side that commits them than errors of a strategic nature, as they are linked more to definitive errors that directly cost the loss of the game.

At the third level of analysis (nodes 8 and 9), derived from node 6 which incorporates the categories “undefended pieces”, “tactical motifs” and “checkmate”, the decision tree includes the dimension “zone”. Node 8 shows that in “centre” and “start” error situations on the board, in “undefended pieces”, “tactical motifs “and “checkmate” errors where there is “1 movement” to calculate, the percentage of definite errors amounts to 31.6%. On the other hand, node 9 shows that in “flanks” error situations on the board, in “undefended pieces”, “tactical motifs” and “checkmate” errors where there is “1 movement” to memorise, the percentage of definite errors drops to 14.7%. Thus, in the chess of the age group in question, what happens in the centre of the board during the game is more relevant than what happens on the flanks, in line with what happens in expert chess, where the strategic pillar of a game is to achieve dominance of the centre of the board -see Beim (2011) and Hellsten (2012)-.

There are three main findings in relation to the aim of the study to analyse the definitive errors, i.e. those that if committed cost the player the game, in beginners’ chess (U8 category, the earliest age at which national competitions are held), and which will enable the work in chess schools to be oriented: 1) For an error to be definitive, the non-offending side has to find the right moves to exploit the weakness created by the opponent. 2) The number of turns to calculate will be relevant for the non-offending side to find the exact moves that lead to victory, the fewer the number of turns to evaluate, the more likely it is that the right combination will be found; the need to avoid making mistakes that are exploited in a single turn is crucial. 3) The errors that lead to the loss of the game for elite U8 players are related to short-term calculation (tactical motifs, undefended pieces or checkmate) as opposed to long-term strategic errors.

The practical application of the results of this study raises a two-fold recommendation which chess schools should implement with players at this level: firstly they must stress a mediate vision of the game (the next turn to be played) in order to find the possible moves available to one particular position, presenting positions in which they have to discover the appropriate moves to find undefended pieces, tactical issues and checkmates; followed by an insistence on the long term aspects and general plans of the game.

In terms of future research, this article raised two relevant points: the analysis of the differences in decision making between elite and non-elite U8 players; and the analysis of whether similar patterns are produced in adult beginners.