Estimating Probability Distributions of FIFA-Defined Phases of Play Based on Inter-Analyst Diversity

Definition of Play Phases

In this study, analysts annotated match footage according to the definitions of phases of play provided by FIFA (FIFA, 2022). FIFA defines 16 phases of play that a team can be in at any given time, among which 9 are considered principal phases. Each of these 9 phases is described by a definition ranging from two to ten lines in English as shown in Table 1. This study targets these 9 principal phases, as shown in Table 1. We denote the set of these target phases as 𝒫, and each element as p_i ∈ 𝒫(i = 1, … , 9) using the symbols p₁ to p₉ as shown in Table 1.

Representation of Probability Distributions

Annotation by a Single Analyst: A single analyst annotates a team’s play phase as either one of the phases p_i ∈ 𝒫 or as ɛ, which signifies “not applicable.” We define the complete set of labels, including ɛ, as 𝒬 = 𝒫 ∪ {ϵ}, with each element q_j ∈ 𝒬 (j = 1, … , 10). The annotation by analyst k ∈ {1, … , N} for team l ∈ {1,2} at time t is formulated as a probability distribution P_k,l,t, defined as a one-hot vector in Equation (1).

Here, P_k,l,t(q_j) takes a value of 1 if analyst k annotates the phase of team l at time t as q_j, and 0 otherwise. The sum of the elements in P_k,l,t satisfies ${{\displaystyle \sum }}_{j=1}^{10}{P}_{k,l,t}\left(q_j\right)=1$ .

Aggregation of Multi-Analyst Annotations: Rather than employing a majority vote to produce a single one-hot label, we treat the annotations from N analysts as a probability distribution that captures the diversity of expert opinions. The probability distribution of play phases for team l at time t, TP_l,t, is defined in Equation (2).

Each element TP_l,t(q_j) is defined in Equation (3) as the proportion of the N analysts who annotated the phase of team l at time t as q_j.

While the value of TP_l,t(q_j) is restricted to discrete increments of 1⁄N within the range [0, 1], it functions as a quasi-continuous representation of collective analyst judgment. This distribution ranges from 0 (unanimous disagreement) to 1 (unanimous agreement) and satisfies the constraint ${{\displaystyle \sum }}_{j=1}^{10}{TP}_{l,t}\left(q_j\right)=1$ .

In conventional approaches that do not consider diversity, the team’s play phases are represented as a one-hot vector ${\widehat{TP}}_{l,t}$ , as shown in Equation (4).

Each element ${\widehat{TP}}_{l,t}\left(q_j\right)$ is defined by Equation (5).

Following the precedent of prior studies using two or three analysts, a play phase is typically assigned only when a consensus is reached by at least two individuals. Consistent with this principle, our formulation defines a majority as agreement from more than half of the analysts. In the event of a tie-break where no single phase achieves a strict majority, the sequence is not assigned to any specific tactical category.

Annotation Datasets

This study uses the SoccerTrack v2 dataset (Scott et al., 2025). This dataset comprises match footage, tracking data, and event data from 10 amateur-level matches, primarily from the University of Tsukuba. The match footage was captured using Bepro’s camera system. This system generates a seamless panoramic video overlooking the entire pitch by automatically stitching footage from three cameras installed on the sidelines.

Due to limited resources regarding the number of analysts and available annotation time, we strategically extracted in-play sequences from the 10 matches, totaling approximately 180 minutes. The in-play sequences range from 30 seconds to 6 minutes. During selection, we prioritized scenes with frequent transitions, specifically ensuring that transitions account for nearly half of each in-play sequence. This selection strategy serves two primary purposes. First, it balances the dataset for model training; while defensive and offensive phases like build up often exceed 30% of match time, transition phases such as counter-attack typically account for less than 5% (FIFA, 2022). Second, it incorporates inherent ambiguity into the dataset. Unlike the more distinct offensive and defensive phases, transition phases are harder to define and more prone to diverse interpretations among experts, providing a more robust foundation for probabilistic modeling.

Annotation Process

The annotation is conducted by a total of 16 analysts, comprising 14 members of the University of Tsukuba Football Club’s analysis team and 2 individuals with equivalent soccer and analysis experience. To manage the workload, the 180 minutes of match footage are divided into four sets of approximately 45 minutes each. Each set is assigned to a group of four analysts, ensuring that every moment in the dataset is independently annotated by four individuals (N = 4 in Equation (2).

While existing studies typically rely on a cross-check between only two or three independent labelers (Bauer et al., 2023; Bauer & Anzer, 2021; Chawla et al., 2017), our framework employs four independent analysts per set. This multi-analyst approach provides a more robust cross-check and significantly higher reliability in representing the ground truth. To ensure the integrity of this independent evaluation, all analysts work individually without access to the annotations of others.

The task is conducted using a dedicated application provided by Bepro, as shown in the interface screenshot in Figure A1. The software allows the use of an AI-powered video that automatically tracks and zooms in on the ball or specific players. The task is performed via keyboard operations, with keys ‘0’ through ‘9’ assigned to each element of 𝒬 (the 9 phases plus ɛ). When an analyst judges that the focused team has transitioned to a specific phase, they press the corresponding key. To maintain consistency with international standards, analysts are permitted to refer to a paper copy of the FIFA phase definitions at any time during the process. Each analyst first focuses on one team for the 45-minute video and then repeats the process for the opposing team, resulting in approximately 90 minutes of work per analyst.

Problem Formulation

Input Representation: This study uses the trajectory data of 22 players and the ball included in the SoccerTrack v2 dataset as input to the deep learning models. First, we define the state variable at time t. The set of positions for the ball and all players from both teams, denoted by X_t, is defined in Equation (4).

Here, $x_{\text{ball}}^t,x_{1,a}^t,x_{2,b}^t\in {ℝ}^2$ represent the 2D on-pitch coordinates of the ball, the a-th player of team 1, and the b-th player of team 2, respectively.

The time-series sequence 𝒳_T, which serves as the model input, is defined as the sequence of states collected over a duration of 2L seconds (L seconds before and after the center time T), as shown in Equation (5).

In this study, we set the window duration to L = 10 s (10 s before and 10 s after the center time) and the sampling frequency to f_s = 5 Hz (Δt = 0.2 s). Consequently, the length of each input sequence is W = 2 × L × f_s = 100 frames.

Target Representation: The target probability distribution of play phases for team l at time T, denoted as Y_l,T, is defined as a 9-dimensional vector whose elements represent the probabilities of the 9 phases, as shown in Equation (6).

Here, y_l,T(p_i) denotes the probability that team l is in phase p_i at time T. As described earlier, we aggregate the annotations from N = 4 analysts to construct the ground-truth label. Consequently, each element of the target vector takes one of the discrete values listed in Equation (7).

Learning Objective: The deep learning model f is trained as a regression function, f(𝒳_T) = Y_l,T, that predicts the probability distribution Y_l,T from the input sequence 𝒳_T.

For the ablation study, we additionally consider two alternative ground-truth labels for comparison:

• ➔ Deterministic Label ${\widehat{Y}}_{l,T}$ (Eq. (8)): A one-hot label obtained from the majority vote among the analysts, which ignores annotation diversity.

  (10)

  $${\widehat{Y}}_{l,T}={\widehat{TP}}_{1,T}\left(p_1\right),\dots ,{\widehat{TP}}_{1,T}\left(p_9\right)\in {\{0,1\}}^9.$$

• ➔ Dual-Team Label Y_T (Eq. (11)): A combined label obtained by concatenating the probability distributions of both teams, enabling joint learning.

  (11)

  $$\begin{array}{c}{Y}_T={\left({TP}_{1,T}\left(p_1\right),\dots ,{TP}_{1,T}\left(p_9\right),{TP}_{2,T}\left(p_1\right),\dots ,{TP}_{2,T}\left(p_9\right)\right)}^T\\ \in {\{0.00,0.25,0.50,0.75,1.00\}}^{18}.\end{array}$$

Training Dataset

We utilize spatiotemporal tracking data from the SoccerTrack v2 dataset. These trajectories, representing the ball and all 22 players, are extracted at 25 fps using Bepro’s proprietary optical tracking system from full-pitch panoramic videos. The reliability of this technology is underscored by its FIFA Quality Certification for Electronic Performance and Tracking Systems (EPTS) (BEPRO, 2022). In official FIFA validation tests, Bepro’s fixed and portable camera systems achieved performance ratings that significantly exceed industry standards for both positional and velocity-based metrics. This high precision is maintained through a human-in-the-loop validation process and systematic data cleaning, ensuring that the trajectories meet professional-grade standards for tactical analysis across diverse match environments (BEPRO Dev Team, 2022). By utilizing these FIFA-certified commercial datasets, we aim to ensure that our proposed model is compatible with the data formats and accuracy levels currently prevalent in professional football coaching and analysis.

Prior to dataset construction, two specific in-play segments are set aside for qualitative evaluation to ensure the model is tested on unseen, continuous match scenarios. From the 180 minutes of annotated match footage, we extract training sequences using a sliding window approach. Specifically, for each in-play segment ranging from 30 s to 6 min, we generate 20-s input sequences by sliding the center time T in 0.2-s increments. To ensure temporal context, windows start 10 s after the beginning and end 10 s before the conclusion of each segment. This process yields a total of 50,553 unique sequence-target pairs.

To leverage the model’s design for single-team estimation and increase the dataset’s volume, we apply point-symmetry augmentation relative to the pitch origin. For each annotated sequence, we construct two separate training samples—one for each team. For the second team’s sample, all coordinates are flipped and normalized so that the attacking direction remains consistent from the model’s perspective. This effectively doubles the dataset size to 101,106 samples.

The augmented dataset is partitioned into training, validation, and test sets on a per-segment basis to prevent data leakage. We utilize an optimization algorithm that assigns each in-play segment based on its internal phase scores. This approach minimizes the statistical discrepancy of play phases across the three sets by minimizing a quadratic error function, as detailed in the Segment Assignment Optimization section of the Appendix. This ensures that each set maintains a balanced representation of tactical patterns. Consequently, this segment-wise partitioning results in an approximate distribution of 80%, 10%, and 10% for the respective datasets relative to the total phase occurrences.

Training Setup

We employ the Huber Loss as the objective function for training. Huber Loss behaves quadratically like the L2 loss for small errors and linearly like the L1 loss for large errors. This characteristic allows it to achieve both stability in training and robustness to outliers, making it well-suited for our task of regressing continuous labels.

The batch size is unified to 128 for all models. We adopt the AdamW optimizer (Loshchilov & Hutter, 2017) with a weight decay of 10⁻⁵ for all experiments. To further prevent overfitting, we set a dropout rate of 0.1 across all models. The maximum number of training epochs is set to 300, and Early Stopping is applied to all models. Specifically, training terminates if the validation loss does not improve for 20 consecutive epochs, and the model weights from the epoch with the minimum validation loss are adopted. To ensure convergence to a finer local optimum, the learning rate is halved if the validation loss fails to improve for 10 consecutive epochs. This adjustment allows the model to perform more precise weight updates when the learning process reaches a plateau. The initial learning rate for each model is selected from a candidate set of {10⁻³, 5 × 10⁻⁴, 10⁻⁴} based on the best performance on the validation dataset. Furthermore, to ensure a fair comparison, we individually adjust the hidden dimensions and architectural parameters of each model so that their total number of parameters is approximately equal to 140k.

Model Architecture

In this study, we design and comparatively evaluate four deep learning models with different characteristics for the task of play phases estimation. For all models, the input consists of the 20 coordinates of the ball and all 22 players. In the Transformer and Baller2vec models, these are structured as a matrix ${\widehat{𝒳}}_T\in {ℝ}^{\text{W}\times \text{D}}$ , where W is the sequence length and D is the feature dimension. The feature dimension D is constructed by concatenating the 20 coordinates of the ball, the 20 coordinates of all 11 players from team 1, and the 20 coordinates of all 11 players from team 2, in that specific order (D = 23 × 2 = 46). Within each team, players are ordered by their tactical roles (e.g., GK, 0F, MF, FW). In contrast, the Graph Convolutional Network (GCN)+Transformer and Graph Attention Network (GAT)+Transformer models treat the input as a graph, denoted as 𝒳_T in Equation (7). The graph structure is defined by a static adjacency matrix where edges exist between teammates and between the ball and all players, while no edges are defined between opposing players.

Transformer: We implement a Transformer encoder (Vaswani et al., 2017), to capture temporal dependencies. The input is projected onto a d_model-dimensional latent space via a linear embedding layer, followed by the addition of sinusoidal positional encodings. The encoder consists of L = 3 layers, each configured with M = 4 attention heads, a d_model of 60, and a feed-forward dimension of 4 × d_model. To aggregate temporal information, the latent representation of the final time step is extracted and passed through a fully connected layer to produce the 9-dimensional output.

Baller2vec: This model extends the Transformer by incorporating agent-wise attention (Alcorn & Nguyen, 2021). The 20 coordinates are first concatenated with player I0 embeddings and projected through a multi-layer perceptron (MLP) to a d_model-dimensional space. At each time step, a learnable CLS token is appended to the set of player and ball embeddings. Unlike the baseline Transformer, baller2vec captures spatiotemporal dependencies by applying self-attention across the unified sequence of all agents over time, governed by a custom self-attention mask to maintain temporal causality. To manage the computational cost of this multi-dimensional attention, the sequence length is downsampled from 100 to 20 frames. The model uses L = 3 layers with M = 4 heads and d_model = 64. The final probability distribution is produced by a linear classifier applied to the CLS token representation from the final time step.

GCN+Transformer: This architecture employs a 2-layer GCN (Kipf & Welling, 2016) for spatial feature extraction, followed by a Transformer for temporal integration. At each time step, the GCN aggregates features from 2-hop neighboring nodes based on the adjacency matrix 𝒳_T. This two-layer configuration is chosen to capture higher-order tactical relationships, such as third-man positioning, while avoiding the over-smoothing common in deeper graph networks. To transform the spatial node embeddings into a format suitable for temporal analysis, we utilize a set-attention mechanism. A single learnable query vector performs multi-head attention over the GCN-derived node embeddings to pool spatial information from the 22 players and the ball into a compact representation. The resulting feature vectors have a dimension d_model of 64, which is the product of the hidden dimension and the number of queries. These vectors form a time-series sequence for a Transformer encoder with L = 3 layers, M = 4 heads, and a feed-forward dimension of 2 × d_model. Finally, the latent representation of the last time step passes through a fully connected layer to produce the play phase probability distribution.

GAT+Transformer: This model replaces the GCN with a 2-layer GAT (Veličković et al., 2017) to enable more adaptive spatial feature extraction. The GAT operates on the same graph structure 𝒳_T but dynamically learns attention weights to prioritize critical tactical interactions, such as those between a ball carrier and support players. The first GAT layer employs M = 4 attention heads followed by an ELU activation, while the second layer aggregates these features into a single-head representation. Similar to the GCN-based model, a single learnable query vector is used via a set-attention mechanism to pool the resulting node embeddings into a d_model-dimensional frame representation. This spatial feature, with a dimension of 64, is then processed by a Transformer encoder with L = 3 layers, M = 4 heads, and a feed-forward dimension of 2 × d_model. Finally, the latent representation of the last time step is passed through a fully connected layer to produce the output.

Quantitative Evaluation of Dataset Characteristics

We assess the characteristics of the constructed probability distribution dataset, focusing on the diversity and reliability among analysts.

First, we quantify the extent of diversity through two complementary approaches. We analyze the occurrence ratio of annotation patterns (e.g., “all four analysts agree”) to grasp the global distribution of consensus across the total match duration. To further examine phase-specific characteristics, we also visualize the consensus levels for each individual play phase. This dual-metric approach allows us to distinguish between the overall dataset diversity and the inherent ambiguity within specific play phase.

Second, we introduce a pairwise confusion matrix to visualize the tendencies of analyst disagreement. To account for all perspectives, we calculate confusion matrices for every possible pair of the four analysts and aggregate the results. The matrix illustrates the probability that another analyst selected phase p_j given that a specific analyst selected phase p_i. This helps identify which specific phases are most prone to perceptual diversity.

Third, we evaluate inter-analyst agreement using Krippendorff’s α (Krippendorff, 2011). While simple agreement metrics like Cohen’s Kappa are commonly used, they are insufficient for this study as we intentionally incorporate diverse expert interpretations. By incorporating a semantic distance metric into the α calculation, we can distinguish whether the observed diversity stems from unreliable random labeling or from meaningful variations within semantically related play phases.

We define this semantic distance based on a three-level hierarchical structure of play phases: ball possession (Level 1), phase stability (Level 2), and specific tactical categories or intensity thresholds (Level 3). In this hierarchy, the 9 play phases are categorized into four high-level phases as follows: “Offensive” includes Build-up, Progression, and Final third; “Transition to Offense” consists of Counter-attack; “Defensive” comprises High press, Mid block, and Low block; and “Transition to Defense” includes Counter-press and Recovery. The distances are assigned according to the severity of the disagreement:

• Distance 0.00 (Agreement): Identical play phases.

• Distance 0.25 (Level 3 Error): Disagreements between sub-categories within the same high-level four phases (e.g., build up vs. progression). This also applies to disagreements involving the “No label” (ɛ) category, as these typically reflect subjective differences in intensity thresholds—whether a phase is distinct enough to be labeled—rather than a structural misunderstanding of the game.

• Distance 0.50 (Level 2 Error): Disagreements regarding phase stability (e.g., offensive vs. transition to offense) while maintaining the same possession context. This reflects the inherent temporal ambiguity in identifying the exact moment of a turnover.

• Distance 1.00 (Level 1 Error): Disagreements in ball possession (e.g., offensive vs. defensive). This represents a fundamental misinterpretation of the tactical situation and is thus assigned the maximum penalty.

The values 0.25, 0.50, and 1.00 were chosen to reflect an exponential scaling of disagreement severity. This ensures that fundamental structural errors (Level 1) are penalized significantly more than minor interpretative variations (Level 3), providing a more robust measure of tactical reliability than linear weighting. The specific distances between the four high-level phases are summarized in Table 2.

Quantitative Evaluation of Model Performance

To evaluate the necessity of the proposed spatial and temporal architectures, we compare our models against two reference baselines. The first is Mean Prediction, a non-learning baseline that consistently outputs the average probability distribution of the training set. This baseline represents the performance achievable by merely accounting for the dataset’s class imbalance. The second is an MLP, a simple deep learning baseline that processes input features as flattened vectors. To ensure a fair comparison, the MLP is configured to have approximately 140k parameters, consistent with the other models in this study. This baseline clarifies whether complex architectures, such as GNNs and Transformers, are necessary for capturing tactical patterns beyond simple mappings from coordinates to labels.

Our task is formulated as a regression problem to predict the probability distributions of play phases. Mean Absolute Error (MAE) is calculated by first averaging the absolute differences between the predicted probabilities and the ground-truth labels across all sequences in the test set for each play phase. The final MAE is then determined by taking the average of these values across all nine play phases. Similarly, Root Mean Square Error (RMSE) is computed by averaging the squared differences for each phase over all test sequences, taking the square root, and then calculating the mean across all phases. By applying this two-step averaging process, RMSE penalizes larger prediction discrepancies more heavily than MAE while providing a balanced assessment of performance across all tactical categories.

Furthermore, we evaluate the model using Top-1 Accuracy (Top-1 Acc) and Top-3 Accuracy (Top-3 Acc). Top-1 Acc represents the percentage of samples where the model’s most probable phase matches the ground-truth’s primary phase. Similarly, Top-3 Acc measures the frequency with which the ground-truth’s primary phase is included within the model’s three most probable phases. These accuracy metrics are calculated exclusively for the subset of samples where the highest probability in the ground truth is greater than or equal to 0.75, ensuring that evaluation is focused on instances with high analyst agreement. The Top-1 Accuracy is defined as follows:

All reported overall metrics are calculated as the macro-average of the values obtained for each play phase, ensuring that the model’s performance is evaluated equitably across all play phases regardless of their frequency in the dataset.

After comparing the aggregated metrics to identify the most effective architecture, we perform a detailed analysis of the results per play phase using the best-performing model. This granular evaluation allows us to investigate the specific predictive difficulty associated with each play phase.

Qualitative Evaluation

We conduct two types of qualitative evaluation to assess different aspects of the model’s performance.

First, to visually inspect how well the trained model replicates the ground truth, we use the sequences previously set aside for visualization. For these sequences, we visualize the probability distribution of play phases as represented by inter-analyst diversity (i.e., the ground-truth distribution) and compare it side by side with the probability distribution predicted by the trained deep learning model. This qualitative analysis focuses on a single team to closely examine the model’s ability to estimate the complex, ambiguous states defined by the analysts.

Second, to evaluate the model’s generalization performance, particularly its ability to adapt to Out-of-Distribution data different from our amateur training data, we use professional match data from La Liga acquired from Skillcorner as an external test set. The reliability of SkillCorner’s tracking technology is validated by its FIFA Basic certification under the Quality Programme for EPTS, ensuring professional-grade accuracy in broadcast-based data (May, 2024). We qualitatively evaluate the prediction results on approximately 30 seconds of data from the Real Madrid vs. FC Barcelona match (23/24 season, April 21, 2024).

Ablation Study

We conduct two ablation studies to verify the effectiveness of the components of our proposed methodology.

Probabilistic Labels vs. Deterministic Labels: This study validates the effectiveness of training on continuous probability distributions as defined in Eq.(9) rather than conventional discrete labels as defined in Eq. Error! Reference source not found.). We compare the performance of our approach against the same model architectures trained using deterministic labels derived from a majority vote as defined in Eq. (4).

Single-Team Learning vs. Dual-Team Learning: We investigate the impact of incorporating the opponent’s play phase as simultaneous contextual information. For this purpose, we compare our proposed model, which predicts a 9-dimensional label for a single target team, against an alternative model that simultaneously predicts 18-dimensional labels covering both teams as shown in Eq. Error! Reference source not found.). This comparison clarifies whether multi-team prediction improves the estimation accuracy of the target team’s tactical phases.