Selective Annotation of Few Data for Beat Tracking of Latin American Music Using Rhythmic Features

3.1 Rhythmic description

Different strategies have been proposed in the MIR literature to characterize rhythm and periodicity patterns within music. These can be used to infer tempo, meter, and small-scale deviations (e.g., swing) in an audio signal, but also serve as a preprocessing step in many tasks that depend on similarity, including genre classification and collection retrieval. The main intuition behind our approach is that rhythmic similarity should play a large part in determining good training candidates for beat tracking. For instance, within a specific style, if we would like to estimate beats for a single track, the best training candidate other than the track itself must be the one that is closest to it rhythmically.

We highlight two influential techniques for rhythmic description. The first one exploits the properties of the scale transform, a particular case of the Mellin transform, to achieve a descriptor that is robust to tempo variations (Holzapfel and Stylianou, 2009, 2011). The other, proposed by Pohle et al. (2009), is a tempo-sensitive descriptor based on band-wise amplitude modulations. Many variations of the latter can be found in the literature. We note that, depending on the tempo distribution of a music genre, it can be advantageous to use tempo-robust features or to encode large tempo changes in the representation when computing rhythmic similarity (Holzapfel et al., 2011).

3.2 Adaptive beat tracking

Much of the previous research on beat tracking with data-driven strategies has focused on developing “universal” models that are trained on large amounts of annotated data. Due to the nature of these state-of-the-art solutions, which typically depend on deep learning methods, high accuracy scores can usually be achieved given a sufficiently large pool of quality data annotations (Fiocchi et al., 2018; Jia et al., 2019; Böck and Davies, 2020). However, this good performance cannot be guaranteed when models are used to estimate beats from challenging or unseen music, e.g., music with highly expressive timing (Pinto and Davies, 2021) or from culturally specific traditions that were not present during training (Fuentes et al., 2019; Maia et al., 2022).

In recent years, there has been an increasing amount of literature on this real-world problem, i.e., when an end user wants to apply state-of-the-art models to a limited subset of examples with unseen rhythmic characteristics. Specifically, Pinto et al. (2021), Pinto and Davies (2021), and Yamamoto (2021) suggest human-in-the-loop approaches that enable a network to adapt to individual users and specific music pieces. These systems leverage the subjective nature of beat induction to allow a user to guide and improve the tracking process. We have investigated (Maia et al., 2022) the beat and downbeat tracking performances of a TCN model trained with small quantities of data from the same Latin American music traditions also used in this work. As we mentioned before, our previous results show that it is possible to train a network with modest annotation efforts, under the assumption that the dataset is homogeneous.

3.3 Active and few-shot learning

The present work is also related to two machine learning subareas. The first is the technique of few-shot learning (Koch et al., 2015; Vinyals et al., 2016; Snell et al., 2017), whose main idea is to train a model that is able to generalize to unseen classes at inference time by exploiting a few examples. Then, there is the concept of active learning, in which it is posited that a supervised learning algorithm performs better and with less data if it is allowed to choose its training samples (Settles, 2009). These instances are selected from the most informative ones of the unlabeled dataset and sent to an oracle, typically a human user, that annotates them and forms a labeled training set, which in turn is used to update the model. Both paradigms have been increasingly used in audio and music-related tasks, most notably sound event detection (Shuyang et al., 2017; Wang et al., 2022a; Kim and Pardo, 2018), drum transcription (Wang et al., 2020), musical source separation (Wang et al., 2022b), and music emotion recognition (Sarasúa et al., 2012).

The most commonly used active learning sampling methods are uncertainty sampling and query-by-committee. In particular, Holzapfel et al. (2012) explore this latter concept to create a committee of beat trackers that allows the determination of difficult-to-annotate examples. Other sampling methods take advantage of the internal structure of the input data distribution, either by analyzing local densities or by trying to construct a diverse labeled dataset. For example, Shuyang et al. (2017) use a k-medoids clustering technique to annotate and classify sound events. This algorithm attempts to minimize the distance of points in a cluster to a referential data point (medoid) using a custom dissimilarity measure.

Moving away from MIR, an influential work by Su et al. (2023) investigates the performance of several selective annotation methods as a first step before retrieving prompts for in-context learning of large language models. They include confidence-based selection as well as methods that promote representativeness, diversity, and both.

In the present work, we aim to select a small set of training examples that are informative to the beat tracking task, and thus can be used to train a model that provides good estimates for the remaining data. Therefore, we want to minimize the amount of data seen by the model, similar to few-shot approaches, but at training time. We build on the idea that informative samples can be extracted from the input data distribution and that the small annotation effort is better employed over these data.

4.1 Features

At the input of our system, we resample the audio tracks to 8000 Hz. A short-time Fourier transform (STFT) of the signal segmented by overlapping sequential 32-ms Hann windows is calculated to produce a 50-Hz frame rate spectrogram. Then, we map the frequency bins to a 40-band mel scale and take the logarithm to represent amplitudes in the dB scale. This mel spectrogram is the base representation from which all rhythm descriptors will be computed.

Scale transform magnitude (STM). To extract this tempo-robust descriptor we follow the original proposal by Holzapfel and Stylianou (2011). First, we compute a spectral flux from the mel-scaled spectrogram. This is possible via first-order differentiation and half-wave rectification of each mel band, followed by the aggregation of all bands. We detrend the resulting onset strength signal (OSS) by removing the local average. Then, we determine the autocorrelation of the OSS with a moving Hamming window of length 8 s and 0.5 s hop. Each frame is transformed into the scale domain by the direct scale transform (Williams and Zalubas, 2000) using an appropriate resolution. Even if preliminary tests with {100, 200, 300, 400} coefficients yielded comparable results on average, we limit our representation to the first 400 scale coefficients (up to scale C = 208) to allow for more complex musical periodicities. At the final step, we average this feature over time, achieving a dimension of 400 for each track.

Onset patterns (OP). The other feature we use to compare audio excerpts is our tempo-sensitive descriptor that draws from the work of Pohle et al. (2009). We will refer to these simply as “onset patterns”, noting that there are a few different implementations of this kind of feature in the literature, as mentioned in Section 3.1. To extract our onset patterns, we first subtract the moving average from each mel channel with a filter length of 0.25 s and half-wave rectify the result. This “unsharp mask” also has an effect of amplitude normalization, since the spectrogram is represented in dB (Seyerlehner et al., 2010). At the second stage, instead of computing modulations with the FFT and mapping the linear scale to log-frequencies (Pohle et al., 2009; Holzapfel et al., 2011), we compute a constant-Q transform (CQT) of the signal in each channel. Like previous work, we define a minimum modulation frequency of 0.5 Hz (30 BPM). Periodicities are described in 25 bins, at five bins per octave, up to 14 Hz. Similarly to Lidy and Rauber (2005); Panteli and Dixon (2016), we average the periodicities over all channels and take the mean feature across all time frames. This results in a descriptor with a dimension of 25.

4.2 Selection schemes

The present study borrows some selective annotation techniques from Su et al. (2023) and Shuyang et al. (2017).

Fast vote-k (VTK). The first selection technique we use is this graph-based selective annotation method, proposed by Su et al. (2023), which determines a set of simultaneously diverse and representative examples given the annotation budget. First, a directed graph G = (V,E) is created where each feature vector in $\mathcal{X}$ is a vertex in V. Edges E are defined from each vertex to its k nearest neighboring vertices in the embedding space, according to the cosine similarity. We start with $ℒ\text{=}\varnothing$ and $\mathcal{U}\text{=}\mathcal{X}$. Then, at every iteration, unlabeled vertices $u\in \mathcal{U}$ receive a score

where

with r > 1. The score $\varsigma (u)$ depends on the vertices v from which u can be reached. Each v contributes with its weight, s(v), which is small for v close to vertices already in $ℒ$. These two properties account for representativeness and diversity in the selected set, respectively. At every iteration, a vertex

is moved from $\mathcal{U}$ to $ℒ$, until $|ℒ|\hspace{0.17em}\text{=}M$. At the first iteration, the algorithm selects the most reachable vertex. We experimented with a few values for k Î {3,5,7,10,15,20,25,30}, but ended up choosing k = 5 as it provided good results across all budgets and datasets.

Diversity (DIV). Another selection technique we use in this work focuses on maximizing the diversity of the labeled set. Following Su et al. (2023), beginning with a random sample, at every iteration i ≤ M the furthest sample from those already in $ℒ$ is selected.

Maximum facility location (MFL). We also employ a representativeness selection based on an algorithm by Lin and Bilmes (2009) adapted for the facility location problem (Su et al., 2023). This greedy algorithm optimizes the representativeness of selected samples by measuring the pairwise cosine similarity between embeddings. At every iteration i ≤ M, it selects the most representative example u* as

where cos(∙,∙) is the cosine similarity function and r_j is the maximum similarity of x_j to the selected samples. At every step, r_j, which starts as –1∀j, is updated to $\max \{{\rho }_j,\cos (x_j,x_{u^{\ast }})\}$.

k-medoids (MED). We include a data selection scheme inspired by the work of Shuyang et al. (2017). We first cluster data with a k-medoids algorithm. Since the medoids returned by this clustering algorithm are the center points that represent local distributions and, at the same time, reside in distinct places of the feature space, we set k = M and directly use the medoids as the set of selected samples $ℒ$. As in the case of vote-k, this selection scheme aims to provide simultaneously diverse and representative examples for training.

Random (RND). All selection schemes are compared to a random baseline: a subset of size M is randomly selected from the dataset to make up $ℒ$.

By leveraging representativeness and diversity, all of the presented selection schemes (except random sampling) are indirectly conditioned by the input data distribution. Since the musical properties (e.g., global tempo, rhythmic patterns, complexity, density) pertinent to our task and features vary differently according to genre and performer, we expect that each distinct dataset might benefit from a different sampling method. In the case of a highly homogeneous dataset, it is perhaps better to annotate a set of more diverse examples. As the data distribution gets increasingly more heterogeneous, representativeness may be weighted more. Moreover, if the dataset is unimodal and highly homogeneous — e.g., composed of a single music genre and displaying little variance in its rhythmic properties —, we would expect to observe little improvement in using smart selection schemes over training on randomly selected data. For less homogeneous (still unimodal) datasets, a proper smart selection should be able to systematically provide better training examples for beat tracking. Finally, in a dataset containing different genres with particular characteristics: (1) A single selection scheme might not be effective for all genres; (2) On average, random sampling will select more examples from the most populated genres, possibly overlooking the less populated ones. In contrast, when genres have the same number of tracks, due to (1) we should not expect large improvements in employing data selection methods.

4.3 Beat tracking model

The main model used throughout our experiments is the TCN-based multi-task model of Böck and Davies (2020), provided in an open-source implementation (Davies et al., 2021). As per the multi-task formulation, it has been shown by Böck and Davies (2020) that it improves the individual estimation of beats, downbeats, and tempo. Since our target is to select good training samples for beat tracking, we ignore both downbeat and tempo estimation heads and consider only beat likelihoods produced by the network. We infer final beat positions with a dynamic Bayesian network (DBN): DBNBeatTracker, from the madmom (v0.17.dev0) package (Böck et al., 2016).

4.4 Datasets

We use three sets of audio tracks to evaluate our methodology under different conditions. Two of them are associated with distinctively percussive Afro-rooted Latin American music traditions that motivate the main problem of this article. The first one is the Candombe dataset (Nunes et al., 2015; Rocamora et al., 2015), which contains 35 recordings of candombe drumming (2.5 h in duration) featuring three to five drummers performing different configurations of the three candombe drums (piano, chico, and repique). The second dataset comprises the ensemble recordings from the BRID dataset (Maia et al., 2018): 93 tracks of about 30 s each featuring two to four Brazilian percussionists playing characteristic rhythm patterns of mostly samba, partido-alto, and samba de enredo. In total, 10 instrument classes are arranged in typical ensembles of Brazilian music — samba in particular. We note that both datasets were collected with the consent of the musicians. Furthermore, we also explored the Ballroom dataset (Gouyon et al., 2006; Krebs et al., 2013), a standard of the beat tracking literature. Unlike BRID and Candombe, it consists of many distinct genres and was selected to serve as a counterpoint in our investigation. It includes 698 tracks of eight ballroom dance genres (cha-cha-cha, jive, quickstep, rumba, samba, tango, Viennese waltz, waltz) of 31 s on average. To allow a direct comparison of results across all datasets, candombe tracks were segmented into 276 non-overlapping 30-second excerpts. All genre labels were ignored in our experiments.

A deeper analysis of the characteristics of each dataset is presented in Section 5.1.

4.5 Evaluation

Using the implementation provided in the madmom package, we evaluate the beat prediction results through the F-measure (Dixon, 2007), which is the harmonic mean between the proportion of correct estimates (precision) and the proportion of correctly estimated beats (recall), with the usual tolerance of ±70 ms around annotations.

5.1 Dataset homogeneity

Preceding our experiments, we investigate tempo and rhythmic variability of tracks from each dataset.

Figure 2 presents the datasets’ global tempo distributions smoothed by a Gaussian kernel density estimation technique. Candombe exhibits a slim distribution, averaging 132 BPM (8 BPM standard deviation), while BRID is approximately bimodal, whose peaks at 95 and 130 BPM can be respectively associated with samba/partido-alto and samba de enredo subgenres. Unsurprisingly, Ballroom’s multi-genre characteristic is disclosed by multiple modes; individual distributions are described by Krebs et al. (2013).

Figure 2

A representation of the rhythmic patterns across all datasets is displayed in Figure 3. The scale transform magnitudes (STM) were obtained from each track following the procedure described in Section 4.1. Then, manifold learning with UMAP (McInnes et al., 2018) was used to reduce the feature space dimension from 400 to 2 using the cosine distance as a metric. A major advantage of UMAP over other commonly used dimensionality reduction approaches like t-SNE is that it can better represent global data structure while preserving local neighborhoods (McInnes et al., 2020). UMAP diagnoses that Ballroom patterns mostly lie in regions whose local dimension is estimated as high. That means they are less accurately represented in this embedding, and thus display greater rhythmic variation than can be represented in two dimensions. Candombe has a small set of outliers but is mostly represented in a compact structure, whereas BRID, despite having fewer examples, is more spread out in the embedding space. Interestingly, the subset of Ballroom located near BRID and Candombe is mostly composed of tracks from the “Samba” genre, with few examples of “Jive”.

Figure 3

5.2 State-of-the-art results without selection

To contextualize the outcomes of our experiments, we present in Table 1 the beat tracking performances on BRID and Candombe of models using the architecture of Böck and Davies (2020) under three different training schemes:

• “Pre-trained”: results of the TCN model from our former work (Maia et al., 2022) — network trained on 38 h of Western music material from six datasets (including Ballroom), and tested on the entire BRID and Candombe datasets;

• “Fine-tuned”: the “pre-trained” model that we fine-tuned for each dataset with 3 min of randomly selected data (tracks were split in half for training and validation), tested on the remaining data. We used 10 random selections, 30 training seeds, and the fine-tuning parameters of Maia et al. (2022);

• “Trained from scratch”: the TCN model initialized randomly and trained for each dataset on full 30-second tracks, using an eight-fold cross-validation scheme. One fold was used for testing, one for validation, and six for training. The training was repeated until all folds had been used for testing. For training parameters see Section 4.3.

5.3 Experiment 1: Does sampling matter?

In this first experiment, we assess the dependence of beat tracking performance on random training sets in low-data scenarios. Depending on the annotation budget, it is often not feasible to explore all possible training sets. By choosing to focus on the BRID dataset, which has the smallest number of tracks of all datasets, we are able to survey a larger proportion of all random combinations. In this sense, we set the annotation budget to M = 4 tracks, which yields around 3 million possible combinations of four distinct elements out of 93 total tracks. Then, we select 1000 of these combinations, ensuring that all tracks in the dataset are about equally represented overall. We use each unique combination to train/validate the TCN model, which we evaluate over the complementary test set of 89 files. We repeat each training 30 times with different randomly initialized weights and seeds.

Figure 4 shows the averages and standard deviations for the performances of all trained models in ascending order of mean beat F-measure. We note that mean beat F-measures range from 46.5% to 90.1% depending on the training set, with the 5th and 95th percentiles corresponding to 61.0% and 85.4%. A mean F-measure of 74.4% is achieved on average.

Figure 4

This experiment shows the importance of adequate data selection for the TCN model when dealing with low-data training scenarios. With an annotation budget of M = 4 tracks, we observe a considerable improvement of about 16 percent points over the average case and 44 percent points over the worst case when estimating beats on the BRID dataset.

5.4 Experiment 2: Feature structure

In this second experiment, we investigate the local structure of the feature space generated by each rhythm description feature (STM and OP) and its capability of conveying meaning for the data selection scheme. For this purpose, considering the distribution of a dataset in the feature space, we analyze the performance of the TCN model when trained with points sampled from different regions around single test tracks. We hypothesize that regions closer to the test sample provide better training examples, thus yielding better beat tracking results. Again we set the annotation budget to M = 4, but this time we observe all datasets separately.

The regions are limited by concentric hyperspheres centered at each test sample, whose radii depend on the distribution of the dataset in the feature space. If Q₁, Q₂, and Q₃ are the first, second, and third quartiles of the pairwise feature distances of points in the dataset, respectively, we define ${ℛ}_1^j$, ${ℛ}_2^j$, and ${ℛ}_3^j$, the regions in increasing distance from a given test file, x_j, as

where i ≠ j. We also define the set of remaining points, which lie outside the largest hypersphere, as

Figure 5 exemplifies the computation of these regions from a normal data distribution in a two-dimensional space using the Euclidean distance. In practice, we use the cosine distance, i.e.,

Figure 5

Once all regions are determined for a reference track, x_j, we can compare how well models trained and validated on sets of randomly selected points from each region perform on the test set $\mathcal{U}\text{=}\left\{x_j\right\}$. We repeat this process for all points in the dataset that contain at least M = 4 examples in each of its regions. This way, we end up training four different models per reference sample. Once again, we repeat the training process 30 times, with different seeds, keeping the same training sets.

The results of this experiment are presented in Figure 6. We show the average F-measure gain $\Delta \text{F}({ℛ}_i)$ across all models when using points from each region $({ℛ}_1,{ℛ}_2,{ℛ}_3)$ over using points from the farthest region $({ℛ}_4)$. Mathematically, if $\text{F}({ℛ}_i^j)$ is the F-measure of a model trained on points from the region ${ℛ}_i^j$ around sample $x_j,j\in \{1,\hspace{0.17em}\dots ,\hspace{0.17em}L\}$, then

Figure 6

where $\Delta \text{F}({ℛ}_i^j)\text{=F}({ℛ}_i^j)–\text{F}({ℛ}_4^j)$, for i Î {1,2,3}. For all datasets, we observe that the best models are trained on points from the closest regions $({ℛ}_1)$, independently of the feature. We may also compare the gain from using ${ℛ}_1$ over ${ℛ}_2$, for example. Using a set of immediate neighbors leads to significant gains in BRID, as shown by the substantial difference between results in the two regions. In Ballroom this gain is much smaller, and in Candombe it is almost negligible. It is worth noting that for Candombe, the absolute beat F-measure values are 91.7% and 96.4% for STM and OP, respectively, in ${ℛ}_2$. These compare to 62.6% and 65.3%, respectively, for the same region in BRID. This means that there is more room for improvement in BRID than in Candombe. Without forgetting that the definition of these regions depends on the data distribution of each dataset, we can say that data selection must be more important in the former than in the latter.

We have shown with this experiment that one can train beat tracking models that are better able to generalize to recordings in local neighborhoods defined in the space of the rhythmic features. This is a promising result that suggests that these features can be employed to retrieve informative samples.

5.5 Experiment 3: Sampling strategies

In this experiment, we examine all combinations of rhythm description and sample selection techniques across different annotation budgets; and evaluate the beat tracking performance of a TCN model trained on the selected data and tested on the remainder of each dataset. This experiment represents the use case of our proposal. As we mentioned before, the main difference to the real-world scenario is that we use ground-truth annotations instead of asking for a human to provide labels for $ℒ$. We wish to investigate how much a sampling strategy can improve tracking performance against random sampling. Naturally, this depends on the properties (e.g., tempo and rhythm pattern distributions) of each dataset as well as on the specified annotation budget.

For BRID and Candombe, we vary the annotation budget from 4 to 14 samples (in steps of 2), i.e., about 2–7 min of annotations. Since Ballroom has many more tracks (about 7.5 and 2.5 times more than BRID and Candombe, respectively) and genres, we use larger budgets for this dataset, M = {10,16,22,28,34,40} (~5–20 min), i.e., around 2.5 times more data. As in all experiments, training files are split in half for training and validation purposes. Regarding the selective sampling techniques, we observe that MFL and VTK are deterministic and as such always provide the same labeling sets. DIV and MED depend on a random initialization. However, we noticed that a considerable number of files (usually M–2 or M–1) were repeatedly selected over multiple executions of the DIV sampling process, which means that, especially for larger budgets, it is nearly deterministic. In the case of MED, smart initialization and multiple runs were used to obtain a more robust clustering. Once these sampling techniques have provided consistent results, we use them to select one set of training examples for each combination of dataset, labeling budget, and feature representation. Finally, for the random baselines (RND), 10 random selections of M files were carried through for each dataset–budget pair; these are used to indicate the expected performance of the TCN model. For each data selection, we repeat the training process 30 times with different seeds.

Beat tracking performance (means and standard deviations) is summarized in Table 2. Looking at these results, we notice that, in most cases, selective sampling techniques are consistently better than random sampling across all budgets, although the best feature–selection pair greatly depends on the dataset and training size. In particular, STM+MFL stands out as the best setup for Ballroom, closely followed by STM+MED and OP+MED, showing gains of up to 5.2 percentage points (M = 16) over the random baseline. In extremely low-data scenarios, OP+MED produces the best results for BRID, with an 18.3 point increase over random at the smallest budget, although STM+MED and OP+MFL provide good results as well. In both Ballroom and BRID, diversity sampling gives worse results than random for STM (–4 percentage points on average). DIV is also worse than RND with OP in Ballroom (almost –5 points on average), and inconsistent in BRID when paired with the same feature representation. Finally, the RND baseline performance in Candombe is already very high (94.0–96.8%), which leaves little room for improvement in this case. However, except for the two smallest budgets, OP+DIV provides moderate gains for this dataset (2.3 points on average).

The general conclusion is that using sampling techniques can provide better training examples for the TCN model, since most feature–selection setups are shown to outperform the random baseline. This performance gain is typically larger the smaller the annotation budget. We also note that a smart data selection can reduce the standard deviation of the results, leading to more stable solutions than those obtained through random selection. Unsurprisingly, since there are many possible dataset configurations (e.g., highly homogeneous, highly heterogeneous), there is no optimal setup. We discuss a set of recommendations based on our results in the following.