PBSCR: The Piano Bootleg Score Composer Recognition Dataset

2.1 Previous work

In this subsection we provide a brief overview of previous methods in composer recognition, and we describe how recent methods motivate a need for larger datasets. Previous work can be divided into two time periods: classical machine learning and deep learning.

Classical machine learning methods were the dominant approach between roughly 2000 and 2015. Most methods for composer classification from this era fall into one of two categories. The first category involves defining a set of manually designed features and then feeding the features into a standard classifier. Some examples of features include absolute and relative values of pitch and duration (Pape et al., 2008), global statistics on pitch intervals and durations (Goienetxea et al., 2018), chroma-based features (Anan et al., 2012), n-gram statistics on intervals and durations (Hajj et al., 2018), high-level musicological features like detecting 9–8 suspensions (Brinkman et al., 2016) or detecting sonata form (Kempfert and Wong, 2020), and using standardized feature sets (Herremans et al., 2016) such as the jSymbolic toolbox (McKay and Fujinaga, 2006). The second category of classical machine learning approaches involves training a sequence-based model. The two most common sequence-based models from this era are N-gram models (e.g., Hontanilla et al., 2013) and Markov chains (e.g., Hedges et al., 2014). In this approach, a sequence-based model is trained on each composer of interest, and test sequences are classified by selecting the model that has the highest likelihood.

Deep learning–based approaches have increasingly become the dominant paradigm since around 2015. Most methods in this era fall into one of two categories.

The first category represents the underlying music information as a continuous signal. Common representations include mel spectrogram and MFCC features for audio (e.g., Micchi, 2018; Kher, 2022); a piano roll–like matrix or a tensor specifying note events for symbolic music (e.g., Verma and Thickstun, 2019; Velardeet al., 2018); and 2-dimensional images for sheet music (Walwadkar et al., 2022). Given an input represented as a continuous signal, various neural network architectures have been explored, including Convolutional Neural Network (CNN) architectures (e.g., Deepaisarn et al., 2022; Kim et al., 2020; Walwadkar et al., 2022)), Convolutional Recurrent Neural Networks (CRNNs) (Kong et al., 2020), and Long Short-Term Memory (LSTM) models (Kher, 2022; Micchi, 2018).

The second category represents the underlying music information as a sequence of discrete tokens. Some methods of forming discrete tokens from music data include: converting piano roll–like data into binary text (Takamoto et al., 2018) or sequence of characters (Yang and Tsai, 2021a), considering (note, duration) tuples (Deepaisarn et al., 2023), or using a REMI (Huang and Yang, 2020) or compound word representation (Hsiao et al., 2021). Once the data is represented as a sequence of discrete tokens, a variety of Transformer architectures have been used to model the data (Chou et al., 2021; Li et al., 2023; Tsai and Ji, 2020; Yang et al., 2021). One benefit of this approach is the ability to pretrain models on unlabeled data in a self-supervised manner. These methods are data hungry and provide a strong incentive to construct benchmarks that contain large amounts of data.

2.2 Previous datasets

In this section we describe the landscape of datasets used to study the composer classification task. This landscape provides historical context for understanding the contributions of the PBSCR dataset.

Table 1 provides an overview of recent works on composer classification and the datasets used in these studies. From left to right, the columns indicate the paper (author and year published), number of composers in the classification task, original source data type (symbolic, audio, sheet music), preprocessed data format (i.e., after any data preprocessing), and dataset size. For dataset size, numbers in parentheses indicate unlabeled data for pretraining. Entries in the table have been arranged according to publication year, and the last entry in the table corresponds to the proposed PBSCR dataset.

There are three things to notice about the landscape of previous datasets described in Table 1. First, most previous works consider a small number of composers. For example, only 6 out of the 32 previous works shown in Table 1 consider more than 10 composers. Second, most previous works consider a relatively small amount of data, by modern standards. It is difficult to compare dataset sizes directly since previous works report sizes in different ways, including number of movements/pieces, total audio duration, and number of sheet music images. Nonetheless, by comparing entries by the number of pieces in the labeled dataset (the most common metric), we can see that most works consider on the order of hundreds of pieces. Third, the vast majority of previous works focuses on symbolic music formats. As a practical matter,the choice to use symbolic music as source data limits the size and the diversity of datasets, given that symbolic music data is less plentiful than sheet music or audio.

It is useful to point out how the PBSCR dataset fits into this data landscape. It is distinctive in three ways. First, it has the highest number of composers (100, tied with Kong et al., 2020). As mentioned above, this is much higher than most previous works, so it poses a more challenging classification task. Second, the PBSCR dataset is among the largest in size. In particular, it is one of the only datasets in Table 1 that comes with a large unlabeled dataset for pretraining. Given the shift in recent years toward pretraining models in a self-supervised manner, this unlabeled data provides an essential resource for supporting the development of competitive models. Based on the number of works in both the labeled (4997) and the unlabeled (29,310) datasets, the PBSCR dataset is almost certainly the largest in terms of total dataset size. Third, it is one of only a few works (Tsai and Ji, 2020; Walwadkar et al., 2022; Yang and Tsai, 2021a) that uses sheet music images as source data. By using a bootleg score representation, the PBSCR dataset maintains the advantage of plentiful sheet music data (on IMSLP) while presenting the data in an extremely compact and simple form (binary 2-dimensional images).

Given this context, we can reasonably make the following claim: the PBSCR dataset presents the most challenging classification task (based on the number ofcomposer classes), has the largest and most diverse set of data available (based on the number of pieces and composers), and has the simplest and most accessible data format (2-dimensional binary images).

It is useful to note that the PBSCR dataset has a very different philosophy than that of most previous works in composer recognition. Whereas previous approaches require full symbolic music information and accept the consequence of limited data size and diversity, the PBSCR dataset requires that the dataset be large, open, diverse, and easy to work with and, accepts the consequence of a noisy, selective feature representation. By using the bootleg score representation, we construct a dataset that is as easy to work with as MNIST data and can facilitate rapid exploration and iteration.

3.1 IMSLP piano bootleg scores v1.1

The IMSLP piano bootleg scores repository (v1.0) was introduced in Yang and Tsai (2020) for a sheet musicidentification task and was first used for composer classification in Tsai and Ji (2020). This repository contains the bootleg scores for all solo piano works in IMSLP. Below, we describe the history of its construction as well as the steps that were taken to clean up the data (v1.1).

The first step in creating this repository was to download IMSLP sheet music. The IMSLP website provides a list of composers, a list of works for each composer, and a webpage for each work that contains audio recordings, sheet music, and metadata. For each composer, we iterated through all of their works and downloaded all PDF sheet music scores and associated metadata. The scraping and downloading took more than a month to complete and resulted in a set of 420,271 PDF files from 164,248 composers that was 1.2 terabytes in size.

The second step was to filter the full dataset by instrumentation tag label in order to compile a list of solo piano works. After filtering, the dataset contained 29,310 works, 31,384 PDFs, and 374,758 individual pages. Note that a work may contain several PDF versions on the IMSLP website. All of the remaining steps were applied only to this filtered dataset.

The third step was to convert each PDF into a sequence of PNG images. We performed the decoding at 300 dpi and then resized the image to have a fixed width of 2550 pixels while preserving the aspect ratio. This resizing step was necessary to appropriately handle the extremely large range of image sizes in IMSLP.

The fourth step was to compute the bootleg score representation from each PNG image (i.e., page of sheet music). The bootleg score (Tsai et al., 2020; Yang et al., 2019) is a mid-level feature representation that encodes the position of filled noteheads relative to staff lines in the sheet music while ignoring many other aspects of thesheet music such as note duration, accidentals, rests, time signatures, clef and octave markings, and non-filled noteheads. The feature extraction process uses classical computer vision techniques to detect notehead and staff line locations, so it is a noisy estimation that contains errors. More details can be found in Yang et al. (2019). Figure 1 shows two examples of a piano sheet music excerpt and its corresponding bootleg score. Note that staff lines are not encoded in the bootleg score representation itself but have been overlaid in Figure 1 as a visual aid. The bootleg score for each page of sheet music is a binary matrix, where 62 indicates the total number of different staff line positions in both the left- and the right-hand staves and where indicates the number of detected simultaneous notehead events on the page. We will refer to each column of the bootleg score as a bootleg score event. So, for example, the bottom example in Figure 1 shows a bootleg score fragment with events (columns), where the first bootleg score event is a 62-length array containing 60 zeros and two “1“ entries corresponding to the noteheads at D4 and A2.

Figure 1

It is worth mentioning a few practical details at this point. First, each 62-bit bootleg score column is encoded as a single 64-bit integer so that the bootleg scores are compactly represented as a list of integers. Each page of sheet music is thus reduced to a list of 64-bit integers that compactly encode the bootleg score events on the page. Since a PDF consists of multiple pages, we store the features for each page in a separate list to keep track of which page each feature comes from. This representation makes it possible to store an inconveniently large ( in PNG format) dataset very compactly in memory (0.5 GB). Second, the dataset is structured as a file hierarchy that is separated first by composer and then by work, which ensures a clean separation among different partitions. Third, the resulting repository after the fourth step above is the IMSLP piano bootleg scores data v1.0, which was originally presented in Yang and Tsai (2020). The fifth and sixth steps (below) describe the improvements in the newly released v1.1 dataset.

The fifth step is to filter out non-music pages from the bootleg score repository. One of the problems with the original repository is that many PNG images arenot sheet music—they may be title pages, blank pages, forewords, a table of contents, etc. In the original v1.0 repository, a bootleg score was computed for every single PNG image without any consideration of the contents in the image. This results in a non-trivial amount of gibberish bootleg score data that has been extracted fromnon-music images. Figure 2 shows some examples of non-music filler pages (left) and their corresponding gibberish bootleg scores (right). In order to identify non-music pages, we trained a Transformer-based model to identify gibberish bootleg scores. The process for training this model is described in detail in Section 3.2. In the revised v1.1 IMSLP piano bootleg scores repository, the bootleg scores for (predicted) non-music pages have been identified and removed.

Figure 2

The sixth step is to scrape, organize, and include metadata that is available on IMSLP for each work in the unlabeled dataset. This process is described in Section 3.3, and this metadata is included in the v1.1 IMSLP piano bootleg scores repository to facilitate multimodal research and to allow for convenient linking to other datasets.

3.2 Identifying non-music pages

In this subsection, we describe the process of identifying non-music pages by training a Transformer-based model on bootleg score fragments.

The first step is to label a set of music and non-music pages. This was done in the following manner. First, we took the original 9-class dataset proposed in Tsai and Ji (2020) in which each page had been manually labeled as music or non-music. We manually relabeled these pages as one of three categories: music, non-music, or mixture. The “mixture” category contains pages that have both sheet music and text, as is often seen in a table of contents (e.g., showing excerpts of pieces) or foreword. We ultimately decided to exclude the mixture pages and include only pure music and pure non-music pages for training our classifier. In total, there were 5938 music pages and 259 non-music pages. We divided these pages into training and validation partitions using a 60–40 split.

The second step is to sample bootleg score fragments. Figure 3 shows a histogram of the number of bootleg score events in music pages (top) and nonmusic pages (bottom). We can see that many filler (non-music) pages have a very small number of bootleg score events, so our classifier will need to handle short bootleg score fragments. Accordingly, we decided to train our model on bootleg score fragments of length 16. We densely sampled bootleg score fragments from the non-music pages by sampling 16-length fragments with overlap. This resulted in a total of 2799 non-music bootleg score fragments (1689 train, 1110 validation). To maintain a balanced dataset, we randomly sampled the same number of fragments from the music pages. This sampling was done by randomly sampling a work (PDF) from thetrain/validation partition, randomly sampling a music page from the PDF, and then randomly sampling a length-16 fragment from the page’s bootleg score. At the end of this step, we have a labeled dataset of 3378 training bootleg score fragments (1689 filler, 1689 non-filler) and 2220 validation bootleg score fragments (1110 filler, 1110 non-filler).

Figure 3

The third step is to train a music vs. non-music fragment classifier. We adopted a similar approach as in Tsai and Ji (2020), which we describe here for completeness. We encode each 62-bit bootleg score column as a sequence of eight 8-bit characters and learn a subword vocabulary using Byte Pair Encoding (Gage, 1994). Using this BPE tokenizer, we pretrain a GPT-2 language model on the entirety of the IMSLP piano bootleg scores repository (v1.0). It is worth noting that this pretrained language model was originally used for composer classification and here we simply fine-tuned it on a different downstream task. Next, we add a classification head with two output classes (music vs. non-music) and fine-tune it on the labeled dataset of music and non-music fragments. In this way, our classifier is trained to classify 16-length bootleg score fragments as music or non-music.

We apply our classifier model to full pages in the following manner: we first extract a bootleg score representation from the page. If the resulting bootleg score has a length smaller than 64, it is automatically classified as non-music. (Note from Figure 3 that very few music pages have bootleg score lengths smaller than 64.) Otherwise, fragments of length 16 are densely sampled from the bootleg score with overlap, and each fragment is passed through our classifier model. We average the outputs of each fragment prediction to get an ensemble prediction for the entire page.

Figure 4 shows a histogram of predicted probabilities on the validation pages, where a higher probability corresponds to a non-music page. We can see that there is a fairly clean separation between the music and the non-music data. We set a very conservative threshold of 0.5, which ensures that non-music pages will be excluded from the data with high confidence (and sometimes music data will be excluded as well, which we are okay with). With this threshold value, we achieve a precision of 0.85 and a recall of 0.98 on the validation pages. Because we care more about ensuring that non-music pages are excluded, the recall of 0.98 is the more important metric. We use this ensemble classifier to identify and remove non-music pages from the IMSLP piano bootleg score repository.

Figure 4

3.3 Adding metadata from IMSLP

In addition to cleaning up the IMSLP piano bootleg scores repository, we also collected and added metadata for these works. This process is described next.

The metadata is scraped from the IMSLP website. For each work, IMSLP has a webpage that contains links to audio performances and various sheet music editions. The webpage also contains a set of metadata for each composition that may include attributes such as the work’s title, composer, opus/catalogue number, key, year/date of composition, composer time period, instrumentation, movements/sections, alternative title, dedication, first publication, etc. We scraped the compositionwebpages to extract these metadata attributes and stored the metadata in a file on our github repository.

This metadata is valuable for two reasons. First, it provides a rich set of information that could be used to study many tasks besides composer classification. Second, it allows for convenient linking to other datasets. As one concrete example, we used simple string matching based on composer name and work title attributes to link the bootleg scores in the PBSCR dataset to corresponding files in the GiantMIDI-Piano dataset (Kong et al., 2020). A file containing 7413 matches is included in our github repository. The provided metadata can similarly be used to link the PBSCR dataset to other datasets.

4.1 100-Class labeled data

The preparation of the 100-class labeled dataset consists of four steps, which are described below. At a high level, it consists of 100,000 bootleg score fragments (70k train, 15k validation, 15k test) that are balanced across 100 different classical composers.

The first step is to identify a list of 100 composers to include. These 100 composers are selected as a subset from the IMSLP Piano Bootleg Scores Data described in Section 3.1. We first ranked all composers by the total amount of bootleg score data they have available on IMSLP. We then manually reviewed the ordered list of composers and selected the top 100, being sure to remove those who are not primarily composers (e.g., some people on the list were primarily arrangers and editors). The full list of 100 composers can be found at https://github.com/HMC-MIR/PBSCR/blob/main/100_class_list.txt.

The second step is to select a set of sheet music PDFs for each composer. Each work in IMSLP may have multiple PDFs associated with it that correspond to different publishers or editions. Because popular works tend to have a large number of sheet music versions, we select one representative PDF per work in order to avoid over-representing a small number of works. In order to maximize the amount of data available to us, we simply selected the PDF that had the highest number of total bootleg score events. Figure 5 shows the total number of works available for these top 100 composers (top), along with the total number of bootleg score events extracted from each composer’s piano sheet music (bottom). In total, there are 4997 works and 70,440 sheet music pages.

Figure 5

The third step is to identify non-music pages in the selected set of PDFs. We used a Transformer-based model to identify filler pages, as described in Section 3.2. In the 100-class labeled data, there are a predicted 64,129 (out of 70,440) pages with music content containing 12.1 million bootleg score events.

The fourth step is to sample bootleg score fragments from each composer. This sampling serves two purposes: it allows us to achieve class balance among the fragments even though the number of works per composer is different, and it standardizes the size of each labeled sample () in order to achieve the data simplicity of MNIST images. This sampling is done in the following manner. First, we divide the works into training, validation, and test sets, using a split of , and , respectively. Next, we decided on the total number of bootleg score fragments to sample from each partition. For the 100-class data, we have 70,000 train fragments, 15,000 validation fragments, and 15,000 test fragments, resulting in a total of 100,000 examples. Based on these numbers, we calculated how many fragments per composer need to be sampled in order to achieve class balance. Each fragment is drawn by randomly selecting a work by a given composer and then by randomly selecting a 64-length fragment from the bootleg score. Our sampling process guarantees that our classes are perfectly balanced, and it gives equal weight to all piano works (in IMSLP) that a composer has composed.

Figure 6 shows a set of example bootleg score images for nine selected composers (those in the 9-class dataset). The staff lines for the left hand and the right hand are not present in the bootleg score representation itself, but they have been overlaid for ease of reference. Even without any information about note durations, key or time signature, or accidentals, one can immediately see some recognizable features: the Bach example has fugue-like texture and movement. The Beethoven example has an alternating octave in the right hand, which is not common in Bach’s music. The Mozart example has scale-like runs in the right hand, with an Alberti bass-like left-hand accompaniment. The Classical and Baroque composers (Bach, Mozart, Haydn) have thinner textures compared to those of the Romantic-era composers. These examples show that even with the minimal bootleg score representation, many aspects of compositional style are preserved.

Figure 6

The labeled 100-class dataset is formatted in a way that resembles the MNIST dataset. Each dataset consists of the following six arrays:

• : a binary tensor specifying the training bootleg score fragments

• : a 70,000 length array specifying the train composer class indices

• : a binary tensor specifying the validation bootleg score fragments

• : a 15,000 length array specifying the validation composer class indices

• : binary tensor specifying the test bootleg score fragments

• : a 15,000 length array specifying the test composer class indices

In addition, we also provide relevant metadata on all train, validation, and test fragments. This metadata includes a unique identifier that specifies the PDF from IMSLP from which the fragment was taken as well as the page and offset in the bootleg score from which the fragment was sampled. This information allows researchers to access the complete, unabridged bootleg scores to study the effect of longer-term structure or to access the original, raw sheet music image data for visualization and deeper understanding.

The 100-class dataset poses a much more challenging classification task than does the 9-class recognition task in Tsai and Ji (2020). One may notice that the shape and the size of the data is similar to MNIST, in keeping with our principle of constructing a dataset that is “as accessible as MNIST and as challenging as ImageNet.” Each composer has only 700 training examples, so the task is difficult both for the large number of composers and for the relative scarcity of labeled data. For these reasons, we believe this dataset will push the boundaries of composer recognition to a higher level.

4.2 9-Class labeled data

The 9-class labeled dataset contains bootleg score fragments for nine classical composers: Bach, Beethoven, Chopin, Haydn, Liszt, Mozart, Schubert, Schumann, and Scriabin. The dataset is constructed in the same way as the 100-class labeled dataset but with one difference: the list of composers was adopted from Tsai and Ji (2020) (rather than selected based on data availability) to enable historical comparisons. In total, the 9-class dataset consists of 28,000 train fragments, 6000 validation fragments, and 6000 test fragments, resulting in a total of 40,000 examples that are balanced across composers. Table 2 shows the number of works, the total number of pages, the number of (predicted) music pages, and the number of bootleg score events for each composer in the 9-class labeled dataset. There are 896 PDFs, 10,305 pages with music content, and 2.2 million bootleg score events. The purpose of providing both 100-class and 9-class datasets is to enable experimentation at varying levels of task difficulty.

4.3 Data leakage

In this section we discuss known data leakage issues with the 9-class and the 100-class labeled datasets.

It is impossible to get a perfectly clean train/test split with IMSLP data for many reasons: composers often re-write pieces or re-use themes in later works (and each may be listed with a separate opus number); composers sometimes create alternate versions of pieces; some composers do transcriptions or arrangements of other composers’ works (which makes the ground truth label ambiguous); some works are partially composed by the composer and completed by others after the composer’s death; and some works have uncertain or incorrect authorship (e.g., the “Valse Melancolique” in F-sharp minor was incorrectly attributed to Chopin and listed among his works).

We discovered an additional source of data leakage late in the review process: idiosyncrasies in IMSLP’s organization. Specifically, there are instances in which one piece appears on two different IMSLP webpages: once as an individual composition and once as part of a collection (e.g., individual preludes and fugues in Bach’s The Well-Tempered Clavier). To quantify how often this happens, we manually checked all works in the 9-class dataset and found that 11 of the 896 works were collections exhibiting this issue (9 Bach, 1 Scriabin, 1 Liszt). Of these 11 collections, we noticed that the IMSLP webpages for 8 of them had a warning dialog box at the top of the webpage that indicated the possibility of duplicate entries. The 100-class dataset was too large to check manually, so we used an automated approach to iterate through all of the 4997 works and detect the warning dialog boxmentioned above. We found only two more collections with this issue (1 Mendelssohn, 1 Handel). Thus, we found 13 works/collections across five composers in the 100-class dataset that exhibit this issue. This automated approach most likely failed to catch some additional instances, but it provides a ballpark estimate of how common this phenomenon is. A list of these collections can be found on our github repository.

The above sources of data leakage mean that our reported accuracy numbers are likely inflated. However, as long as other researchers use the same dataset for comparison, the benchmark can serve its purpose of tracking progress in the composer classification task. Also, given how low the accuracies are for the 100-way classification (e.g., the best top-1 accuracy is in Table 4), the accuracy inflation due to train/test leakage still leaves an enormous amount of room for improvement and is likely only a minor factor in overall performance.

5.1 Supervised composer recognition

The most obvious task is a supervised composer recognition task. Here, the goal is to classify a bootleg score fragment according to its composer class. In addition to a labeled set of training pairs, unlabeled data from the IMSLP Piano Bootleg Scores v1.1 dataset (Section 3.1) is available for pretraining.

There are several metrics of performance that might be appropriate in this scenario. Following the convention in ImageNet, one useful metric of performance is top- accuracy, which indicates the percentage of queries that have the correct composer in the highest-ranked composers. Another useful metric is mean reciprocal rank (MRR), which is calculated as where indicates the rank of the true composer. MRR ranges between 0 and 1, and a higher MRR is better. MRR offers more nuanced information about the rank of the true composer than does the strict binary threshold of a top- accuracy metric. We recommend reporting results with several of these metrics since the most appropriate metric may depend on the difficulty of the task.

Tables 3 and 4 show the performance of three different baseline systems on the 9-class and 100-class recognition tasks, respectively. The first baseline system is a CNN model with two convolutional layers, followed by global average pooling across time, and then a finaloutput linear classification layer. This model is based on the architecture proposed by Verma and Thickstun (2019) but is adapted to a bootleg score representation (instead of MIDI). The second baseline system is a GPT-2 model (Radford et al., 2019) that is trained in the same manner as in Section 3.2: each bootleg score column is represented as a sequence of 8-bit characters, a subword vocabulary is learned using Byte Pair Encoding, a small 6-layer GPT-2 language model is trained on unlabeled bootleg scores in IMSLP, and the model is fine-tuned on the labeled data. To tease apart the effects of pretraining and fine-tuning, we report results of the GPT-2 model using three different training conditions: (1) training the model from scratch on the labeled data without pretraining (“GPT-2 (no pretrain)”), (2) pretraining the language model and learning a linearprobe (“GPT-2 (LP)”), and (3) pretraining the language model, learning a linear probe, and then unfreezing and fine-tuning the whole model (“GPT2 (LP-FT)”). For (3), we followed the recommended practices in Kumaret al. (2022), which were shown to have good generalization to out-of-distribution data. The third baseline system is a RoBERTa model (Liu et al., 2019) with six Transformer encoder layers. This model is pretrained using a masked language modeling task but is otherwise trained in a similar manner as GPT-2. We report results of the RoBERTa model using the same three training conditions described above. These three model architectures were previously explored in Tsai and Ji (2020) and in Yang and Tsai (2021a) on a 9-class composer recognition task, and here we present results on the (new) 9-class and 100-class PBSCR benchmarks. These results are intended to serve as baselines that future approaches can be compared to.

There are two things to notice about the baseline results in Tables 3 and 4. First, the GPT-2 model has the best performance among the three models on both the 9-class and the 100-class recognition tasks. We can see that pretraining on the unlabeled IMSLP data makes a big difference, improving top-1 accuracy on the 9-class recognition task from to and improving top-5 accuracy on the 100-class recognition task from to . This underscores the importance of having a large, diverse set of data for pretraining. We also see that full model fine-tuning makes a big difference, improving top-1 accuracy on the 9-class recognition task from to and improving top-5 accuracy on the 100-class recognition task from to . Second, there is a lot of room for improvement. The best GPT-2 model achieves a top-5 accuracy of on the 100-class recognition task, showing that there is a massive amount of room for improvement.

Our hope is that this dataset can spur progress on this challenging task.

5.2 1-Shot and low-shot composer recognition

An interesting modification to the above problems is to study few-shot composer recognition. The problem setup is the same as before, but the number of training examples per composer is artificially limited to . We recommend the following tasks: (a) a9-class recognition task with and (b) a 100-class recognition task with . This set of tasks encourages the development of approaches that are data-efficient (with labeled data), and it allows one to study the effect of the number of training examples as well as the generalizability of model representations.

We consider three baseline models for the few-shot tasks. The first system is a GPT-2 model that is trained by: (a) pretraining a 6-layer GPT-2 language model on the unlabeled IMSLP bootleg score data, as described in Section 5.1; (b) using the penultimate activations of the language model as a feature representation for the training samples; (c) identifying the nearest neighbors for each composer that are closest in Euclidean distance to a given test query; and (d) rank-ordering the composers by average Euclidean nearest neighbor distance. The second system is a 6-layer RoBERTa model that is trained and used in a similar manner, except with a masked language modeling task during pretraining. These models adopt the pretraining strategies of the classification models described in Section 5.1 but use -nearest neighbors for classification instead of training a classification layer. We also evaluate the performance of a random-guessing baseline for reference.

Tables 5 and 6 show the performance of the three baseline systems on the few-shot 9-class and 100-class recognition tasks, respectively. The upper, middle, and bottom sections of the table show performance for an-shot task with , and , respectively. For we use the nearest neighbor foreach composer (by necessity), and for and we use the nearest neighbors. In each trial, we randomly sample training examples from each composer to simulate a few-shot scenario, and then we calculate the performance on the entire test set. We report the mean and standard deviation of performance across 30 trials.

We can see that the pretrained models perform significantly better than random, and that both models perform comparably across all settings. While these results show that the pretrained models are indeed extracting style information, the performance of these models is quite poor overall, indicating how much room there is for improvement. We provide these results as a baseline against which future works can compare.

5.3 Zero-shot composer recognition

Another interesting problem to consider is zero-shot composer recognition. In this task, the goal is to predict the composer of a bootleg score fragment when no previous training examples of that composer have been seen. We are not aware of previous work studying this topic within the composer recognition literature until recently, where researchers explored zero-shot composer classification with music-text data (Wu et al., 2023). Here, we simply define the task and suggest some possible avenues of exploration.

This task is possible due to the rich metadata available on IMSLP. For example, given the Wikipedia articles for some unknown composers, one could infer aspects of compositional style based on their date of birth, country of origin, connections to other composers, or other knowledge about the composer that is embedded in a large language model. Wu et al. (2023) trained a model to embed both symbolic music and text descriptions into a common embedding space using a CLIP-like approach (Radford et al., 2021). This work does not release their music-text training pairs, however, and also evaluates on a small evaluation dataset (411 pieces, 8 classes). The PBSCR data would be sufficiently large to train such models, is open to the research community, and offers a much more challenging classification task. Setting up a benchmark for multimodal tasks such as this is an area for future work.

The zero-shot task opens the door to multimodal approaches to composer recognition. Given the size, richness of metadata, and open nature of IMSLP, we believe that the PBSCR data is well poised to facilitate interesting and novel directions in multimodal research.

6.1 Encoding schemes

One open research question is “How should we encode music data when feeding it into a model?” We may want to select the encoding scheme to maximize performance on a particular task of interest, to have certain desirable properties such as key or tempo invariance, or some combination of factors. Because of our design decision to use a bootleg score representation, the PBSCR data has discarded a significant amount of musical information. Nonetheless, due to its simple format, it is well poised to facilitate rapid, iterative exploration of many interesting questions, some of which we describe next.

Image vs. Tokens. The fact that the bootleg score is a binary matrix raises the question: “Is it better to treat the data as a 2-dimensional binary image or as a sequence of discrete tokens (e.g., each bootleg score column is interpreted as a discrete “word”)?” These two options lead to different kinds of models: 2-dimensional images lend themselves to CNN or ViT-based architectures, while token sequences lend themselves to Transformer-based language models. Previous work (Yang and Tsai, 2021a) has compared simple CNN architectures with GPT-2 and RoBERTa, but a lot of other work has developed effective strategies for applying Transformers to images (e.g., ViT (Dosovitskiy et al., 2021)) and utilizing pretraining strategies like masked autoencoding (e.g., ViT-MAE (He et al., 2022)). Recent work has explored this topic (Zhang et al., 2023), and it remains an open research question.

Harmonic vs. Temporal. What are effective ways to capture both harmonic and temporal information? The approach described in Section 5.1 encodes bootleg score columns (or parts thereof) as discrete tokens and then models temporal information with a Transformer. But one could alternately encode rectangular blocks of the bootleg score image as discrete tokens, similar to the patches in a ViT model. In this case, each token would capture both harmonic and temporal information, rather than only harmonic information.

Raw vs. Processed. At what level of semantic representation is it best to represent discrete tokens? On one end of the spectrum, we could represent a bootleg score fragment simply as a sequence of zeros and ones and then let a Byte Pair Encoder combined with a Transformer model learn the most suitable representation. On the other end of the spectrum, we could design musically informed encodings, taking into account domain knowledge such as the split between the left- and the right-hand staves, the fact that staff line positions are cyclical and octave-based, etc. For example, one could encode a bootleg score column as the text “C3–G3–E4–C5”, which explicitly decomposes the staff line position into octave and class information. Although recent work has explored this topic (Fradet et al., 2023), this remains an open question.

Absolute vs. Relative. How much should the encodingof discrete tokens capture absolute vs. relative position information? One intuitive shortcoming of the Transformer models in Section 5.1 is that they encode the absolute staff line positions of noteheads, rather than relative position or movement. In the example given above, the bootleg score column encoded as “C3–G3–E4–C5” could instead be encoded as “C3–4–5–5” to capture the relative staff line intervals between notes in the chord. Furthermore, the root of the chord (C3) could itself be expressed relative to a notehead in a previous bootleg score column.

The topics above are issues that could be studied conveniently with the PBSCR data. Many other similar questions could be rapidly explored, given the simplicity and format of the data. Thus, even though the dataset does not have complete symbolic score information, it can facilitate rapid exploration of ideas and progress on research questions of broad interest to the MIR community.

6.2 Data augmentation

Another open research question is “What are effective ways to perform data augmentation of symbolic music data?” The PBSCR data is ideal for exploring data augmentation techniques for two reasons, which we describe below.

First, the PBSCR data has a very simple format. The fact that the bootleg score is a simple 2-dimensional image makes it much easier to explore data-augmentation techniques. For example, many data-augmentation strategies developed for computer vision can be applied out-of-the-box, with no additional effort such as cropping, shifting, or MixUp (Zhang et al., 2018). In contrast, for symbolic music formats like MusicXML, it would be much more cumbersome to rapidly explore the same types of data augmentation. Both existing and novel techniques would be far easier to implement with bootleg score images than with MusicXML data.

Second, the PBSCR data format has several key properties that make such augmentations musically meaningful. One such property of the bootleg score is the nature of key changes: because the staff line positions (A through G) are cyclical, key changes correspond to simple vertical shifts in the bootleg score representation (assuming that the key signature is properly adjusted). Similarly, shifts in time correspond to simple horizontal shifts in the bootleg score representation. The bootleg score also has the property of additivity—if you add a bootleg score event (i.e., a binary vector of length 62) describing a chord in the right hand to a bootleg score event describing a left-hand chord, the resulting event is simply the sum of the two constituent bootleg score events. (Note that discrete token-based representations do not have this property.) It is also worth pointing out that the bootleg score representation is inherently invariant to tempo since it describes only the sequence of noteheads rather than the absolute time between note events. Because of these properties, operations like cropping and shifting and MixUp are very easy to implement and have clear musical interpretations.

The above properties make the PBSCR dataset ideal for exploring data-augmentation strategies. This includes exploring the effectiveness of existing data-augmentation techniques from computer vision, as well as quickly implementing and trying domain-specific data-augmentation techniques.

6.3 Integrating multimodal information

Yet another interesting research question is “How can we train models with multiple modalities of data?” The PBSCR dataset is ideal for exploring this question in MIR for three reasons, which we describe next.

First, the PBSCR data is linked to rich metadata on IMSLP. In particular, each work in IMSLP has a lot of rich multimodal information: audio recordings, MIDI files, sheet music scores, arrangements and transcriptions, relevant metadata (e.g., composer, publisher information, composition date, composer time period, instrumentation), and links to relevant Wikipedia article pages and descriptions (e.g., All Music Guide). Importantly, in keeping with IMSLP’s philosophy, these resources generally have very research-friendly licenses: most audio recordings and sheet music scores have a Creative Commons license or are in the public domain.

Second, the PBSCR dataset is large enough to study multimodal problems at a nontrivial scale. Given the size and scale of modern models, it is necessary to have a large enough quantity of data to train large models. The PBSCR dataset—and certainly IMSLP—fulfills this requirement. One additional benefit of utilizing IMSLP data is that the website is actively maintained, so the quantity of data will presumably only increase into the future.

Third, the bootleg score representation is well suited for cross-modal and multimodal tasks. Previous works have demonstrated the effectiveness of the bootleg score representation for cross-modal tasks. For example, it has been used in cross-modal retrieval to find matches between the Lakh MIDI Dataset and sheet music in IMSLP (Yang and Tsai, 2021b), and it has been used in cross-modal transfer learning to perform composer classification of audio recordings using sheet music as training data (Yang and Tsai, 2021a). As such, it is well suited to connect multiple representations of music, including sheet music images, symbolic files, and audio.

For these reasons, we believe the PBSCR dataset is particularly well situated to facilitate multimodal research in MIR. Setting up the infrastructure for specific multimodal tasks is an area for future work.