Beethoven Symphony Excerpt Dataset (BSED): An Evaluation Dataset for Orchestral Music Transcription

2.1 Symbolic sheet music–encoding

Creating reliable symbolic score datasets is a long‑standing challenge in MIR, particularly when combined with orchestral recordings. While MIDI provides a basic note‑level representation, richer formats such as MusicXML, **kern, MEI, MuseData, or LilyPond capture articulations, ornaments, and structural markers, though each comes with trade‑offs between readability, interoperability, and machine processing (Good, 2001; Huron, 2002; Nienhuys and Nieuwenhuizen, 2003; Selfridge‑Field, 1997). For orchestral repertoire, several large collections exist. The MuseData database³ includes many Western classical music scores, including all nine Beethoven symphonies, which we use as a foundation in this work. Related resources include KernScores (Sapp, 2005) and the crowdsourced MuseScore library. The OpenScore Orchestra dataset,⁴ which is based on Gotham et al. (2022), also provides the score of Beethoven’s symphonies but was developed independently of our efforts.

2.2 Datasets for AMT

Beyond symbolic encoding, a wide range of datasets has been developed for AMT, using different strategies to obtain annotations. These datasets can be broadly categorized into solo‑instrument and multi‑instrument collections, further subdivided into multi‑channel (facilitating track‑wise annotation) and full‑mix recordings. Annotation strategies include capturing note data with specialized hardware (e.g., MIDI‑ready piano), aligning scores to recordings, or using transcription models. Audio material may come from existing recordings, purpose‑made performances, or synthesized scores. While synthesized audio enables perfect annotations, the gap between synthetic and real recordings can hinder generalization (Maman and Bermano, 2022; Zehren et al., 2024).

Solo‑instrument datasets. The piano is the most extensively studied instrument for AMT. Datasets such as MAESTRO (Hawthorne et al., 2019), MAPS (Emiya et al., 2010), and SMD (Müller et al., 2011) provide time‑aligned annotations, often via a MIDI‑ready piano. The ASAP dataset (Foscarin et al., 2020) adds aligned sheet music. More recent resources include PiJAMA (Edwards et al., 2023) and ATEPP (Zhang et al., 2022), which rely on automatic transcription for annotation. Beyond piano, datasets exist for guitar (Riley et al., 2024b; Xi et al., 2018), bass (Abeßer and Schuller, 2017), violin (Tamer et al., 2022), and drums (Weber et al., 2025).

Multi‑instrument datasets. In Western classical music, multi‑instrument datasets typically focus on chamber ensembles and small orchestras. Annotation is usually performed via score–audio alignment (e.g., MusicNet, Thickstun et al. (2017)) or through synchronized multi‑track recordings (e.g., Bach10 (Duan et al., 2010); URMP (Li et al., 2019); ChoraleBricks (Balke et al. 2025)). While multi‑track data allows flexible re‑mixing and semi‑automated annotation, such datasets are usually small due to the high cost of controlled recording. Synthetic collections such as Slakh (Manilow et al., 2019) and EnsembleSet (Sarkar et al., 2022) provide a larger scale but lack real‑world performance characteristics.

Orchestral datasets. Resources for full orchestra remain scarce. The PHENICX Anechoic dataset (Miron et al., 2016; Pätynen et al., 2008) and the Beethoven Anechoic dataset (Böhm et al., 2021) provide separately recorded orchestral parts under controlled conditions but are limited in duration (ca. 10–20 min). More recently, the SynthSOD dataset (Garcia‑Martinez et al., 2025) has offered large‑scale synthetic orchestral data. However, to date, no dataset has combined existing real orchestral recordings with aligned score information—a gap the present work seeks to address.

2.3 Score–audio alignment and structural inconsistencies

Closely related to dataset creation is the task of score–audio alignment, which maps the symbolic timeline of a score to the physical timeline of a performance. A common approach is sequence alignment, where methods such as dynamic time warping (DTW) (Müller, 2021; Müller et al., 2021) warp the score axis to match the recording. Chroma and onset features are typically used for robustness across conditions, while more recent approaches employ transcription‑based features (Maman and Bermano, 2022; Zeitler et al., 2024a).

While sequence alignment provides reliable measure‑level mappings, it cannot capture fine‑grained timing deviations. For instance, cases where multiple notes are notated at the same measure positions but played with slightly different timing cannot be resolved by sequence alignment. Note‑level alignment addresses this by matching individual note onsets (and sometimes offsets) to their precise timing in the audio. Applications include piano (Niedermayer and Widmer, 2010), orchestral recordings (Miron et al., 2014), and MIDI‑to‑MIDI alignment (Peter et al., 2023). Recent refinements again leverage transcription‑based features for improved precision (Riley et al., 2024a; Saha et al., 2026).

A major challenge is handling structural inconsistencies between notated scores and performances, such as omitted or repeated sections or mismatched measure numbering. Fremerey et al. (2010) laid the foundation for structure‑aware synchronization by modeling repeats and jumps explicitly. Building on this idea, subsequent work has proposed strategies such as excluding non‑matching pairs (Thickstun et al., 2017), adapting performances to a reference structure (Zeitler et al., 2024b), encoding measure numbering for cross‑version transfer (Gotham et al., 2023), or extending alignment algorithms to account for skipped sections (Grachten et al., 2013). Together, these approaches show that robust alignment requires not only temporal accuracy but also explicit treatment of structural variation. These issues are also relevant in orchestral settings, where interpretation of symphonies is shaped by the conductor, both in overall tempo and tempo changes, as well as in decisions about which repetitions are performed.

2.4 AMT

AMT aims to convert audio into symbolic note representations, with methods ranging from frame‑level pitch detection to full score generation (Benetos et al., 2019). Recent research has focused on note‑level transcription, which identifies the onsets and offsets of individual notes. Onset‑based approaches, first shown to be highly effective for piano (Hawthorne et al., 2018; Kong et al., 2021), have since been extended to strings, winds, and small ensembles (Chang et al., 2024; Gardner et al., 2022; Kim et al., 2023; Riley et al., 2024a; Tamer et al., 2023; Wu et al., 2024). Despite this progress, AMT for orchestral music remains largely unexplored: most studies concentrate on solo instruments or chamber ensembles, and systematic evaluations on orchestral datasets are rare. One exception is Bittner et al. (2022), who applied an instrument‑agnostic transcriber to orchestral stems from the PHENICX dataset, though this was limited to isolated instrument groups rather than complete recordings. The dataset proposed in this work addresses this gap by providing a foundation for systematic note‑level transcription research on orchestral music.

3.1 Overview

The BSED is based on 20 short score excerpts from Beethoven’s symphonies and is tailored for the evaluation of AMT models. For each excerpt, we provide symbolic score data together with five corresponding audio renditions: four concert recordings and one synthetic version, yielding 100 renditions in total. For each rendition, we provide note‑level annotations obtained through score–audio synchronization methods. The design of the dataset enables robust evaluation of AMT models across both different musical content and diverse acoustic conditions.

Table 1 provides an overview of the excerpts. Each excerpt is specified by an identifier, BSED‑ID, where the ID ranges from 01 to 20. The excerpts are drawn from various movements across the symphonies. In total, the dataset contains 6342 symbolic note events, which correspond to 31,710 annotated note events within the audio versions. The total duration of all audio versions is approximately 37 min, with individual excerpt durations ranging between 17 and 30 seconds. We first discuss the musical content of the excerpts, followed by a more technical description of the symbolic data. Afterwards, we discuss the audio data and the alignment process.

3.2 The excerpts in detail

Twenty excerpts were selected from the corpus of Beethoven’s Symphonies to represent the diverse musical content. However, we purposefully excluded the fourth movement of Symphony No. 9, as it uniquely features choir and unpitched percussion, instead focusing on the more consistent orchestral instrumentation found across the other symphonies. Ten out of the 20 excerpts were chosen manually by the authors to capture musically meaningful phrases and iconic themes, many of which are movement openings, such as the opening of the Fifth Symphony. The remaining 10 excerpts were selected at random to introduce unbiased diversity. As a result, these 10 excerpts are not necessarily complete musical phrases. We made sure to have a diverse test dataset in terms of musical content. We will show this at the end of this section.

The performed durations of the individual excerpts are in the range of 15–30 s. For each excerpt, the corresponding measure numbers from the score can be found in Table 1. The measure numbers refer to the Urtext edition by Del Mar (2020), which serves as a basis for the BSED. We use inclusive measure ranges (e.g., measures 1–4 refer to the first four measures of the movement) and only selected complete measures from the score, except for the opening anacrusis of BSED‑03, which is denoted as measure 0 to remain consistent with the numbering in the score edition.

Figure 2 shows the distribution of total notes by instrument, reflecting overall trends in Beethoven’s orchestration. Twelve instruments are represented, grouped into four families: strings, woodwinds, brass, and percussion (timpani only). The violin is by far the most frequently used instrument, while the trombone appears least often, with only 23 notes in total. Some instruments occur in multiple variants (e.g., bass trombone, contrabassoon, piccolo flute), but, for simplicity, we treat these as single instruments here. In the scores, however, they are labeled with their correct specific names.

Figure 2

Figure 3 breaks down the number of notes per excerpt by instrument, color‑coded by instrument. Across all excerpts, there are 6342 notes, with individual excerpts ranging from 72 to 769 notes. These variations stem primarily from differences in note density, i.e., number of onsets per time, and, not excerpt duration, as actual playing durations are all within a range of roughly 15–30 s (see Table 1). While most excerpts feature a mix of instrument families, some are dominated by a single group. For example, BSED‑03 contains mainly winds, while BSED‑15 contains mainly strings.

Figure 3

Figure 4 displays the overall pitch distribution, color‑coded by instrument. Pitches range from a Contrabass C1 (MIDI number 24) to a Violin G6 (MIDI number 91), with D4 (MIDI number 62) being the most frequently occurring pitch.

Figure 4

3.3 Symbolic data

The symbolic layer of the BSED contains digital representations of the sheet music for the selected excerpts in various formats. Each score file is named according to the pattern

where BSED‑ID is the unique identifier for each excerpt, opNr is the opus number of the corresponding symphony, and mvmtNr is the movement number from which the excerpt was taken. Thus, for instance, BSED‑01_Beethoven_Op‑021‑01 refers to the first excerpt, taken from Symphony No. 1 (Op. 21), first movement. Across the nine symphonies, there are 37 movements in total. To uniquely identify each movement, we use the combination opNr‑mvmtNr, denoted as movementID, for the remainder of this work. The scores are based on the Urtext edition by Del Mar (2020) and were manually created by the authors using the Sibelius notation software.⁸ The accuracy of the scores was cross‑checked by expert annotators using the Urtext edition as a reference, with further validation performed through score sonification. Despite the availability of existing encodings, such as the aforementioned MuseData database, we created the scores manually to ensure high quality and consistent encoding across all BSED excerpts.

The symbolic scores are provided in the folder 1_ScoreData in the folllowing five formats:

1. Sheet music in the form of Portable Document Format (PDF) files

2. Sibelius files (dedicated files for Sibelius 8 and Sibelius Ultimate)

3. Music Extensible Markup Language (MusicXML) files exported from Sibelius with the Dolet 8 plugin for Sibelius⁹

4. Note lists in .csv format

5. MIDI files

The note event lists are derived from the MusicXML files and serve as a MIDI‑like encoding enriched with score‑specific information. Each note event with a distinct onset is represented by a single row in the .csv file; tied notes (i.e., consecutive occurrences of the same pitch connected by a slur and performed as a single sustained note) are merged into one event.

Each note event comprises several attributes (columns in the .csv). Onset and offset are first specified as measure positions (start_meas, end_meas), encoded as single real‑valued numbers following Zeitler et al. (2024b). The integer part denotes the measure number, and three decimal digits indicate the relative position within the measure (e.g., in 3/4 time, the second beat of measure 137 is encoded as 137.333). In addition, physical onset and offset times in seconds (start_sec and end_sec) are provided, computed from the annotated tempo in the score. Further attributes include pitch (MIDI pitch number), instrument, velocity (derived from score dynamics), and part (e.g., Violin 1 or Violin 2). A comprehensive description of all attributes is available in 04_Extra/note_encoding_info.md. The provided MIDI files are directly derived from the note event lists using physical time, pitch, instrument, and velocity information.

Since the dataset is designed for note‑level AMT rather than sheet music layout, we apply several simplifications at the notation level (PDF, Sibelius, MusicXML) to ensure consistent processing with software libraries. Dynamic and expression markings are standardized (e.g., crescendo indicated by hairpins instead of textual annotations), abbreviations for repeated notes—common in string parts—are written out explicitly, and timpani trills (occurring in BSED‑12) are realized as individual notes. We provide two versions of the sheet music files: one in concert pitch to avoid parsing errors with respect to sounding pitch, and one retaining the original transposing notation for readability.

3.4 Audio data

For each of the score excerpts, we provide five different versions, indexed with a splitID from 1 to 5, as can be seen in Table 1. Splits 1–4, collectively referred to as BSED‑Real, are taken from concert recordings, while split 5, also referred to as BSED‑Sytnh, is synthesized using the NotePerformer software¹⁰.

Of the four versions from BSED‑Real, splits 1–3 are taken from public domain recordings from the International Music Score Library Project (IMSLP)¹¹ with varying conductors, while split 4 is always taken from a CD collection of performances conducted by Herbert Blomstedt, which is not in the public domain. The NotePerformer software used for BSED‑Synth is a sample‑based playback engine that incorporates articulation markings from the score to produce realistic and expressive output directly from the Sibelius scores. However, our primary objective in the synthesized excerpts is to have perfect ground‑truth note annotations rather than maximum realism; therefore, we disable NotePerformer’s humanization feature as well as fermatas in the score, as both would otherwise introduce timing deviations between the annotations and the audio. The total duration of the audio data is 37 min and 14 s.

The audio files stored in the folder 2_Audio are named according to the format

where BSED‑ID is the unique ID of the score excerpt and splitID denotes the audio split indexed from 1 to 5, as discussed above. The versionID typically consists of the conductor’s last name with the four‑digit recording year, but, in the case of the synthetic version, we use Synth. For example, as can be seen in Table 1,

is the first excerpt taken from Op. 21, first movement, cut from a recording of Karajan in 1963.

The recording excerpts were cut manually from the full recordings to ensure the correct start and end of the excerpts. We provide the audio files in the folder 2_Audio as stereo .wav files with a sampling rate of 44.1 kHz. However, some recordings are only upsampled from lower sampling rates and therefore do not contain the full bandwidth, and some only contain a duplicated second channel, instead of proper stereo recordings.

3.5 Score–audio alignment

To obtain physically aligned note annotations, we apply a two‑stage score–audio alignment procedure to map musical onset times (given in measures) to physical onset times (given in seconds).

As discussed in Section 3.5, we distinguish between sequence‑level and note‑level alignment. Sequence‑level alignment provides a robust global mapping from musical time (score domain) to physical time (audio domain), assigning all notes with the same notated onset to a common physical time and thereby preserving structural coherence, which suffices, e.g., for score‑following applications (Dannenberg and Raphael, 2006). Note‑level alignment refines this mapping by adjusting individual onsets to better match their realized performance timing, resolving local asynchronies and enabling precise note‑level analysis. In the following, we describe the resulting semi‑automated pipeline.

3.5.3 Discussion

The two alignment variants reflect complementary perspectives. Sequence‑level alignment preserves structural coherence and is less sensitive to local pitch‑wise ambiguities. Note‑level alignment emphasizes physically realized timing and is therefore closer to the performance domain.

Figure 5 illustrates this principle: sequence‑level alignment assigns a single physical onset time to notes that share the same musical onset time, whereas note‑level alignment can resolve small onset asynchronies between these notes. However, its quality ultimately depends on the transcription model providing sufficient onset cues for all notes.

Figure 5

We note that neither representation constitutes a perfect ground truth, as achieving such precision would require clean multitrack recordings or dedicated MIDI capture equipment for every instrument (e.g., a MIDI‑ready piano). Instead, both should be understood as estimates from score‑centric and performance‑centric viewpoints, respectively. As we will show in Section 5, both provide a meaningful basis for training and evaluation. The note‑level alignment in particular proves effective for training onset‑based transcription models.

Aligned annotations are provided in 03_Note Annotations, organized into Sequence‑Alignment and Note‑Level‑Alignment. For each audio excerpt and alignment type, we provide a note‑list .csv file and a corresponding MIDI file matching the audio filename. Compared to the unaligned symbolic note lists, onset and offset times now refer to physically aligned times in seconds.

4.1 Audio data

We collected public‑domain recordings of the Beethoven Symphonies from IMSLP. Table 2 gives an overview of the symphonies and the number of recordings collected for each movement. In total, we collected 415 recordings of single movements, resulting in 63 h of music. As already discussed previously, the fourth movement of Symphony No. 9 is an outlier in terms of instrumentation, which is why we report it separately in Table 2 and note that it may be advisable to exclude it when training transcribers on the more typical instrumentation of the other symphonies.

Each audio file name follows the same format as in the BSED, but without the BSED‑ID and the splitID in front—for example, Beethoven_Op021‑01_Karajan 1963. The audio, stored in the folder 02_AudioData, follows the same format as the BSED and similarly is also available both in its original tuning and in a ‘tuning‑corrected’ version (A4 = 440 Hz).

Similar to the BSED, the audio quality varies significantly. To give users more control over the data that they want to use, we provide quality ratings for each audio file from 4 (highest) to 1 (lowest). Ratings are derived using a semi‑automated approach based on the Fréchet audio distance (FAD) (Kilgour et al., 2019). Specifically, we use a set of high‑quality orchestral CD recordings as a reference and compute the FAD of each audio file relative to this set. The embeddings required for FAD computation are extracted using the TRILL model (Shor et al., 2020). We define decision thresholds based on the FAD for the quality categories through an informal listening test. However, these ratings provide only a coarse indication of audio quality, without distinguishing between specific degradations (e.g., noise, clipping, compression). All quality ratings and corresponding FAD scores are provided in the file quality_ratings_BSD.csv.

4.2 Symbolic data

Our annotations for the BSD are derived from the previously mentioned MuseData repository based on the work by Selfridge‑Field (1997), which offers digitally encoded versions of all Beethoven symphonies, among others. The MuseData format was developed with a focus on producing human‑readable sheet music and accurate notational detail. Unlike the BSED, we do not provide any sheet‑music–oriented formats such as Sibelius or MusicXML, nor symbolic score representations in MIDI or note‑list format and rather refer to the original MuseData repository. Instead, we only use the MuseData scores as a basis for deriving aligned note‑level annotations of the collected recordings, including onset and offset time, pitch, and instrument.

4.3 Score–audio alignment

To align BSD note annotations with audio, we apply the same two‑stage procedure as in BSED (Section 3.5): an initial sequence alignment followed by note‑level refinement. For robustness, we manually annotated the start and end of each recording to serve as DTW anchor points.

In contrast to BSED, BSD involves full symphonies, requiring us to ensure structural consistency between score and audio. To this end, we create a set of possible unfoldings of the note annotations for each movement. Each set initially contains only a standard unfolding, where repetitions are expanded according to common conventions in classical scores. Each recording is then tested against all available unfoldings; if none match—that is, if no structural equivalence is found—a new unfolding is generated manually and added to the set. Since many recordings of the same movement share an identical structure, newly created unfoldings often apply to multiple cases, thereby reducing manual effort.

Structural equivalence is evaluated semi‑automatically by comparing the DTW warping path to an ideal diagonal. Paths that remain close are accepted, while larger deviations trigger manual inspection. Such deviations may indicate structural differences but can also arise from local tempo changes or expressive timing, such as fermatas. Figure 6 illustrates this with two alignments of the same recording against structurally different scores.

Figure 6

Although this process ensures structural correctness, we do not manually verify local timing accuracy, and some errors may remain, particularly in lower‑quality recordings. As a heuristic, paths close to the diagonal generally indicate good alignment, though this is neither a necessary nor sufficient condition. Stronger deviations can still correspond to valid alignments but require caution. For transparency, we provide the average deviation from the diagonal (in seconds) for each piece in warping_path_deviations.csv.

After sequence alignment, we apply the same note‑level refinement as in BSED, but with a wider snapping window (1 second compared to 150 ms for BSED) to accommodate larger initial alignment errors, since we do not set any anchor points for the BSD. For each recording, we provide three structurally matched annotation sets: (1) an unaligned version based on score tempo, (2) a sequence‑aligned version via DTW, and (3) a fine‑grained note‑aligned version. All annotations are available both as note‑list .csv files and as derived MIDI files. Since our transcriber used for note‑level alignment is not trained on singing or unpitched percussion, we cannot generate note‑level alignments for the fourth movement of Op. 125. For these recordings, we therefore provide only the initial sequence alignment.

4.4 Data splits between BSD and BSED

The recordings of which splitIDs 1–3 of the BSED were cut from also appear in the BSD. This creates an overlap between the training and test sets when using the full BSD for training and the BSED for testing. To prevent evaluating on data seen during training, we have to exclude some data from the BSD. There are, however, multiple ways to handle this removal, which has been noted before in datasets containing multiple versions of the same piece (Schreiber et al., 2020; Weiß et al., 2021; Zeitler et al., 2024b).

To address this issue, we build on the definitions of Schreiber et al. (2020) and define several splitting strategies that exclude recordings from the BSD in different ways. These strategies differ in how strictly and at what level they prevent overlap. At the movement level, recordings with the same movementID as a BSED excerpt are removed to ensure that no movement appears in both training and test sets. At the version level, recordings are excluded to avoid shared acoustic conditions between training and test data. Concretely, we define the following splits (excluding only BSD recordings unless noted otherwise):

• Movement Split: Excludes all recordings with the same movementID as a BSED excerpt.

• Version Split—Whole Symphony: Excludes all recordings with the same versionID as a BSED excerpt.

• Version Split—Single Movement: Excludes all recordings with both the same versionID and movementID as a BSED excerpt.

• Neither Split: Excludes all recordings with either the same versionID or the same movementID as a BSED excerpt.

• Version Split—Reduced BSED: Does not exclude BSD recordings but restricts evaluation to BSED splitIDs 4 and 5 (Blomstedt and Synthetic), which do not overlap with BSD versionIDs.

Table 3 provides an overview of these approaches along with the resulting number of training movement recordings and test excerpts. The choice of split depends on the evaluation goal: version‑level tests generalization across acoustic conditions, while movement‑level targets generalization across musical content. The neither split enforces the strictest separation but greatly reduces training data. For scenarios including additional cross‑dataset evaluation, we suggest the Version Split—Single Movement as a balanced compromise. Note that the Version Split—Reduced BSED differs substantially from the full BSED in audio quality and proportion of synthetic audio, making results not directly comparable. To facilitate reproducibility, we provide Python functions in data_splits.py that implement these splitting strategies and return the corresponding train and test file lists. Further details can be found in the inline documentation.

4.5 Annotation limitations

The parsing process of the MuseData scores used for the training data introduces several limitations related to the quality and completeness of the annotations, which must be considered: (1) Dynamic markings are sometimes inaccurate or inconsistent; therefore, we fill the velocity parameter with a default value of 64 in the note list and MIDI files. (2) Pizzicato indications for the strings are not preserved, and such notes are treated as regular bowed string notes. (3) Grace notes are missing in the annotations. Please note that these limitations only apply to the training data and not to the BSED, since the BSED scores are created manually based on the printed sheet music.

5.1 Models and evaluation

Our transcription models are based on the Onsets & Frames architecture (Hawthorne et al., 2018), which has been shown to handle multi‑instrument settings such as chamber music (Maman and Bermano, 2022) but has not yet been applied to full orchestral data. In contrast with Maman and Bermano (2022), we train our models from scratch on a fixed set of labels without any iterative label refinement to demonstrate that our dataset can be used directly for training, without additional alignment adaptation steps. For training on BSD, we use the Version Split—Single Movement with a small subset of held‑out movements for validation and use pitch shift‑augmentation of $\pm 2$ semitones to make the transcribers more robust across tonality.

Our evaluation focuses on instrument‑agnostic note‑level transcription using the mir_eval library (Raffel et al. (2014)), computing precision, recall, and F1‑score with a 50‑ms onset tolerance, without considering note offset or instrument. We evaluate on both BSED and PHENICX. For BSED, we evaluate on the versions pitch‑shifted to the reference tuning of $\mathrm{A}4=440\mathrm{Hz}$.

Instrument agnostic transcription for orchestral music can constitute an ill‑posed problem, as multiple sources may produce overlapping notes of identical pitch (Berendes et al., 2026). We therefore treat the existing—potentially simplified—metric as a baseline in this work, while advocating for the development of more appropriate evaluation metrics in future research.

PHENICX plays an important role in our study: as a non‑synthetic multi‑track dataset, its annotations were performed on separate monophonic tracks, enabling fairly accurate and reliable note‑level annotation. It therefore serves as a valuable cross‑dataset evaluation for our transcriütion models. As we show, results on BSED are generally consistent with those obtained on PHENICX.

For PHENICX, we combine all instrument tracks and apply a reverberation of 0.3 s to resemble real recording conditions, enabling assessment on unseen composers and acoustic environments. Since multiple notes of identical pitch may coincide within a single frame in the PHENICX annotations, we retain only one instance per pitch per frame to match the capabilities of our frame‑based models.

We additionally evaluate two pre‑trained models from the literature: the instrument‑aware transcriber from Chang et al. (2024), denoted YMT3+,¹³ and the instrument‑agnostic transcriber from Bittner et al. (2022), denoted BasicPitch.¹⁴ Although YMT3+ is designed for multi‑instrument transcription, we evaluate it in an instrument‑agnostic manner by discarding the predicted instrument labels. As for the PHENICX labels, overlapping notes with identical pitch within the same time frame are merged into a single note in order to ensure a fair instrument‑agnostic evaluation.

5.2 Impact of alignment type on training and evaluation

To assess annotation quality and systematically compare sequence‑ and note‑level alignments in training and evaluation, we consider three alignment variants on BSD and BSED. As a baseline, we use sequence‑aligned annotations derived from signal‑processing features via the SyncToolbox (Müller et al., 2021), denoted SeqSP. We further employ our transcription‑based sequence alignment, SeqTr, and, lastly, the refined note‑level alignment, Note. Separate models are trained on BSD for each label variant and denoted by $\mathcal{T}$ with alignment‑specific subscripts (i.e., ${\mathcal{T}}_{\mathrm{SeqSP}}$, ${\mathcal{T}}_{\mathrm{SeqTr}}$, ${\mathcal{T}}_{\mathrm{Note}}$). For evaluation on BSED‑Real, we analogously consider all three alignment types, denoted ${\mathcal{A}}_{\mathrm{SeqSP}}$, ${\mathcal{A}}_{\mathrm{SeqTr}}$, and ${\mathcal{A}}_{\mathrm{Note}}$. Additionally, we evaluate on BSED‑Synth and PHENICX using ‘ground‑truth’ annotations, ${\mathcal{A}}_{\mathrm{GT}}$, to assess alignment‑independent performance and cross‑dataset generalization. However, please note that, although PHENICX onset annotations are mostly correct within 50 ms, they may include slight temporal errors or ambiguous onset positions, e.g., for very soft onsets.

Table 4 reports mean note‑level precision, recall, and F1‑scores across transcribers and evaluation sets. The ${\mathcal{T}}_{\mathrm{SeqSP}}$ model performs consistently worse across almost all evaluations except for ${\mathcal{A}}_{\mathrm{SeqSP}}$. In particular, on BSED‑Synth and PHENICX, it has F1‑scores of 36.5% and 49.4%, respectively, versus 70.6% and 65.7% with ${\mathcal{T}}_{\mathrm{Note}}$. This indicates that the ${\mathcal{A}}_{\mathrm{SeqSP}}$ labels are weak targets for training and, by extension, also problematic for evaluation. In contrast, models trained on transcription‑informed sequence alignment (${\mathcal{T}}_{\mathrm{SeqTr}}$) and note‑level alignment (${\mathcal{T}}_{\mathrm{Note}}$) yield substantially higher overall performance. However, the two training strategies exhibit complementary behavior: ${\mathcal{T}}_{\mathrm{SeqTr}}$ tends to achieve higher precision but suffers from comparatively low recall, whereas ${\mathcal{T}}_{\mathrm{Note}}$ shows the opposite trend, with greater recall at the cost of reduced precision. While differences bet‑ ween the evaluation sets ${\mathcal{A}}_{\mathrm{SeqTr}}$ and ${\mathcal{A}}_{\mathrm{Note}}$ are on the order of only a few percentages, ${\mathcal{T}}_{\mathrm{Note}}$ yields higher overall scores on PHENICX, reaching 65.7% compared to 60.4% for ${\mathcal{T}}_{\mathrm{SeqTr}}$.

Overall, both transcription‑informed alignment variants yield meaningful training targets, enabling models trained from scratch on BSD to generalize to PHENICX. However, the resulting transcribers exhibit opposite precision–recall trade‑offs. Training on the note‑aligned labels seems to enable slightly better cross‑dataset generalization, as evident in the results on PHENICX. Evaluating on ${\mathcal{A}}_{\mathrm{Note}}$ appears slightly more informative for note‑level transcription, as it better reflects the fine‑grained temporal structure the model is expected to capture, which we explore further in the next section.

5.3 Note‑level transcription baseline

In the following, we focus on training and evaluation using our note‑level–aligned labels ${\mathcal{A}}_{\mathrm{Note}}$ to establish a note‑level transcription baseline. We assess note‑level transcription performance on BSED and PHENICX across multiple transcribers, including the pre‑trained models YMT3+ and BasicPitch. We train three models from scratch: ${\mathcal{T}}_{\mathrm{MN}}$, trained on MusicNet with MusicNetEM labels (Maman and Bermano, 2022); ${\mathcal{T}}_{\mathrm{BSD}}$, trained on BSD with note‑level alignments; and ${\mathcal{T}}_{\mathrm{BSD}+\mathrm{MN}}$, trained on the combined datasets.

Table 5 reports note‑level precision, recall, and F1‑scores across all transcribers on BSED and PHENICX. BasicPitch and YMT3+ perform substantially below the other models. Training on BSD yields higher performance compared to MusicNetEM across both BSED and PHENICX, with a substantial increase on BSED (from 60.6% to 71.4%, +10.8 percentage points). When the two datasets are combined, the best overall results are achieved, showing consistent gains over MusicNetEM: the F1‑score on BSED rises from 60.6% to 70.8%, while, on PHENICX, it improves from 64.9% to 69.0%.

Notably, ${\mathcal{T}}_{\mathrm{BSD}}$ is as good as ${\mathcal{T}}_{\mathrm{BSD}+\mathrm{MN}}$ on BSED but worse on PHENICX, suggesting slight overfitting when training exclusively on BSD. On PHENICX, ${\mathcal{T}}_{\mathrm{BSD}+\mathrm{MN}}$ achieves the highest scores overall, demonstrating that combining BSD with MusicNet improves cross‑dataset generalization.

The performance spread across transcribers is notably larger on BSED than on PHENICX, which may reflect the challenging acoustic conditions of some of the older recordings in BSED, where models trained on clean audio data or non‑orchestral data tend to struggle.

Figure 7 shows the mean F1‑score of ${\mathcal{T}}_{\mathrm{BSD}+\mathrm{MN}}$ across onset tolerances for all evaluation subsets. For a high onset tolerance, all performance metrics are close together. For stricter onset tolerances, we see that the F1‑scores are slightly lower in general, but also the differences between them become larger. This is especially relevant when comparing the two BSED‑Real alignments, where we can see that ${\mathcal{A}}_{\mathrm{Note}}$ yields a circa four percentage points higher F1 score, compared to ${\mathcal{T}}_{\mathrm{SeqTr}}$.

Figure 7

Overall, our experiments confirm that training on BSD yields strong transcription performance and combining it with MusicNet further improves generalization.