Time to Align! Modelling Musical Timelines for Music Information Retrieval and Digital Musicology

2.1 Temporal domains in musical data

We adopt a conceptualisation of musical time through three domains – physical, logical and graphical – each providing a distinct perspective on how time is represented in music‑related data (Figure 1). The physical domain concerns data on physical, real‑time manifestations of sound or sound production: audio recordings, synthesised renditions, sensor data and derived features such as spectrograms. Time in this domain is measured in seconds, samples or frames. The logical domain concerns symbolic, conceptual representations of musical structure: note events in MIDI or MusicXML, measure grids, beat positions and analytical annotations. Time is measured in beats, quarter notes, MIDI ticks or other abstract units. The graphical domain concerns spatial representations that encode timelines from the first two domains visually: sheet music scans, graphical analyses, piano‑roll images and bounding boxes produced by an Optical Music Recognition (OMR) system. Images are not themselves temporal objects, but they map musical time – whether physical or logical – to spatial coordinates (pixels, millimetres), which is why we treat them as a temporal domain in their own right. Score‑following videos and animated graphical analyses (e.g. the spectromorphological analysis from Thoresen (2010) available on YouTube²) straddle the physical and graphical domains: they are physical renderings of graphical timelines, and our model accommodates this by linking both.

Similar conceptual triads appear in prior work, often using different terminology that emphasises a particular modality, production technique, representation or operationalisation of the same three domains (Table 1). Selfridge‑Field’s ‘sound or phonological context, [. . .] graphical context of notation, rational context’ (Selfridge‑Field, 1997, pp. 7–8) corresponds to our physical, graphical and logical domains. Gotham et al. (2023a) propose an equivalent triad of ‘performance’, ‘print’ and ‘logic’, focused on production techniques. Wiggins et al. (2010) and Liem et al. (2011), following Babbitt (1965), distinguish an acoustic, an auditory/perceptual and a graphemic domain. Our Figure 1 adapts Figure 1 of Liem et al. (2011) while adjusting the transformations between domains. We depart from the Babbitt‑derived triad in one respect: what they call the auditory/perceptual domain is, from a temporal standpoint, a subdomain of the physical; the cognitive processes they associate with it (score reading, listening, transcription) are relevant for engaging with all three domains rather than constituting a domain of their own.

A consequence of our triad is that music notation is both graphical and logical: a score is, in effect, an alignment between a graphical layout (where notes are placed on the page) and a logical structure (which events they represent and in what order). This does not entail that the transfer between the graphical and physical domains must always pass through a logical intermediary, as recent end‑to‑end approaches demonstrate (e.g. Jung et al., 2025). By clearly differentiating graphical from logical aspects, the three domains can add conceptual clarity to notation‑related applications such as score following, score annotation and score retrieval, each of which may focus on one aspect or the other, if not both (cf. Gotham et al., 2025).

2.2 Music synchronisation tasks

Navigating between corresponding timelines is central to both MIR and DM. As schematised in Figure 2, this takes two forms. Domain transfer converts a timeline from one domain to another (e.g. transcription converts audio to a symbolic score), producing two aligned timelines. Alignment identifies temporal equivalences between two existing timelines from the same or different domains. We subsume both under ‘music synchronisation’ (Arifi et al., 2003; Dannenberg and Raphael, 2006; Müller, 2021).

Figure 2

Music synchronisation underpins a wide range of applications: score following (Dorfer et al., 2016; Fremerey et al., 2008; Peter et al., 2025b), automatic music transcription (Benetos et al., 2019), score annotation (Ballester et al., 2025; Hentschel et al., 2025), expressive performance modelling (Berndt, 2021; Cancino‑ Chacón et al., 2018), automatic accompaniment (Cancino‑Chacón et al., 2023; Dannenberg and Raphael, 2006), version mapping (Grünbacher and Neuwirth, 2024) and cross‑format annotation transfer (Hu and Widmer, 2023). Within DM, there is growing interest in integrated data resources that connect diverse representations of the same music (e.g. Dreyfus et al., 2025).

Cross‑domain alignment typically requires projecting data into a shared feature representation (for instance, chroma features derived from both audio and symbolic data (Gómez, 2006)) or using deep learning to map different modalities into a common embedding space (Dorfer et al., 2018; Müller et al., 2019; Weiß and Peeters, 2021). Synchronisation algorithms such as Dynamic Time Warping (DTW) then establish temporal correspondences (Müller, 2021). The process is complicated by feature discrepancies between domains (Thomas et al., 2012), expressive performance variations (Arzt and Widmer, 2015; Dannenberg and Mukaino, 1988) and structural differences such as repeated sections and jumps that require unfolding before alignment can proceed (Agrawal et al., 2021; Brazier and Widmer, 2021; Fremerey et al., 2010; Peter et al., 2025a).

The critical point for our purposes is that every one of these tasks produces alignment data – a set of correspondences between events on different timelines – yet these data are typically stored in task‑specific, ad hoc formats (if stored at all), making them expensive to reproduce and difficult to build upon. No widely applicable standard for distributing these data in a reusable way yet exists.

2.3 Existing approaches and their limitations

Any music encoding format constitutes a model of musical time, at least implicitly. Research data that depend on music representations (such as score annotations, audio features and electroencephalography data aligned with auditory stimuli) typically adopt their time model or overlay their own, often equally implicit, model. Interoperability problems arise when these time models are not explicitly communicated.

Task‑specific formal models. Formal mathematical models of musical time have been developed for specific use cases. For instance, Foscarin’s (2020) model for automatic music transcription defines timelines in the physical and logical domains for contiguous events but does not generalise beyond this context.

Semantic Web approaches. The Timeline Ontology (Abdallah et al., 2006) and Segment Ontology (Fields et al., 2011) use Semantic Web technologies to model music events and temporal relations. The Timeline Ontology operates with two principal classes, ContinuousTimeLine and DiscreteTimeLine, each with respective time instants and intervals (adopted from the OWL‑Time Ontology³), connected by TimeLineMaps (UniformSamplingMap, ShiftMap, etc.). The Segment Ontology extends this with SegmentLine, Segment and SegmentLineMap. Despite their potential generality, these ontologies have been applied principally to audio recordings and signal processing (Cannam et al., 2010), as exemplified by the CALMA dataset (Page et al., 2017) and, more recently, the Music Encoding and Linked Data (MELD) framework (Dreyfus et al., 2025; Weigl and Page, 2017).

Storage formats. The match file format (Foscarin et al., 2022) stores aligned note events for score– performance pairs. Music Performance Markup (MPM) (Berndt, 2021) provides a toolbox and exchange format for alignment and annotation data across the logical, graphical and physical domains. IEEE 1599 (Baggi and Haus, 2009, 2013) is an XML standard for multimodal alignment that references external sources from all three domains and represents synchrony by projecting events onto a single abstract timeline called the ‘spine’.

De facto practices. Many datasets do not follow a fully specified format. Multimodal platforms such as Repovizz (Mayor et al., 2013) organise data streams along a shared physical timeline. In Optical Music Recognition, alignments between images and symbolic representations are typically encoded as ad hoc JSON or XML files enumerating bounding boxes (Calvo‑Zaragoza et al., 2020).

Despite these contributions, several gaps remain:

1. No cross‑domain coverage. No existing format or model covers all three temporal domains simultaneously. The Timeline and Segment ontologies focus on the physical domain, IEEE 1599 references all three but does not model their internal structure and the match file format is limited to scores and performances.

2. No reusable alignment artefact. There is no standard way to represent the alignment itself (the set of correspondences between events on different timelines) as a shareable, self‑describing artefact independent of the datasets it connects.

3. Ambiguous encoding conventions. Measure numbers, beat sizes and coordinate systems vary between datasets and are rarely documented, making data reuse error‑prone even within a single domain.

4. No provenance or certainty metadata. The agent, method and confidence level of alignment claims are almost never recorded, making it impossible to assess the reliability of existing alignments or to combine them systematically.

5. No runnable toolchain. No open‑source library exists for creating, validating and exchanging cross‑domain alignments with typed timelines and conversion maps.

3.1 Aligning a historical piano roll with a digital score

The Stanford University Piano Roll Archive (SUPRA) provides, for each historical Welte–Mignon roll, a high‑resolution scan, a file of hole‑punch positions, two MIDI files (one preserving the raw hole data, one with expressive dynamics) and a synthesised MP3 (Shi et al., 2019). Consider a researcher who wishes to build a player application that displays a digital score alongside the piano roll and plays the synthesised audio, while a cursor indicates the corresponding position in the scan (cf. Ballester et al., 2025). The challenge is that the five artefacts span all three temporal domains and use incommensurable coordinate systems: the scan measures 299,400 pixels in height; the raw MIDI file contains 30,092 events counted in ticks; the MP3 lasts approximately 573 seconds; and the annotated score counts 888 quarter‑note beats across 222 measures, carrying 1,074 harmonic labels. No documented alignment between these representations exists.

Figure 4 shows how we model this scenario. Each artefact becomes a timeline: a positive coordinate axis defined by an origin and a measuring unit. The scanned roll is a discrete graphical timeline (DGT1, the red dotted arrow at the bottom of Figure 4), measured in pixels. Although the timeline is discrete according to the scan, its C‑map (the red double line below DGT1) maps each coordinate to the material real‑world position in inches, a first demonstration of the ConversionMap concept. The annotated score is a continuous logical timeline (CLT1, the green solid arrow at the top), measured in quarter‑note coordinates that come as fractions for precision; a discrete logical C‑map (the green double line) additionally converts beats to measure numbers. The synthesised audio is a discrete physical timeline (DPT1, the blue dotted arrow), measured in samples, with a C‑map to seconds. The two MIDI files are discrete logical timelines (DLT1, DLT2, the green dotted arrows), measured in ticks (sometimes called pulses, an integer‑valued unit suited to quantised performance data), each carrying a C‑map to seconds. The distinction between discrete and continuous timelines within the same domain is not merely typographic: it determines the permissible arithmetic (integer vs real‑valued interpolation) and prevents silent precision loss when coordinates are transferred. Each of these C‑maps also permits the inverse conversion back to the timeline’s own unit, which proves highly convenient for addressing time points and intervals in whichever coordinate system is most natural for the task or data at hand.

Figure 4

Because only a portion of the scanned roll contains musical content (pixel rows 15,343–293,119), we model this region as a child timeline: a nested coordinate system (the shorter red dotted arrow inside DGT1’s box) that shares the parent’s measuring unit but defines its own origin. In the child, the first hole punch sits at coordinate 0 rather than at absolute pixel 15,343, simplifying all downstream references while preserving the relationship to the parent via a known offset. The C‑map on the parent reveals that the musical content spans 23.5 metres of paper, enabling data to be referenced not only in pixel coordinates but also in real‑world positions on the material roll in the archive.

To make all five artefacts mutually navigable, we collect them in a TimelineGroup – the black rectangle in Figure 4. By placing timelines into a group, we claim that they are commensurable – regardless of their underlying units or the presence of C‑maps. The group is an important tool that enables interpolation between timelines and allows one to obtain TimeStamps: cross‑sections showing the coordinates of all member timelines plus the respective values converted by every attached C‑map. Figure 4 shows a TimeStamp as blue dashed vertical lines through the group at a chosen coordinate, with interpolated values labelled on each timeline: a single query that returns positions in all five coordinate systems simultaneously. For our player application, a TimeStamp at each beat position provides exactly the data needed: the pixel row in the scan, the sample position in the audio and the tick offsets in both MIDI files. Note, however, that the timelines in this group are not genuinely aligned: the group provides interpolation‑based coordinate transfer, which gives a rough estimate but not a musically verified correspondence. To create a genuine alignment, one would match the scanned holes to processed MIDI events to score notes – a straightforward task for this particular example, given the data available, and a workflow that the model supports through MatchClaims (Section 3.3).

In sum, this example has rendered five artefacts across three temporal domains mutually navigable through a handful of typed containers and maps – including, crucially, the distinction between interpolated group membership and verified alignment that the model makes explicit from the outset.

3.2 Flow control in a school song arrangement

Consider a digitised double page of a school song arrangement: Un poquito cantas from the Chorissimo! Blue songbook (Weigele and Brecht, 2016). It contains repeat signs, a dal segno al coda structure and a percussion ostinato notated on a separate staff (Figure 5). A score‑following application must know which bars to play in what order and how the seven staff systems spread across two pages map onto the logical structure.

Figure 5

The graphical layout is a SegmentLine (DGT1, the red dotted arrow at the bottom of Figure 5): a sequence of contiguous Segments – that is, child timelines joined end to end. Each of the seven segments (seg1–seg7, visible as the red diamond connectors) corresponds to one staff system, with pixel widths ranging from 646 to 698 px. Each segment carries a single C‑map (the green double line above): a quick fix that simply divides the segment’s width into four equal parts for a rough mapping to measure numbers. A more precise mapping would be conceivable by creating an array of $x$‑coordinates of all measure boundaries, which could either constitute the segment line or actually be embedded as children in their respective system segments.

The logical structure (CLT1, the green solid arrow in the centre of Figure 5), measured in quarter‑note beats, introduces three new concepts. First, named Regions (curly braces above CLT1) partition the timeline into semantically meaningful parts without creating separate child timelines. Six Regions are visible: Intro, Core (repeated), Core (last time), Verse, Chorus and Coda; the Verse and Chorus Regions group further into the two different Core Regions. Second, Breaks (the red vertical lines crossing CLT1 and DGT1) void contiguity at specific coordinates: the break at the Coda mark prevents intervals from spanning the boundary between the main body and the Coda, correctly encoding that these sections are not temporally adjacent in performance. Third, Jumps (the dotted curved arrows below CLT1) create non‑linear contiguity: the jump from the end of the Chorus back to the Verse encodes the repeat sign; the jump from the end of the second pass to the Coda encodes the dal segno al coda.

Breaks and jumps are flow control events – a feature of our model that determines all possible flows we can compute for a given timeline. One of these is the Default FlowMap shown in Figure 5, which our model implementation computes automatically from the notated structure. Apart from such automatically computed flows, users can define custom flows as any sequence of interval events, including Regions. Different performances may follow different paths: one choir may take all repeats, while another may skip a verse. A FlowMap unfolds the timeline by slicing it at section boundaries and concatenating the slices into a new SegmentLine – the mechanism used at scale in the Beethoven example (Section 3.4).

A second logical timeline (CLT2, the green solid arrow on the right of Figure 5) represents the four‑bar percussion ostinato. Because the ostinato repeats cyclically throughout the song, it is associated with a RotationMap (the purple box labelled $f(n)\mapsto n\cdot \text{Ostinato}$): a periodic ConversionMap that wraps coordinates modulo the pattern length. The RotationMap acts like a generator, returning the ostinato as many times as the flow requires. Figure 5 shows two groups (the black rectangles): one enclosing CLT1 and DGT1 (the main body) and one enclosing CLT2 (the Ostinato). The blue dashed vertical lines in Figure 5 mark a TimeStamp at a chosen logical coordinate. FlowMaps may also be applied to groups: the idea here would be to unfold the ostinato (timeline or group) as many times as needed and to unfold the main body according to the appropriate flow, then either combine the timelines from the resulting groups into another group (if the lengths are known to match) or add them to each other as children (e.g. the unfolded CLT1 as a child to the unfolded CLT2, assuming the performance starts with the ostinato alone, and the unfolded DGT1 ‘body’ as a child to the unfolded DGT1 ‘ostinato’).

The flow‑control apparatus demonstrated here – at the modest scale of a 24‑measure school song – handles the full complexity of repeat signs, dal segno and cyclic ostinati without duplicating any underlying data. That the same mechanism applies without modification to a Classical string quartet movement is demonstrated in Section 3.4.

3.3 Transferring annotations between graphical analyses

Consider a researcher studying Lasse Thoresen’s spectromorphological analysis of Åke Parmerud’s acousmatic composition Les objets obscurs, composed in 1991. The same analysis was published in two versions: as a single raster image in Thoresen (2009) and as five separate images, each extended with a form‑analytical layer beneath, in Thoresen (2010). Both depict the same 150‑second audio excerpt, but with different pixel dimensions. An annotation placed on one image – say, the bounding box of sound object H – cannot simply be copied to the other: the pixel coordinates differ, the images are laid out differently and no documented mapping between them exists.

We model each published version as a discrete graphical timeline (DGT1 and DGT2, the red dotted arrows in Figure 6). Each image’s multi‑line layout is decomposed into a SegmentLine (the concept introduced in Section 3.2) with one Segment per visual system. DGT1 (the 2009 version) has five segments of equal length (975 pixels each, totalling 4,875 pixels); DGT2 (the 2010 version) has five segments of slightly varying length (864–867 pixels each, totalling 4,328 pixels). A C‑map attached to each timeline (the blue solid line labelled ‘SamplesToSeconds’ in Figure 6) maps its pixel coordinates to seconds on a shared continuous physical timeline representing the audio recording. For DGT1, a single linear C‑map maps 4,875 pixels to 150 seconds – again, a pragmatic decision that could be modelled more precisely if circumstances required it; for DGT2, five separate linear C‑maps each map one segment’s pixel length to 30 seconds.

Figure 6

For event H, whose pixel coordinates in DGT2’s second segment span 378 to 517 px, the associated TimeIntervalStamp (effectively a pair of TimeStamps marking start and end) records this pixel range together with the physical time in seconds (43.1”–47.9”) derived from the attached C‑map; DGT1 produces its own TimeIntervalStamp for event H’ (both visible as the coordinate tuples in Figure 6).

To assert that event H in DGT2 and event H in DGT1 represent the same sound object, we create a MatchClaim: an explicit claim connecting events across timelines (the peach box labelled ‘Match Claim’ in Figure 6). The MatchClaim carries structured metadata: the agent who made the claim, the decision criteria and the certainty level. In the simple version, the transfer is purely coordinate‑based: H’s pixel $x$‑coordinates in DGT2 are converted to seconds via the C‑map and then back to pixels in DGT1 via the inverse of DGT1’s C‑map. In a more elaborate version, one can also transfer the y‑coordinates, which represent event properties (the vertical extent of the sound object in the image). By defining a custom C‑map that interpolates between the $y$‑segments of DGT1 and DGT2 (visible as the vertical arrows and hollow coordinate wedges in Figure 6), the vertical positions are warped from one image to the other.

The same principle extends to any image that encodes time spatially (for instance, piano‑roll scans, sonograms or even the spiral notations of George Crumb’s Makrokosmos) by defining the appropriate mapping from pixel coordinates to physical or logical time coordinates. What appears to be a tricky problem – transferring information between two heterogeneous versions of the same graphical analysis – turns out to be conceptually straightforward thanks to the model’s core machinery: SegmentLines, C‑maps with inversion and MatchClaims with structured provenance.

3.4 Multimodal alignment at scale

This example, the most elaborate in our collection, demonstrates how the model scales to a complex, real‑world research scenario spanning all three temporal domains (Figure 7). A performance researcher has access to three multimodal recordings of the fourth movement of Beethoven’s String Quartet no. 4, op. 18/4, captured in a controlled environment with six microphones, motion‑capture sensors and derived audio features at varying sampling rates (Marchini et al., 2014). The underlying dataset catalogues well over a hundred signals per recording: audio, motion capture, bowing gestures and Essentia audio descriptors. Two digital score encodings are available: one from the Annotated Beethoven Corpus (ABC), version 2.6, which carries harmony and phrase analyses (Neuwirth et al., 2018), and one from the OpenScore String Quartet corpus (Gotham et al., 2023b), which encodes all four movements in a single file. OMR ground‑truth data provide pixel‑level bounding boxes for 3,190 note heads across 22 score pages (Shatri and Fazekas, 2021). The researcher’s goal: inter‑relate the harmony and phrase annotations with the performances and their sensor data while correctly handling the movement’s repeat structure.

Figure 7

Figure 7 is subdivided into three zones. The upper zone (white background, right) shows the Folded Score Group: CLT1 (the green solid arrow), the ABC v2.6 encoding with 878.5 quarter‑note beats across 13 atomic sections (A–M), and DGT1 (the red dotted arrow), the OMR ground‑truth data. The score contains repeats and alternative endings (the same kind of flow‑control structure modelled via Breaks and Jumps in Section 3.2), meaning that notated measures do not represent the performance order. The Default FlowMap (the lettered blocks at the right of the score group) specifies the traversal according to conventional unfolding logic. In the upper‑left corner sits the Multimodal Recording Group (Marchini et al., 2014), shown as a black rectangle. We show only a representative subset: five discrete physical timelines (blue dotted arrows) for the main modalities – recording, tonal features, low‑level features, rhythm features and motion capture – each with children (visible as stacked segments inside the labelled boxes) representing individual musicians. Every timeline carries a C‑map to seconds. The dataset contains two additional recording groups (‘Mechanical’ and ‘Exaggerated’) with a similar structure, which we omit from the figure for clarity.

The black rectangle in the middle shows the Unfolded Score Group: CLT1_unfolded (the green solid arrow, 1,116 quarter‑note beats) and DGT1_unfolded (the red dotted arrow, 135,198 pixels). The unfolded section sequence (A, A, B, B, C, D, E, D, . . ., M) now runs linearly, expanding the 878.5 quarter‑note beats to 1,116 quarters. The data stores (Notes, Labels) are carried along and multiplied by the unfolding operation.

The lilac background represents the first AlignmentBundle: a structure that combines the Multimodal Recording Group with the Unfolded Score Group via inter‑group MatchClaims. The orange MatchClaims box at the boundary between the two zones represents these claims, each asserting that a note event in the recording corresponds to one in the unfolded score. These event pairs resolve to AlignmentAnchors: coordinate pairs recording which point on the score corresponds to which point on the recording, according to the respective events’ start and end coordinates. At this point, harmony and phrase annotations can be transferred from the score to the motion‑sensor data and instrument‑pickup recordings: Those coinciding with a note can be transferred directly; the remaining ones would necessitate the creation of a WarpMap (see further below).

The lower zone (cyan background) represents a second, independent AlignmentBundle. It contains CLT2_unfolded (the green solid arrow, an earlier ABC v1.0 edition measured in floating‑point measures) and DPT16 (the blue dotted arrow, the Emerson String Quartet recording, 4’01.2”). Between them sits a MatchLine: an ordered sequence of MatchClaims shown as orange boxes ($\alpha$–$\kappa$), each within a dotted blue box representing its AlignmentAnchor. Each Greek letter corresponds to a Region of the unfolded score; the MatchLine thus communicates a coarse‑grained score–audio alignment encoded as a Flow CSV in the dataset.

Now, the key insight: the two AlignmentBundles are currently independent. However, CLT1_unfolded and CLT2_unfolded encode the same music – merely from different editions. By adding CLT2_unfolded to the Unfolded Score Group, we bridge the two bundles: all transferable data from the Marchini recordings becomes available to the Emerson timeline and vice versa. This cascading alignment – connecting independent datasets by aligning one timeline each – is the central payoff of the AlignmentBundle architecture. The practical consequence is that a single additional group membership – adding an independent score edition to an existing group – retroactively enriches every timeline already present in both bundles, without modifying any of the original data or alignment claims.

3.5 Encoding conceptual and temporal relationships in song genesis

The previous examples dealt with alignment as a fundamentally direct temporal relationship: events on different timelines correspond because they occur at equivalent points in time. Musicological research, however, often requires a broader notion of temporal correspondence. Consider a study of the genesis of Jimi Hendrix’s 1983 . . . (A Merman I Should Turn to Be) that compares three versions: a solo demo (CPT3), a band demo with Mitch Mitchell on drums (CPT2) and the studio recording on Electric Ladyland (CPT1). The three versions share a recognisable form – all have an Intro, several Verses and a Bridge – but sections are added, removed and restructured across versions. The instrumental middle section present in the studio version and the band demo is entirely absent from the solo demo. Some correspondences are merely conceptual (‘both have a Bridge’), while others are also temporal (‘these Bridge subsections can be aligned beat‑by‑beat’). How can these heterogeneous relationships be encoded in a single, queryable structure?

As introduced in Section 3.3, a MatchClaim is an explicit claim, issued by a human or algorithmic agent, positing that events from different timelines are equivalent or synchronous, carrying structured metadata on the agent, decision criteria and certainty. MatchClaims can be automatically extended into MatchGraphs through a coordinate that they share with one or multiple other MatchClaims. A MatchGraph is an auxiliary construct for creating a MatchStamp, which is the union of TimeStamps from the groups synchronously connected by a MatchGraph. However, in other cases, we may want to view MatchGraphs as hyperedges connecting multiple coordinates from different timelines. Critically, a MatchClaim may be either synchronous (asserting both structural and temporal equivalence, thereby producing AlignmentAnchors) or conceptual (asserting structural equivalence without temporal commitment). One can create MatchGraphs to include synchronous claims, conceptual claims or both; however, MatchStamps (like those shown as dotted blue lines in the figure) can only ever include TimeStamps from those groups where at least one MatchClaim is synchronous. Figure 8 shows 15 section‑level MatchGraphs (M1–M15), each formed from one or more MatchClaims. M1 (Intro) is purely conceptual: all three versions have an introduction, but they differ harmonically and temporally. M3 (Verse A1) is synchronous across all three versions, enabling beat‑level alignment. M12 (Instrumental Part) explicitly records a NOMATCH sentinel for the solo demo: a positive assertion that CPT3 has no corresponding section, rather than a mere absence of data. These sentinel values make structural omissions first‑class objects in the alignment structure, preventing ambiguity about whether a gap represents missing data or a genuine musical absence.

Figure 8

The conceptual/synchronous distinction proves its worth in the Bridge section. M5 asserts a conceptual MatchClaim between the Bridge sections of all three versions, but, because the sections differ in length (the solo demo has an extra opening measure; the studio version has additional iterations), no global synchronous alignment is warranted. However, four subsection‑level MatchClaims (M6–M9) are synchronous, yielding AlignmentAnchors that enable a fine‑grained chord comparison (excerpt shown in the bottom‑right rectangle of Figure 8). This reveals Hendrix’s harmonic evolution: the solo demo mostly uses a three‑chord progression (D–A–C), matching the fourth chord (G) only in the final iteration; the band demo introduces passing chords (G^7#9) absent from the other versions; and the studio version settles on a clean four‑chord cycle but extends the section with additional instrumental iterations. A MatchLine – an ordered sequence of AlignmentAnchors derived from these MatchClaims, sorted by coordinate on a chosen source timeline – serves as the input for WarpMap generation and enables systematic comparison across versions.

Throughout, the three primary timelines (TiLiA annotations of the audio recordings) remain immutable.⁴ The MatchClaims form an overlay that adds regions, sentinels and certainty metadata without modifying the source data – a separation of concerns that is essential when multiple analysts may produce competing alignment claims for the same material. The Hendrix example is the most interpretively rich of the five: it shows that the same machinery used for temporal alignment serves equally for encoding musicological judgements about structural correspondence, absence, and evolution across versions of a work.

5.1 Achievements

The TimeToAlign! model unifies the three temporal domains – graphical, logical and physical – under a single framework of typed timelines, ConversionMaps and MatchClaims. The five examples in Section 3 demonstrate that this framework handles scenarios ranging from simple two‑image annotation transfer to a large‑scale multimodal alignment spanning all three domains and that the same concepts apply uniformly across all of them.

A recurring theme across the examples is the role of nesting. Child timelines, Segments and SegmentLines allow complex structures to be composed from simple parts, while the coordinate arithmetic that accompanies nesting (offset transfer between parent and child, interpolation within groups, WarpMaps across groups) is handled by the framework rather than by the user. The Beethoven example illustrates this most directly: each of the recording group’s physical timelines carries between four and six children, yet a single TimeStamp query traverses the entire hierarchy without the user specifying how coordinates are to be transferred.

A further consequence of the model’s design is that alignment structures can exist independently of the original data they describe. Because a timeline is defined solely by an origin, a measuring unit and a length, it can be constructed from metadata alone – a manifest listing track durations, an image catalogue recording pixel dimensions or simply a known recording length – without requiring access to the underlying files. The resulting time skeleton carries the full alignment topology (MatchClaims, TimelineGroups, AlignmentBundles) and can be shared, archived and reused even when the original data are large, proprietary or subject to copyright restrictions. When new data become available – say, a fresh recording or a new labelset – they attach to the existing skeleton via a single additional MatchClaim (minimally), inheriting all previously established connections (as demonstrated in Section 3.4).

The timetoalign library moves these contributions from specification to a working tool. Typed timelines and loader plugins lower the barrier to constructing aligned representations: the SUPRA example (Section 3.1), which connects five artefacts across three domains, requires fewer than 10 lines of loader code and one TimelineGroup constructor. The planned Parquet‑based exchange format will enable publishing alignment data as self‑describing, shareable artefacts independent of the datasets they connect – addressing one of the central gaps identified in Section 2.3.

5.2 Limitations

The model and library have several limitations that should be stated plainly. First, automatic alignment algorithms (DTW, sequence alignment, deep learning–based matchers) are external to the library: timetoalign provides the data model for their inputs and outputs, not the algorithms themselves. Integration with existing toolkits such as Sync Toolbox (Müller et al., 2021) is a natural next step. Second, the logical domain as currently modelled is tailored to Common Western Music Notation: concepts such as measures, beats and time signatures are first‑class objects, whereas other paradigms are not yet represented. The three‑domain triad itself is not Western‑specific (graphical, logical and physical time transcend any specific musical tradition) and extending the logical domain to accommodate alternative time units should be straightforward, provided they can be expressed as discrete or continuous coordinate axes. Third, no user study or adoption data exist yet. Fourth, the library currently requires Python programming; GUI tooling for non‑programmer users is not yet available, although integration with annotation tools such as TiLiA is under consideration. Finally, the data‑exchange format is not yet implemented.

5.3 Relation to existing standards

Our model draws heavily on several existing paradigms. The Timeline Ontology (Abdallah et al., 2006) and Segment Ontology (Fields et al., 2011) provide the closest structural antecedents: their ContinuousTimeLine and DiscreteTimeLine classes, TimeLineMaps, SegmentLine and Segment all have direct counterparts in our model. We extend this paradigm in three directions: the graphical domain (yielding six timeline types rather than two), flow control (Breaks, Jumps, FlowMaps) and MatchClaim objects carrying provenance and certainty metadata. The two approaches are complementary: the ontologies provide a semantic foundation for Linked Data integration, while our model provides a computational framework for typed coordinate transfer.

IEEE 1599 (Baggi and Haus, 2013) shares the goal of multimodal synchronisation but organises all connections through a single abstract ‘spine’ that generally models the score. Our MatchLine concept addresses a limitation of this design: because MatchClaims and AlignmentAnchors are modelled in their own right – with metadata, uncertainty values and flexible assembly – a MatchLine can be constructed for any reference timeline, effectively creating a purpose‑specific spine. Any timeline that is commensurable with two others can serve as an intermediary, and the connection topology is determined by the data rather than prescribed by the format.

The match file format (Foscarin et al., 2022) encodes aligned note events for score–performance pairs, always pairing events on the two aligned sides (or using sentinels for unmatched notes). Our model generalises this in two ways: a MatchClaim may assert equivalence between two events, between an event and its projection onto another timeline (via C‑maps), or – though less informatively – between bare coordinates, and MatchClaims need not imply synchrony at all, catering to musicological research on structural correspondence without temporal commitment (as demonstrated in Section 3.5).

MPM (Berndt, 2021) covers logical‑to‑physical alignment with detailed performance markup. Our model is more general (covering all three domains) but less specialised for the fine‑grained performance parameters (articulation, dynamics, rubato maps) that MPM provides. More generic alignment structures found in OMR datasets (Calvo‑Zaragoza et al., 2020) inspired our treatment of graphical timelines and cross‑domain bounding‑box transfer.

As a consequence of these affinities, we envisage not only ingesting data from these formats – which the implementation already does to some extent – but also converting to them, enabling round‑trip interoperability with established toolchains.

5.4 Future directions

Several directions merit further development. The most immediate is the Parquet‑based exchange format, which will allow alignment data to be published and reused independently of the datasets it connects. We plan to translate existing alignment datasets into this format and to make them available as a shared specimen collection; we invite the community to use these resources and to suggest additional datasets for inclusion. Integration with automatic alignment algorithms – particularly Sync Toolbox (Müller et al., 2021) for DTW‑based alignment and deep‑learning approaches for cross‑modal matching – would make the library useful as an end‑to‑end pipeline rather than a data‑model layer alone. A formal ontology mapping (RDF/JSON‑LD serialisation of the alignment data) would bridge the gap to Semantic Web infrastructure and enable federated queries across distributed alignment datasets. Application to non‑Western and contemporary repertoires would test the model’s generality – particularly the logical domain’s current orientation towards Common Western Music Notation – and likely prompt useful extensions. Finally, GUI tooling would lower the barrier for musicologists who do not program.