Figure 1
Three temporal domains, each associated with a human organ (brain, eye and ear) and typical file formats. Arrows with italic captions represent activities of transfer from one domain to another. White rectangles name select MIR and DM applications situated between two domains or (in the middle) potentially involving all three.
Table 1
Disentangled terminology concerning the three domains.
| Domain | Modality | Production techniques | Representation | Operationalisations |
|---|---|---|---|---|
| Graphical | Seeing (static, architectural), sense of touch, spatial thinking | Printing, sketching, drawing, encoding visually | Visual | Raster images (coordinate system), shapes |
| Logical | Conceptualising, understanding, reading | Encoding symbolically, representing, modelling | Symbolic | Musical objects, notational objects |
| Physical | Hearing, seeing (dynamic in time, directional) | Performing, editing/producing, animating | Audio‑visual, sound | Audio features, sound studies, acoustic modelling |
Figure 2
Schema showing transfer and alignment across domains. Transfer converts a timeline from one domain to another (left‑to‑right arrows); alignment identifies temporal equivalence between two timelines from the same or different domains (two‑headed arrows). Both may use an intermediary representation (‘projection’) and one or several algorithms (rectangles).
Figure 3
Visual representation of our model’s core components: timelines, ConversionMaps (C‑maps) and parent–child relationships.
Table 2
Glossary of key terms in the TimeToAlign! model. This table collects the terms introduced across Sections 3.1–3.5; the full glossary, including additional terms, is available on the documentation homepage.
| Term | Definition |
|---|---|
| AlignmentAnchor | A coordinate pair associating one coordinate on timeline A with one on timeline B; a neutral record with no claim semantics. |
| AlignmentBundle | A collection of TimelineGroups with inter‑group match connections; provides coordinate transfer via offset arithmetic, interpolation and WarpMaps. |
| Break | A control event that voids contiguity at its instant; intervals cannot span a break. |
| Child timeline | A timeline nested within a parent, sharing its measuring unit but defining its own origin. |
| Commensurability | Two timelines are commensurable if connected by a chain of C‑maps, membership in the same TimelineGroup or cross‑group MatchClaims. |
| ConversionMap (C‑map) | A typed function attached to a timeline that maps each coordinate to exactly one value (coordinate, label or filename); also permits inverse conversion back to the timeline’s own unit. |
| Coordinate | A positive number representing distance from a timeline’s origin. |
| Event | Anything associated with a timeline via instants (InstantEvent or TimeIntervalEvent). |
| FlowMap | A sequence of TimeIntervals specifying a particular path through a score’s control structure (repeats, jumps); not itself a timeline. Used for unfolding. |
| Instant | A specific point on a timeline, associated with a coordinate. |
| Jump | A control event linking a ‘jump‑from’ to a ‘jump‑to’ instant, creating non‑linear contiguity. |
| MatchClaim | A claim of equivalence between events on different timelines, with provenance metadata (agent, method, certainty). May be synchronous (producing AlignmentAnchors) or conceptual (structural equivalence without temporal commitment). |
| MatchGraph | An auxiliary structure formed by extending MatchClaims through shared coordinates; may be viewed as a hyperedge connecting multiple coordinates from different timelines. |
| MatchLine | An ordered sequence of AlignmentAnchors derived from MatchClaims, sorted by coordinate on a source timeline; input for WarpMap generation. |
| MatchStamp | The union of TimeStamps from groups synchronously connected by a MatchGraph. |
| NOMATCH | A sentinel value explicitly encoding the absence of a corresponding event. |
| Region | A named part of a timeline defined by a TimeInterval (e.g. ‘Verse’, ‘Bridge’). |
| RotationMap | A periodic ConversionMap that wraps coordinates modulo a pattern length (e.g. a looped ostinato). |
| Segment | A child timeline that is contiguous with its siblings within a parent timeline. |
| SegmentLine | A parent timeline containing only contiguous segments. |
| Timeline | A positive coordinate axis defined by an origin and a measuring unit; continuous (real) or discrete (integer). |
| TimeInterval | A left‑inclusive, right‑exclusive interval defined by a start and an end instant. |
| TimelineGroup | A container for timelines claimed to be commensurable; placing timelines in a group asserts commensurability regardless of underlying units. Coordinates transfer via interpolation. |
| TimeIntervalStamp | The interval analogue of a TimeStamp: a pair of TimeStamps marking the start and end of an interval, recording its extent in all connected coordinate systems. |
| TimeStamp | A cross‑section through a timeline or group at a given coordinate, capturing positions in all connected systems plus values converted by all attached C‑maps. |
| WarpMap | An interpolation‑based coordinate mapping derived from a MatchLine; enables non‑linear coordinate transfer between TimelineGroups. |
Figure 4
Modelling the alignment between a historical piano roll (DGT1, red dotted arrow), its region of musical content (DGT1 child), two MIDI files (DLT1, DLT2, green dotted arrows), a synthesised audio recording (DPT1, blue dotted arrow) and an annotated score (CLT1, green solid arrow). C‑maps appear as thin, double lines coloured by their target domain. The black rectangle denotes the TimelineGroup; the blue dashed vertical line shows a TimeStamp cross‑section at a chosen coordinate.
Figure 5
Modelling flow control in a school song arrangement. DGT1 (bottom, red dotted arrow): Discrete graphical timeline representing two scanned pages decomposed into seven staff‑system segments (red diamond connectors). CLT1 (centre, green solid arrow): Continuous logical timeline representing the arrangement’s 24 measures, with named Regions (curly braces: Intro, Verse, Chorus, Coda), Breaks (red vertical lines) and Jumps (dotted curved arrows) encoding the repeat signs and dal segno al coda structure. CLT2 (right, green solid arrow): Continuous logical timeline representing a four‑bar percussion ostinato mapped via a RotationMap (purple box). Two groups (black rectangles) enclose the main body and the ostinato, respectively. The Default FlowMap (purple box, left) shows the traversal path. Blue dashed vertical lines show a TimeStamp cross‑section.
Figure 6
Transferring a graphically defined event H from one image to another. DGT1 and DGT2 (red dotted arrows) are discrete graphical timelines with five segments each (red diamond connectors), representing two published versions of the same spectromorphological analysis. C‑maps (blue solid lines) convert pixel coordinates to seconds on a shared continuous physical timeline. The peach box labelled ‘Match Claim’ represents the MatchClaim connecting event H across the two images; blue vertical lines through the box show the TimeIntervalStamps of the event’s temporal boundaries.
Figure 7
Multimodal alignment scenario for the fourth movement of Beethoven’s String Quartet no. 4, Op. 18/4. The figure is divided into three zones. Upper zone (white background): the Folded Score Group (right, containing CLT1 and DGT1) and the Multimodal Recording Group (left, containing five blue dotted arrows for recording and feature timelines, each with stacked children). The Default FlowMap (lettered blocks A, A, B, B, C . . . M) shows the traversal path. Middle zone (lilac background): the Unfolded Score Group (CLT1_unfolded, DGT1_unfolded), representing AlignmentBundle #1. The orange MatchClaims box connects the Recording Group to the unfolded score. Lower zone (cyan background): AlignmentBundle #2 with CLT2_unfolded (green solid arrow), the MatchLine (orange boxes – in dotted blue AlignmentAnchor rectangles) and DPT16 (the Emerson String Quartet recording, blue dotted arrow).
Figure 8
Conceptual and temporal alignment of formal analyses for three versions of Hendrix’s 1983 . . . (A Merman I Should Turn to Be): a solo demo (CPT3), a band demo (CPT2) and the studio recording (CPT1). Peach rectangles represent MatchGraphs (M1–M15), each formed from one or more MatchClaims; those marked with X are synchronous as well as conceptual. Dashed blue lines indicate MatchStamps derived from synchronous MatchClaims. The lower‑right inset shows how synchronous alignment of Bridge subsections (M6–M9) enables fine‑grained chord comparison.