Time to Align! Modelling Musical Timelines for Music Information Retrieval and Digital Musicology
By: Johannes Hentschel, Axel Berndt, Carlos Cancino-Chacón, Simon Dixon, Anne Foo, Mark Gotham, Patricia Hu, Maik Köster, Felipe D. Martins, Davide A. Mauro, Meinard Müller, Markus Neuwirth, Alexander Pacha, Kevin R. Page, Silvan Peter, Egor Polyakov, Laurent Pugin, David M. Weigl, Christof Weiß and Gerhard Widmer
References
- Abdallah, S. A., Raimond, Y., and Sandler, M. (2006).
An ontology‑based approach to information management for music analysis systems . In Proceedings of the 120th Convention of the Audio Engineering Society (AES), Paris, France. - Agrawal, R., Wolff, D., and Dixon, S. (2021). Structure‑aware audio‑to‑score alignment using progressively dilated convolutional neural networks. In ICASSP 2021–2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Toronto, Ontario, Canada (pp. 571–575).
- Arifi, V., Clausen, M., Kurth, F., and Müller, M. (2003). Automatic synchronization of music data in score‑, MIDI‑ and PCM‑format. In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), Baltimore, Maryland, USA.
- Arzt, A., and Widmer, G. (2015). Real‑time music tracking using multiple performances as a reference. In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), Málaga, Spain (pp. 357–363).
- Babbitt, M. (1965). The use of computers in musicological research. Perspectives of New Music, 3(2), 74. 10.2307/832505.
- Baggi, D. L., and Haus, G. (Eds.). (2009). Journal of Multimedia, Special Issue: The New Standard IEEE 1599 – Interacting with Music Contents by XML Symbols, Vol. 4.
- Baggi, D. L., and Haus, G. M. (2013).
The IEEE 1599 standard . In D. L. Baggi and G. M. Haus (Eds.), Music Navigation with Symbols and Layers (1st ed., pp. 1–20). Wiley. - Ballester, C., Bacot, B., Bigo, L., Borsan, V. N., Couturier, L., Déguernel, K., Dinel, Q., Feisthauer, L., Frieler, K., Gotham, M., Groult, R., Hentschel, J., d’Hooge, A., Le, D.‑V.‑T., Levé, F., Maccarini, F., Maričić, I., Micchi, G., Müller, M. … Giraud, M. (2025). Interacting with annotated and synchronized music corpora on the Dezrann web platform. Transactions of the International Society for Music Information Retrieval, 8(1), 121–139. 10.5334/tismir.212.
- Benetos, E., Dixon, S., Duan, Z., and Ewert, S. (2019). Automatic music transcription: An overview. IEEE Signal Processing Magazine, 36(1), 20–30. 10.1109/msp.2018.2869928.
- Berndt, A. (2021). The music performance markup format and ecosystem. In Proceedings of the 22nd International Society for Music Information Retrieval Conference (ISMIR), Online (pp. 50–57). 10.5281/zenodo.5624429.
- Brazier, C., and Widmer, G. (2021). Handling structural mismatches in real‑time opera tracking. In Proceedings of the European Signal Processing Conference (EUSIPCO), Dublin, Ireland (pp. 366–370). 10.23919/EUSIPCO54536.2021.9616109.
- Calvo‑Zaragoza, J., Hajič, J.
Jr. , and Pacha, A. (2020). Understanding optical music recognition. ACM Computing Surveys, 53(4), 1–35. 10.1145/3397499. - Cancino‑Chacón, C. E., Grachten, M., Goebl, W., and Widmer, G. (2018). Computational models of expressive music performance: A comprehensive and critical review. Frontiers in Digital Humanities, 25(5). 10.3389/fdigh.2018.00025.
- Cancino‑Chacón, C. E., Peter, S., Hu, P., Karystinaios, E., Henkel, F., Foscarin, F., Varga, N., and Widmer, G. (2023). The ACCompanion: Combining reactivity, robustness, and musical expressivity in an automatic piano accompanist. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), Macao, China (pp. 5779–5787). 10.24963/ijcai.2023/641.
- Cannam, C., Sandler, M., Jewell, M. O., Rhodes, C., and d’Inverno, M. (2010). Linked data and you: Bringing music research software into the semantic web. Journal of New Music Research, 39(4), 313–325. 10.1080/09298215.2010.522715.
- Dannenberg, R. B., and Mukaino, H. (1988). New techniques for enhanced quality of computer accompaniment. In Proceedings of the International Computer Music Conference (ICMC), Cologne, Germany (pp. 243–249).
- Dannenberg, R. B., and Raphael, C. (2006). Music score alignment and computer accompaniment. Communications of the ACM, Special Issue: Music Information Retrieval, 49(8), 38–43. 10.1145/1145287.1145311.
- Dorfer, M., Arzt, A., and Widmer, G. (2016). Towards score following in sheet music images. In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), New York City, New York, USA (pp. 789–795).
- Dorfer, M., Hajič, Jr., J., Arzt, A., Frostel, H., and Widmer, G. (2018). Learning audio‑sheet music correspondences for cross‑modal retrieval and piece identification. Transactions of the International Society for Music Information Retrieval (TISMIR), 1(1), 22–31. 10.5334/tismir.12.
- Dreyfus, L., Lewis, D., and Page, K. (2025). A digital companion for musicological scholarship: The Lohengrin TimeMachine. Journal of New Music Research, 53(3–4), 297–311. 10.1080/09298215.2025.2487100.
- Fields, B., Page, K., De Roure, D., and Crawford, T. (2011). The segment ontology: Bridging music‑generic and domain‑ specific. In 2011 IEEE International Conference on Multimedia and Expo, Barcelona, Spain (pp. 1–6). IEEE.
- Foscarin, F. (2020). The Musical Score: A Challenging Goal for Automatic Music Transcription. PhD thesis, Centre d’études et de recherche en informatique et communication.
- Foscarin, F., Karystinaios, E., Peter, S. D., Cancino‑Chacón, C., Grachten, M., and Widmer, G. (2022). The match file format: Encoding alignments between scores and performances. In Proceedings of the Music Encoding Conference (MEC), Halifax, Nova Scotia, Canada (pp. 52–60).
- Fremerey, C., Müller, M., Kurth, F., and Clausen, M. (2008). Automatic mapping of scanned sheet music to audio recordings. In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), Philadelphia, PA, USA (pp. 413–418).
- Fremerey, C., Müller, M., and Clausen, M. (2010). Handling repeats and jumps in score‑performance synchronization. In Proceedings of the 11th International Society for Music Information Retrieval Conference (ISMIR) (pp. 243–248).
- Gómez Gutiérrez, E. (2006). Tonal Description of Music Audio Signals. PhD thesis. Universitat Pompeu Fabra.
- Gotham, M., Bemman, B., and Vatolkin, I. (2025). Towards an ‘everything corpus’: A framework and guidelines for the curation of more comprehensive multimodal music data. Transactions of the International Society for Music Information Retrieval, 8(1), 70–92. 10.5334/tismir.228.
- Gotham, M., Hentschel, J., Couturier, L., Dyke‑Aylen, N., Rohrmeier, M., and Giraud, M. (2023a). The ‘Measure Map’: An inter‑operable standard for aligning symbolic music. In Proceedings of the 10th International Conference on Digital Libraries for Musicology, Milan, Italy (pp. 91–99). ACM.
- Gotham, M., Redbond, M., Bower, B., and Jonas, P. (2023b). The “OpenScore String Quartet” corpus. In Proceedings of the 10th International Conference on Digital Libraries for Musicology, Milan, Italy (pp. 49–57). ACM.
- Grünbacher, P., and Neuwirth, M. (2024). Towards feature‑based versioning for musicological research. In Proceedings of the 18th International Working Conference on Variability Modelling of Software‑Intensive Systems, Bern, Switzerland (pp. 77–82).
- Hentschel, J., Rammos, Y., Neuwirth, M., and Rohrmeier, M. (2025). A corpus and a modular infrastructure for the empirical study of (an)notated music. Scientific Data, 12(1), 685. 10.1038/s41597-025-04976-z.
- Hu, P., and Widmer, G. (2023). The Batik‑plays‑Mozart corpus: Linking performance to score to musicological annotations. arXiv preprint arXiv:2309.02399.
- Jung, J., Kim, D., Lee, S., Cho, S., Soh, H., Bukey, I., Donahue, C., and Jeong, D. (2025). Unified cross‑modal translation of score images, symbolic music, and performance audio. arXiv preprint.
- Liem, C. C. S., Müller, M., Eck, D., Tzanetakis, G., and Hanjalic, A. (2011).
The need for music information retrieval with user‑centered and multimodal strategies . In Proceedings of the International ACM Workshop on Music Information Retrieval with User‑Centered and Multimodal Strategies (MIRUM), Scottsdale, Arizona, USA (pp. 1–6). - Marchini, M., Ramirez, R., Papiotis, P., and Maestre, E. (2014). The sense of ensemble: A machine learning approach to expressive performance modelling in string quartets. Journal of New Music Research, 43(3), 303–317. 10.1080/09298215.2014.922999.
- Martins, F., and Gotham, M. (2023). “TiLiA”: A timeline annotator for all. 25th International Conference on Human‑Computer Interaction (HCI International 2023), Copenhagen, Denmark. 10.5281/ZENODO.14779020.
- Mayor, O., Llimona, Q., Marchini, M., Papiotis, P., and Maestre, E. (2013). repoVizz: A framework for remote storage, browsing, annotation, and exchange of multi‑modal data. In Proceedings of the 21st ACM International Conference on Multimedia, Barcelona, Spain (pp. 415–416). ACM.
- Müller, M. (2021). Fundamentals of Music Processing: Using Python and Jupyter Notebooks (2nd ed.). Springer Verlag.
- Müller, M., Arzt, A., Balke, S., Dorfer, M., and Widmer, G. (2019). Cross‑modal music retrieval and applications: An overview of key methodologies. IEEE Signal Processing Magazine, 36(1), 52–62. 10.1109/msp.2018.2868887.
- Müller, M., Özer, Y., Krause, M., Prätzlich, T., and Driedger, J. (2021). Sync toolbox: A Python package for efficient, robust, and accurate music synchronization. Journal of Open Source Software, 6(64), 3434. 10.21105/joss.03434.
- Neuwirth, M., Harasim, D., Moss, F. C., and Rohrmeier, M. (2018). The annotated Beethoven corpus (ABC): A dataset of harmonic analyses of all Beethoven string quartets. Frontiers in Digital Humanities, 5, 1–5. 10.3389/fdigh.2018.00016.
- Page, K. R., Bechhofer, S., Fazekas, G., Weigl, D. M., and Wilmering, T. (2017). Realising a layered digital library: Exploration and analysis of the live music archive through linked data. In 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL), Toronto, Ontario, Canada (pp. 1–10).
- Peter, S. D., Hu, P., and Widmer, G. (2025a). How to infer repeat structures in MIDI performances. In Proceedings of the Music Encoding Conference (MEC), London, United Kingdom.
- Peter, S. D., Hu, P., and Widmer, G. (2025b). Pairing real‑time piano transcription with symbol‑level tracking for precise and robust score following. In Proceedings of the 22nd Sound and Music Computing Conference (SMC), Graz, Austria.
- Selfridge‑Field, E. (1997).
Introduction: Describing musical information . In E. Selfridge‑Field (Ed.), Beyond MIDI: The Handbook of Musical Codes (pp. 3–38). MIT Press. - Shatri, E., and Fazekas, G. (2021). DoReMi: First glance at a universal OMR dataset. arXiv preprint.
https://ui.adsabs. harvard.edu/link_gateway/2021arXiv210707786S/doi:10.48550/arXiv.2107.07786 . - Shi, Z., Sapp, C., Arul, K., McBride, J., and Smith
III , J. O. (2019). SUPRA: Digitizing the Stanford University Piano Roll Archive. In Proceedings of the 20th International Society for Music Information Retrieval Conference (ISMIR), Delft, Netherlands (pp. 517–523). 10.5281/zenodo.3527858. - Thomas, V., Fremerey, C., Müller, M., and Clausen, M. (2012).
Linking sheet music and audio – Challenges and new approaches . In M. Müller, M. Goto, and M. Schedl (Eds.), Multimodal Music Processing, volume 3 of Dagstuhl Follow‑Ups (pp. 1–22). Schloss Dagstuhl–Leibniz‑Zentrum für Informatik. 10.4230/DFU.Vol3.11041.1. - Thoresen, L. (2009). Sound‑objects, values and characters in Åke Parmerud’s Les objets obscurs, 3rd section. Organised Sound, 14(3), 310–320. 10.1017/s1355771809990124.
- Thoresen, L. (2010). Form‑building patterns and metaphorical meaning. Organised Sound, 15(2), 82–95. 10.1017/s1355771810000075.
- Weigele, K. K., and Brecht, K. (Eds.). (2016). Chorissimo! Blue: Chorbuch für die Schule. Carus, Stuttgart.
- Weigl, D., and Page, K. (2017). A framework for distributed semantic annotation of musical score: Take it to the bridge! In Proceedings of the 18th International Society for Music Information Retrieval Conference (ISMIR), Suzhou, China.
- Weiß, C., and Peeters, G. (2021). Training deep pitch‑class representations with a multi‑label CTC loss. In Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), Online (pp. 754–761). 10.5281/zenodo.5624359.
- Wiggins, G., Müllensiefen, D., and Pearce, M. (2010). On the non‑existence of music: Why music theory is a figment of the imagination. Musicae Scientiae, 5, 231–255. 10.1177/10298649100140S110.
DOI: https://doi.org/10.5334/tismir.296 | Journal eISSN: 2514-3298
Language: English
Page range: 384 - 404
Submitted on: Jul 1, 2025
Accepted on: Apr 21, 2026
Published on: Jul 23, 2026
Published by: Ubiquity Press
In partnership with: Paradigm Publishing Services
Keywords:
© 2026 Johannes Hentschel, Axel Berndt, Carlos Cancino-Chacón, Simon Dixon, Anne Foo, Mark Gotham, Patricia Hu, Maik Köster, Felipe D. Martins, Davide A. Mauro, Meinard Müller, Markus Neuwirth, Alexander Pacha, Kevin R. Page, Silvan Peter, Egor Polyakov, Laurent Pugin, David M. Weigl, Christof Weiß, Gerhard Widmer, published by Ubiquity Press
This work is licensed under the Creative Commons Attribution 4.0 License.