Piano Sheet Music Identification Using Dynamic N-gram Fingerprinting

Daniel Yang; T. J. Tsai

doi:10.5334/tismir.70

Abstract

This article introduces a method for large-scale retrieval of piano sheet music images. We study this problem in two different scenarios: camera-based sheet music identification and MIDI-sheet image retrieval. Our proposed method combines bootleg score features with a novel hashing scheme called dynamic N-gram fingerprinting. This hashing scheme ensures that every fingerprint is discriminative enough to warrant a table lookup, which improves both retrieval accuracy and runtime. On experiments using all piano sheet music images in the IMSLP database, the proposed method achieves >0.8 mean reciprocal rank with sub-second runtimes. As a practical application, we use our system to find matches between the Lakh MIDI dataset and IMSLP, which augments the IMSLP sheet music data with symbolic music information for a subset of pieces. We release our code and Lakh-IMSLP matches to facilitate future study.

References

1Achankunju, S. P. (2018). Music search engine from noisy OMR data. In International Workshop on Reading Music Systems, pages 23–24.
Back to article
2Babenko, A., and Lempitsky, V. (2015). Aggregating local deep features for image retrieval. In Proceedings of the IEEE International Conference on Computer Vision, pages 1269–1297. DOI: 10.1109/ICCV.2015.150
Back to article
3Calvo-Zaragoza, J., Hajič, J., Jr. and Pacha, A. (2020). Understanding optical music recognition. ACM Computing Surveys (CSUR), 53(4): 1–35. DOI: 10.1145/3397499
Back to article
4Calvo-Zaragoza, J., Toselli, A. H., and Vidal, E. (2018). Probabilistic music-symbol spotting in handwritten scores. In IEEE International Conference on Frontiers in Handwriting Recognition, pages 558–563. DOI: 10.1109/ICFHR-2018.2018.00103
Back to article
5Damm, D., Fremerey, C., Kurth, F., Müller, M., and Clausen, M. (2008). Multimodal presentation and browsing of music. In Proceedings of the International Conference on Multimodal Interfaces, pages 205–208. DOI: 10.1145/1452392.1452436
Back to article
6Dorfer, M., Arzt, A., and Widmer, G. (2016). Towards score following in sheet music images. In Proceedings of the International Society for Music Information Retrieval Conference, pages 789–795.
Back to article
7Dorfer, M., Arzt, A., and Widmer, G. (2017). Learning audio-sheet music correspondences for score identification and offline alignment. In Proceedings of the International Society for Music Information Retrieval Conference, pages 115–122.
Back to article
8Dorfer, M., Hajič, J., Jr., Arzt, A., Frostel, H., and Widmer, G. (2018a). Learning audio-sheet music correspondences for cross-modal retrieval and piece identification. Transactions of the International Society for Music Information Retrieval, 1(1): 22–33. DOI: 10.5334/tismir.12
Back to article
9Dorfer, M., Henkel, F., and Widmer, G. (2018b). Learning to listen, read, and follow: Score following as a reinforcement learning game. In Proceedings of the International Society for Music Information Retrieval Conference, pages 784–791.
Back to article
10Dorfer, M., Schlüter, J., Vall, A., Korzeniowski, F., and Widmer, G. (2018c). End-to-end crossmodality retrieval with CCA projections and pairwise ranking loss. International Journal of Multimedia Information Retrieval, 7(2): 117–128. DOI: 10.1007/s13735-018-0151-5
Back to article
11Fremerey, C., Clausen, M., Ewert, S., and Müller, M. (2009). Sheet music-audio identification. In Proceedings of the International Society for Music Information Retrieval Conference, pages 645–650.
Back to article
12Fremerey, C., Müller, M., and Clausen, M. (2010). Handling repeats and jumps in score-performance synchronization. In Proceedings of the International Society for Music Information Retrieval Conference, pages 243–248.
Back to article
13Fremerey, C., Müller, M., Kurth, F., and Clausen, M. (2008). Automatic mapping of scanned sheet music to audio recordings. In Proceedings of the International Conference on Music Information Retrieval, pages 413–418.
Back to article
14Hajič, J., Kolárová, M., Pacha, A., and Calvo-Zaragoza, J. (2018). How current optical music recognition systems are becoming useful for digital libraries. In Proceedings of the 5th International Conference on Digital Libraries for Musicology, pages 57–61. DOI: 10.1145/3273024.3273034
Back to article
15Henkel, F., Balke, S., Dorfer, M., and Widmer, G. (2019). Score following as a multimodal reinforcement learning problem. Transactions of the International Society for Music Information Retrieval, 2(1): 67–81. DOI: 10.5334/tismir.31
Back to article
16Izmirli, Ö., and Sharma, G. (2012). Bridging printed music and audio through alignment using a midlevel score representation. In Proceedings of the International Society for Music Information Retrieval Conference, pages 61–66.
Back to article
17Kurth, F., Müller, M., Fremerey, C., Chang, Y.-H., and Clausen, M. (2007). Automated synchronization of scanned sheet music with audio recordings. In Proceedings of the International Conference on Music Information Retrieval, pages 261–266.
Back to article
18Malik, R., Roy, P. P., Pal, U., and Kimura, F. (2013). Handwritten musical document retrieval using music-score spotting. In IEEE International Conference on Document Analysis and Recognition, pages 832–836. DOI: 10.1109/ICDAR.2013.170
Back to article
19Radenovic, F., Tolias, G., and Chum, O. (2016). CNN image retrieval learns from BoW: Unsupervised fine-tuning with hard examples. In Proceedings of the European Conference on Computer Vision, pages 3–20. DOI: 10.1007/978-3-319-46448-0_1
Back to article
20Radenovic, F., Tolias, G., and Chum, O. (2018). Finetuning CNN image retrieval with no human annotation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(7): 1655–1668. DOI: 10.1109/TPAMI.2018.2846566
Back to article
21Raffel, C. (2016). Learning-Based Methods for Comparing Sequences, with Applications to Audio-to-MIDI Alignment and Matching. PhD thesis, Columbia University. DOI: 10.1109/ICASSP.2016.7471641
Back to article
22Shan, M., and Tsai, T. (2020). Improved handling of repeats and jumps in audio-sheet image synchronization. In Proceedings of the International Society for Music Information Retrieval Conference, pages 62–69.
Back to article
23Tanprasert, T., Jenrungrot, T., Müller, M., and Tsai, T. (2019). MIDI-sheet music alignment using bootleg score synthesis. In Proceedings of the International Society for Music Information Retrieval Conference, pages 91–98.
Back to article
24Thomas, V., Fremerey, C., Müller, M., and Clausen, M. (2016). Linking sheet music and audio – challenges and new approaches. In Müller, M., Goto, M., and Schedl, M., editors, Dagstuhl Follow-Ups, 3: 1–22. Schloss Dagstuhl–Leibniz-Zentrum für Informatik, Dagstuhl, Germany. DOI: 10.4230/dfu.vol3.11041.1
Back to article
25Thompson, J., Hankinson, A., and Fujinaga, I. (2011). Searching the liber usualis: Using CouchDB and ElasticSearch to query graphical music documents. In Proceedings of the International Society for Music Information Retrieval Conference.
Back to article
26Tolias, G., Sicre, R., and Jégou, H. (2016). Particular object retrieval with integral max-pooling of CNN activations. In Proceedings of the International Conference on Learning Representations.
Back to article
27Tsai, T. (2020). Towards linking the Lakh and IMSLP datasets. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pages 546–550. DOI: 10.1109/ICASSP40776.2020.9053815
Back to article
28Tsai, T., Yang, D., Shan, M., Tanprasert, T., and Jenrungrot, T. (2020). Using cell phone pictures of sheet music to retrieve MIDI passages. IEEE Transactions on Multimedia. DOI: 10.1109/TMM.2020.2973831
Back to article
29Wang, A. (2003). An industrial strength audio search algorithm. In Proceedings of the International Conference on Music Information Retrieval, pages 7–13.
Back to article
30Yang, D., Tanprasert, T., Jenrungrot, T., Shan, M., and Tsai, T. (2019). MIDI passage retrieval using cell phone pictures of sheet music. In Proceedings of the International Society for Music Information Retrieval Conference, pages 916–923.
Back to article
31Yang, D., and Tsai, T. (2020). Camera-based piano sheet music identification. In Proceedings of the International Society for Music Information Retrieval Conference, pages 481–488.
Back to article

Piano Sheet Music Identification Using Dynamic N-gram Fingerprinting

Abstract

Paradigm

My account