CCOM-HuQin: An Annotated Multimodal Chinese Fiddle Performance Dataset

Yu Zhang; Ziya Zhou; Xiaobing Li; Feng Yu; Maosong Sun

doi:10.5334/tismir.146

Abstract

HuQin (胡琴) is a family of traditional Chinese bowed string instruments. Playing techniques (PTs) embodied in various playing styles add abundant emotional coloring and aesthetic feelings to HuQin performance. The complex applied techniques make HuQin music a challenging source for fundamental MIR tasks such as pitch analysis, transcription and score-audio alignment. In this paper, we present a multimodal performance dataset of HuQin music that contains audio-visual recordings of 11,992 single PT clips and 57 annotated musical excerpts of classical pieces. We systematically describe the HuQin PT taxonomy based on musicological theory and practical use cases. Then we introduce the dataset creation methodology and highlight the annotation principles featuring PTs. We analyze the statistics in different aspects to demonstrate the variety of PTs played in HuQin subcategories and perform preliminary experiments to show the potential applications of the dataset in various MIR tasks and cross-cultural music studies. Finally, we propose future work to extend the dataset.

References

1Benetos, E., Dixon, S., Duan, Z., and Ewert, S. (2018). Automatic music transcription: An overview. IEEE Signal Processing Magazine, 36(1):20–30. DOI: 10.1109/MSP.2018.2869928
Back to article
2Chen, Y. (Song(宋),1101). Yue Shu (乐书). Yuan(元),1347 edition.
Back to article
3Choi, K., Fazekas, G., Sandler, M., and Cho, K. (2017). Convolutional recurrent neural networks for music classification. In 2017 IEEE International conference on acoustics, speech and signal processing (ICASSP), pages 2392–2396. IEEE. DOI: 10.1109/ICASSP.2017.7952585
Back to article
4Dalmazzo, D. and Ramirez, R. (2019). Bowing gestures classification in violin performance: a machine learning approach. Frontiers in psychology, 10:344. DOI: 10.3389/fpsyg.2019.00344
Back to article
5Drugman, T., Huybrechts, G., Klimkov, V., and Moinet, A. (2018). Traditional machine learning for pitch detection. IEEE Signal Processing Letters, 25(11):1745–1749. DOI: 10.1109/LSP.2018.2874155
Back to article
6Ducher, J.-F. and Esling, P. (2019). Folded cqt rcnn for real-time recognition of instrument playing techniques. In International Society for Music Information Retrieval.
Back to article
7D’Amato, V., Volta, E., Oneto, L., Volpe, G., Camurri, A., and Anguita, D. (2020). Understanding violin players’ skill level based on motion capture: a data-driven perspective. Cognitive Computation, 12:1356–1369. DOI: 10.1007/s12559-020-09768-8
Back to article
8Elowsson, A. and Lartillot, O. (2021). A hardanger fiddle dataset with performances spanning emotional expressions and annotations aligned using image registration.
Back to article
9Fu, H. (2007). A study on Erhu Performance(论二胡演奏). International Publishing House For China’s Culture.
Back to article
10Goto, M., Hashiguchi, H., Nishimura, T., and Oka, R. (2002). Rwc music database: Popular, classical and jazz music databases. In Ismir, volume 2, pages 287–288.
Back to article
11Goto, M., Hashiguchi, H., Nishimura, T., and Oka, R. (2003). Rwc music database: Music genre database and musical instrument sound database.
Back to article
12Hao, H. and Ma, Z. (2004). Performance Methods on Yu-Ju Banhu(豫剧板胡演奏法). Henan Literary and Art Press.
Back to article
13Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. In (International Conference on Learning Representations (ICLR).
Back to article
14Konkol, M. and Konopik, M. (2015). Segment representations in named entity recognition. In International conference on text, speech, and dialogue, pages 61–70. Springer. DOI: 10.1007/978-3-319-24033-6_7
Back to article
15Kruger, A. and Jacobs, J. (2020). Playing technique classification for bowed string instruments from raw audio. Journal of New Music Research, 49(4):320–333. DOI: 10.1080/09298215.2020.1784957
Back to article
16Li, B., Dinesh, K., Sharma, G., and Duan, Z. (2017). Video-based vibrato detection and analysis for polyphonic string music. In ISMIR, pages 123–130.
Back to article
17Li, B., Liu, X., Dinesh, K., Duan, Z., and Sharma, G. (2018). Creating a multitrack classical music performance dataset for multimodal music analysis: Challenges, insights, and applications. IEEE Transactions on Multimedia, 21(2):522–535. DOI: 10.1109/TMM.2018.2856090
Back to article
18Li, N. (2007). Left-hand playing techniques of erhu and their applications. The New Voice of Yue-Fu-The Academic Periodical of Shenyang Conservatory of Music, pages 180–183.
Back to article
19Liang, X., Li, Z., Liu, J., Li, W., Zhu, J., and Han, B. (2019). Constructing a multimedia chinese musical instrument database. In Proceedings of the 6th Conference on Sound and Music Technology (CSMT), pages 53–60. Springer. DOI: 10.1007/978-981-13-8707-4_5
Back to article
20Liu, C. (1986). Stylistic skills in erhu performance. Journal of the Central Conservatory of Music, pages 54–58.
Back to article
21Liu, C., Li, H., Tian, Z., Xue, k., Yan, J., Yu, H., Zhao, H., and Zhu, J. (2012). Exibition of Chinese traditional instrumental music(中国民族器乐曲博览), volume Solo. People’s Music Publishing House.
Back to article
22Liu, D. (1992). Illustrated catalogue of Chinese musical instruments(中国乐器图鉴). Shandong Education Press.
Back to article
23Lostanlen, V., Anden, J., and Lagrange, M. (2018). Extended playing techniques: The next milestone in musical instrument recognition. In Proceedings of the 5th International Conference on Digital Libraries for Musicology, DLfM ’18, page 1–10, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3273024.3273036
Back to article
24Mauch, M., Cannam, C., Bittner, R., Fazekas, G., Salamon, J., Dai, J., Bello, J., and Dixon, S. (2015). Computer-aided melody note transcription using the tony software: Accuracy and efficiency.
Back to article
25Mauch, M. and Dixon, S. (2014). pyin: A fundamental frequency estimator using probabilistic threshold distributions. In 2014 ieee international conference on acoustics, speech and signal processing (icassp), pages 659–663. IEEE. DOI: 10.1109/ICASSP.2014.6853678
Back to article
26McFee, B., Raffel, C., Liang, D., Ellis, D. P., McVicar, M., Battenberg, E., and Nieto, O. (2015). librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science conference, volume 8, pages 18–25. Citeseer. DOI: 10.25080/Majora-7b98e3ed-003
Back to article
27Montesinos, J. F., Slizovskaia, O., and Haro, G. (2020). Solos: A dataset for audio-visual music analysis. In 2020 IEEE 22nd International Workshop on Multimedia Signal Processing (MMSP), pages 1–6. DOI: 10.1109/MMSP48831.2020.9287124
Back to article
28Qiao, J., Yang, G., Yu, Q., and Zhao, H. (2010). China Music(华乐大典), volume Erhu. Shanghai Music Press.
Back to article
29Shen, C. (1997). Local style and skills of banhu(板胡的地方风格与技巧). Chinese Music, pages 31–33.
Back to article
30Simonetta, F., Ntalampiras, S., and Avanzini, F. (2019). Multimodal music information processing and retrieval: Survey and future challenges. In 2019 international workshop on multilayer music representation and processing (MMRP), pages 10–18. IEEE. DOI: 10.1109/MMRP.2019.00012
Back to article
31Su, L., Lin, H.-M., and Yang, Y.-H. (2014). Sparse modeling of magnitude and phase-derived spectra for playing technique classification. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 22(12):2122–2132. DOI: 10.1109/TASLP.2014.2362006
Back to article
32Subramani, K. and Rao, P. (2020). Hprnet: Incorporating residual noise modeling for violin in a variational parametric synthesizer.
Back to article
33Thickstun, J., Harchaoui, Z., and Kakade, S. (2016). Learning features of music from scratch. arXiv preprint arXiv:1611.09827.
Back to article
34Thomas, V., Fremerey, C., Muller, M., and Clausen, M. (2012). Linking sheet music and audio–challenges and new approaches.
Back to article
35Tsou, J. (2001). The new grove dictionary of music and musicians.
Back to article
36Volpe, G., Kolykhalova, K., Volta, E., Ghisio, S., Waddell, G., Alborno, P., Piana, S., Canepa, C., and Ramirez-Melendez, R. (2017). A multimodal corpus for technology-enhanced learning of violin playing. volume Part F131371. Association for Computing Machinery. DOI: 10.1145/3125571.3125588
Back to article
37von Coler, H. (2018). Tu-note violin sample library–a database of violin sounds with segmentation ground truth. In Proceedings of the 21st International Conference on Digital Audio Effects (DAFx-18), Aveiro, Portugal, pages 4–8.
Back to article
38von Coler, H. and Lerch, A. (2014). Cmmsd: A data set for note-level segmentation of monophonic music. In Audio Engineering Society Conference: 53rd International Conference: Semantic Audio. Audio Engineering Society.
Back to article
39Wang, C., Lostanlen, V., Benetos, E., and Chew, E. (2020). Playing technique recognition by joint time–frequency scattering. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 881–885. IEEE. DOI: 10.1109/ICASSP40776.2020.9053474
Back to article
40Wang, Z., Li, J., Chen, X., Li, Z., Zhang, S., Han, B., and Yang, D. (2019). Musical instrument playing technique detection based on fcn: Using chinese bowed-stringed instrument as an example. arXiv preprint arXiv:1910.09021.
Back to article
41Yang, L. (2016). Computational modelling and analysis of vibrato and portamento in expressive music performance. PhD thesis, Queen Mary University of London.
Back to article
42Zeng, M. (2006). Bowing and vibrato on the erhu. Master’s thesis, Master dissertation, Shanghai Conservatory of Music.
Back to article
43Zhang, F., Bazarevsky, V., Vakunov, A., Tkachenka, A., Sung, G., Chang, C., and Grundmann, M. (2020). Mediapipe hands: On-device real-time hand tracking. CoRR, abs/2006.10214.
Back to article
44Zhang, W., Lei, W., Xu, X., and Xing, X. (2016). Improved music genre classification with convolutional neural networks. In Interspeech, pages 3304–3308. DOI: 10.21437/Interspeech.2016-1236
Back to article
45Zhao, H. (1999). The usage of portamento techniques in erhu performance(二胡演奏中滑音技法的运用). Journal of the Central Conservatory of Music, pages 53–57.
Back to article
46Zhu, H., Li, Y., Zhu, F., Zheng, A., and He, R. (2021). Let’s play music: Audio-driven performance video generation. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 3574–3581. IEEE. DOI: 10.1109/ICPR48806.2021.9412698
Back to article

CCOM-HuQin: An Annotated Multimodal Chinese Fiddle Performance Dataset

Abstract

Paradigm

My account