MuVGM: The Multimodal Video Game Music Dataset

Tyler H. McIntosh; Simon Dixon

doi:10.5334/tismir.381

MuVGM: The Multimodal Video Game Music Dataset

Transactions of the International Society for Music Information Retrieval

Volume 9 (2026): Issue 1

By: Tyler H. McIntosh and Simon Dixon

Open Access

|Oct 2026

Abstract

We introduce MuVGM, a large‑scale Multimodal dataset for Video Game Music research, combining symbolic transcriptions, reference audio, gameplay videos, box art imagery, textual descriptions, audio captions, and affective annotations across a broad range of titles and platforms. Video game music is uniquely suited to multimodal research because it is composed for specific, directly observable visual environments and narrative contexts, providing explicit functional grounding absent from most existing music datasets. The dataset comprises 7129 tracks from 1484 games across 39 consoles, spanning 37 years of game music history, and was constructed through a six‑stage pipeline combining web scraping, Wikipedia matching, cover song detection, audio fingerprinting, retrieval‑augmented text generation, and automatic music captioning. The dataset, including an interactive web‑based dataset explorer, is available at https://tylerhmc.com/muvgm/.

References

Agostinelli, A., Denk, T. I., Borsos, Z., Engel, J., Verzetti, M., Caillon, A., Huang, Q., Jansen, A., Roberts, A., Tagliasacchi, M., Sharifi, M., Zeghidour, N., and Frank, C. (2023). MusicLM: Generating music from text. arXiv preprint arXiv:2301.11325.
Search in Google Scholar Back to article
Barthet, M., Kaselimi, M., Pinitas, K., Makantasis, K., Liapis, A., and Yannakakis, G. N. (2024). Gamevibe: a multimodal affective game corpus. Scientific Data, 11(1), 1306. 10.1038/s41597-024-04022-4.
Open DOI Search in Google Scholar Back to article
Bertin‑Mahieux, T., Ellis, D. P. W., Whitman, B., and Lamere, P. (2011). The million song dataset. Miami, Florida, USA, In Proceedings of the 12th International Conference on Music Information Retrieval (ISMIR) (pp. 591–596). 10.5281/zenodo.1415820.
Open DOI Search in Google Scholar Back to article
Cardoso, I., Moraes, R. O., and Ferreira, L. N. (2024). The NES video‑music database: A dataset of symbolic video game music paired with gameplay videos. In Proceedings of the 19th International Conference on the Foundations of Digital Games (FDG) (pp. 1–6). ACM. 10.1145/3649921.3650011.
Open DOI Search in Google Scholar Back to article
Choi, E., Chung, Y., Lee, S., Jeon, J., Kwon, T., and Nam, J. (2022). YM2413‑MDB: A multi‑instrumental FM video game music dataset with emotion annotations. Bengaluru, India, In Proceedings of the 23rd International Society for Music Information Retrieval Conference (ISMIR) (pp. 100–108). 10.5281/zenodo.7316606.
Open DOI Search in Google Scholar Back to article
Christodoulou, A.‑M., Lartillot, O., and Jensenius, A. R. (2024). Multimodal music datasets? challenges and future goals in music processing. International Journal of Multimedia Information Retrieval, 13(3), 37. 10.1007/s13735-024-00344-6.
Open DOI Search in Google Scholar Back to article
Cohen, J. (2013). Statistical Power Analysis for the Behavioral Sciences. Routledge. 10.4324/9780203771587.
Open DOI Search in Google Scholar Back to article
Collins, K. (2008). Game Sound: An Introduction to the History, Theory, and Practice of Video Game Music and Sound Design. MIT Press. 10.7551/mitpress/7909.001.0001.
Open DOI Search in Google Scholar Back to article
Di, S., Jiang, Z., Liu, S., Wang, Z., Zhu, L., He, Z., Liu, H., and Yan, S. (2021). Video background music generation with controllable music transformer. New York, NY, USA, In Proceedings of the 29th ACM International Conference on Multimedia, MM ‘21 (pp. 2037–2045). Association for Computing Machinery. 10.1145/3474085.3475195.
Open DOI Search in Google Scholar Back to article
Doh, S., Choi, K., Lee, J., and Nam, J. (2023). LP‑MusicCaps: LLM‑based pseudo music captioning. Milan, Italy, In Proceedings of the 24th International Society for Music Information Retrieval Conference (ISMIR) (pp. 409–416). 10.5281/zenodo.10265311.
Open DOI Search in Google Scholar Back to article
Donahue, C., Mao, H. H., and McAuley, J. (2018). The NES music database: A multi‑instrumental dataset with expressive performance attributes. Paris, France, In Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR) (pp. 475–482). 10.5281/zenodo.1492455.
Open DOI Search in Google Scholar Back to article
Elizalde, B., Deshmukh, S., Ismail, M. A., and Wang, H. (2023). CLAP: Learning audio concepts from natural language supervision. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1–5). 10.1109/icassp49357.2023.10095889.
Open DOI Search in Google Scholar Back to article
Ferreira, L. N., and Whitehead, J. (2019). Learning to generate music with sentiment. Delft, The Netherlands, In Proceedings of the 20th International Society for Music Information Retrieval Conference (ISMIR) (pp. 384–390).
Search in Google Scholar Back to article
Gan, C., Huang, D., Chen, P., Tenenbaum, J. B., and Torralba, A. (2020). Foley music: Learning to generate music from videos. In In Computer Vision – ECCV 2020 (pp. 758–775). Springer. 10.1007/978-3-030-58621-8_44.
Open DOI Search in Google Scholar Back to article
Ghosh, S., Goel, A., Koroshinadze, L., Lee, S.‑g., Kong, Z., Santos, J. F., Duraiswami, R., Manocha, D., Ping, W., Shoeybi, M., and Catanzaro, B. (2026). Music flamingo: Scaling music understanding in audio language models. In International Conference on Learning Representations (ICLR). 10.48550/arXiv.2511.10289.
Open DOI Search in Google Scholar Back to article
Girdhar, R., El‑Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., and Misra, I. (2023). ImageBind: One embedding space to bind them all. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 15180–15190). 10.1109/cvpr52729.2023.01457.
Open DOI Search in Google Scholar Back to article
Gotham, M., Bemman, B., and Vatolkin, I. (2025). Towards an ‘everything corpus’: A framework and guidelines for the curation of more comprehensive multimodal music data. Transactions of the International Society for Music Information Retrieval, 8(1), 70–92. 10.5334/tismir.228.
Open DOI Search in Google Scholar Back to article
Hewitt, J., and Liang, P. (2019). Designing and interpreting probes with control tasks. Hong Kong, China, K. Inui, J. Jiang, V. Ng, and X. Wan (Eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP‑IJCNLP) (pp. 2733–2743). Association for Computational Linguistics. 10.18653/v1/d19-1275.
Open DOI Search in Google Scholar Back to article
Hsieh, F.‑C., Lee, W.‑J., Wang, C.‑P., Lee, H.‑y., Dong, H.‑W., and Yang, Y.‑H. (2026). Academic text‑to‑music grand challenge: Datasets, baselines, and evaluation methods. In IEEE International Conference on Multimedia and Expo (ICME), Grand Challenge Paper. 10.48550/arXiv.2605.21538.
Open DOI Search in Google Scholar Back to article
Huang, Q., Park, D. S., Wang, T., Denk, T. I., Ly, A., Chen, N., Zhang, Z., Zhang, Z., Yu, J., Frank, C., Engel, J., Le, Q. V., Chan, W., Chen, Z., and Han, W. (2023). Noise2Music: Text‑conditioned music generation with diffusion models. arXiv preprint arXiv:2302.03917.
Search in Google Scholar Back to article
Kai, W.‑H., and Xing, K.‑X. (2024). Video‑driven musical composition using large language model with memory‑augmented state space. The Visual Computer, 41, 3345–3357. 10.1007/s00371-024-03606-w.
Open DOI Search in Google Scholar Back to article
Khanal, P., and Donnelly, P. J. (2025). Emotionotes dataset: Decoding emotions in classical music through concert program notes. In P. Machado, C. Johnson, and I. Santos (Eds.), Artificial Intelligence in Music, Sound, Art and Design (pp. 339–356). Springer Nature Switzerland. 10.1007/978-3-031-90167-6_23.
Open DOI Search in Google Scholar Back to article
Kim, C. D., Kim, B., Lee, H., and Kim, G. (2019). Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL‑HLT) (pp. 119–132). 10.18653/v1/n19-1011.
Open DOI Search in Google Scholar Back to article
Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. (2019). Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learning (ICML) (Vol. 97, pp. 3519–3529). PMLR. https://proceedings.mlr.press/v97/kornblith19a.html.
Search in Google Scholar Back to article
Li, B., Liu, X., Dinesh, K., Duan, Z., and Sharma, G. (2019). Creating a multitrack classical music performance dataset for multimodal music analysis: Challenges, insights, and applications. IEEE Transactions on Multimedia, 21(2), 522–535. 10.1109/tmm.2018.2856090.
Open DOI Search in Google Scholar Back to article
Li, S., Ji, S., Wang, Z., Wu, S., Yu, J., and Zhang, K. (2026). A survey on music generation from single‑modal, cross‑modal, and multi‑modal perspectives. ACM Computing Surveys, 58(11). 10.1145/3800682.
Open DOI Search in Google Scholar Back to article
Li, S., Qin, Y., Zheng, M., Jin, X., and Liu, Y. (2024). Diff‑BGM: A diffusion model for video background music generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 27348–27357). 10.1109/cvpr52733.2024.02582.
Open DOI Search in Google Scholar Back to article
Liang, Q., and Wang, Y. (2024). Drawlody: Sketch‑based melody creation with enhanced usability and interpretability. IEEE Transactions on Multimedia, 26, 7074–7088. 10.1109/tmm.2024.3360695.
Open DOI Search in Google Scholar Back to article
Liu, S., Hussain, A. S., Wu, Q., Sun, C., and Shan, Y. (2026). MuMu‑LLaMA: Multi‑modal music understanding and generation via large language models. Expert Systems with Applications, 305. 10.1016/j.eswa.2025.130688.
Open DOI Search in Google Scholar Back to article
Lu, P., Xu, X., Kang, C., Yu, B., Xing, C., Tan, X., and Bian, J. (2023). MuseCoco: Generating symbolic music from text. arXiv preprint arXiv:2306.00110.
Search in Google Scholar Back to article
Luo, S., and Reiss, J. (2026). Supervised learning for game music segmentation.
Search in Google Scholar Back to article
Maniparambil, M., Akshulakov, R., Djilali, Y. A. D., Seddik, E. A., Narayan, M., K, S., Mangalam, and O’Connor, N. E. (2024). Do vision and language encoders represent the world similarly? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 14334–14343). 10.1109/cvpr52733.2024.01359.
Open DOI Search in Google Scholar Back to article
McFee, B., Raffel, C., Liang, D., Ellis, D. P. W., McVicar, M., Battenberg, E., and Nieto, O. (2015). librosa: Audio and music signal analysis in python. In Proceedings of the 14th Python in Science Conference (pp. 18–25). 10.25080/majora-7b98e3ed-003.
Open DOI Search in Google Scholar Back to article
McKay, C., and Fujinaga, I. (2008). Combining features extracted from audio, symbolic and cultural sources. Philadelphia, PA, In Proceedings of the 9th International Society for Music Information Retrieval Conference (ISMIR) (pp. 597–602). 10.5281/zenodo.1415575.
Open DOI Search in Google Scholar Back to article
Melechovsky, J., Roy, A., and Herremans, D. (2024). MidiCaps: A large‑scale MIDI dataset with text captions. San Francisco, USA, In Proceedings of the 25th International Society for Music Information Retrieval Conference (ISMIR) (pp. 858–865). 10.5281/zenodo.14877464.
Open DOI Search in Google Scholar Back to article
Meseguer‑Brocal, G., Cohen‑Hadria, A., and Peeters, G. (2020). Creating DALI, a large dataset of synchronized audio, lyrics, and notes. Transactions of the International Society for Music Information Retrieval, 3(1), 55–67. 10.5334/tismir.30.
Open DOI Search in Google Scholar Back to article
Mohammad, S. M. (2025). NRC VAD Lexicon v2: Norms for valence, arousal, and dominance for over 55k English terms.
Search in Google Scholar Back to article
Montesinos, J. F., Slizovskaia, O., and Haro, G. (2020). Solos: A dataset for audio‑visual music analysis. In 22nd IEEE International Workshop on Multimedia Signal Processing, MMSP (pp. 1–6). IEEE. 10.1109/mmsp48831.2020.9287124.
Open DOI Search in Google Scholar Back to article
Namgyal, T., Santos‑Rodriguez, R., and Flach, P. (2022). MIDI‑Draw: Sketching to control melody generation. Bengaluru, India, In Extended Abstracts for the Late‑Breaking Demo Session of the 23rd International Society for Music Information Retrieval Conference (ISMIR). 10.48550/arXiv.2305.11605.
Open DOI Search in Google Scholar Back to article
Oramas, S., Barbieri, F., Nieto, O., and Serra, X. (2018). Multimodal deep learning for music genre classification. Transactions of the International Society for Music Information Retrieval, 1(1), 4–21. 10.5334/tismir.10.
Open DOI Search in Google Scholar Back to article
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML) (Vol. 139, pp. 8748–8763). PMLR. 10.48550/arXiv.2103.00020.
Open DOI Search in Google Scholar Back to article
Raffel, C. (2016). Learning‑Based Methods for Comparing Sequences, with Applications to Audio‑to‑MIDI Alignment and Matching (Doctoral dissertation). Columbia University.
Search in Google Scholar Back to article
Rinaldi, I., Fanelli, N., Castellano, G., and Vessio, G. (2025). Art2Mus: Bridging visual arts and music through cross‑modal generation. In Computer Vision – ECCV 2024 Workshops (pp. 173–186). Springer Nature. 10.1007/978-3-031-92808-6_11.
Open DOI Search in Google Scholar Back to article
Romano, J., Kromrey, J., Coraggio, J., and Skowronek, J. (2006). Appropriate statistics for ordinal level data: Should we really be using t‑test and Cohen’s d for evaluating group differences on the NSSE and other surveys? In Annual Meeting of the Florida Association of Institutional Research (pp. 1–3).
Search in Google Scholar Back to article
Serrà, J., Gómez, E., Herrera, P., and Serra, X. (2008). Chroma binary similarity and local alignment applied to cover song identification. IEEE Transactions on Audio, Speech, and Language Processing, 16, 1138–1151. 10.1109/TASL.2008.924595.
Open DOI Search in Google Scholar Back to article
Serrà, J., Serra, X., and Andrzejak, R. G. (2009). Cross recurrence quantification for cover song identification. New Journal of Physics, 11(9), 093017. 10.1109/TASL.2008.924595.
Open DOI Search in Google Scholar Back to article
Sharma, M., Haseeb, M. T., Xia, G., and Tsuruoka, Y. (2026). A multimodal framework for automated background music generation in Japanese manga using large language models. International Journal of Film and Media Arts, 11(1). 10.24140/ijfma.v11i1.10913.
Open DOI Search in Google Scholar Back to article
Su, K., Li, J. Y., Huang, Q., Kuzmin, D., Lee, J., Donahue, C., Sha, F., Jansen, A., Wang, Y., Verzetti, M., and Denk, T. (2024). V2Meow: Meowing to the visual beat via video‑to‑music generation. In Proceedings of the Thirty‑Eighth AAAI Conference on Artificial Intelligence and Thirty‑Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’24/IAAI’24/EAAI’24. AAAI Press. 10.1609/aaai.v38i5.28299.
Open DOI Search in Google Scholar Back to article
Su, K., Liu, X., and Shlizerman, E. (2020). Multi‑instrumentalist net: Unsupervised generation of music from body movements. arXiv preprint arXiv:2012.03478.
Search in Google Scholar Back to article
Su, K., Liu, X., and Shlizerman, E. (2021). How does it sound? In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan (Eds.), Advances in Neural Information Processing Systems. 10.1080/09537325.2021.1926965.
Open DOI Search in Google Scholar Back to article
Tian, S., Zhang, C., Yuan, W., Tan, W., and Zhu, W. (2025). XMusic: Towards a generalized and controllable symbolic music generation framework. IEEE Transactions on Multimedia, 27, 6857–6871. 10.1109/tmm.2025.3590912.
Open DOI Search in Google Scholar Back to article
Ullah, U., and Choi, H.‑C. (2024). MuIm: Analyzing music–image correlations from an artistic perspective. Applied Sciences, 14(23). 10.3390/app142311470.
Open DOI Search in Google Scholar Back to article
Wu, S., Zhancheng, G., Yuan, R., Jiang, J., Doh, S., Xia, G., Nam, J., Li, X., Yu, F., and Sun, M. (2025). CLaMP 3: Universal music information retrieval across unaligned modalities and unseen languages. In W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Findings of the Association for Computational Linguistics: ACL 2025 (pp. 2605–2625). Association for Computational Linguistics. 10.18653/v1/2025.findings-acl.133.
Open DOI Search in Google Scholar Back to article
Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg‑Kirkpatrick, T., and Dubnov, S. (2023). Large‑scale contrastive language‑audio pretraining with feature fusion and keyword‑to‑caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1–5). 10.1109/icassp49357.2023.10095969.
Open DOI Search in Google Scholar Back to article
Yu, J., Wang, Y., Chen, X., Sun, X., and Qiao, Y. (2023). Long‑term rhythmic video soundtracker. In Proceedings of the 40th International Conference on Machine Learning (ICML) (Vol. 202, pp. 40339–40353). PMLR. https://proceedings.mlr.press/v202/yu23e.html.
Search in Google Scholar Back to article
Zalkow, F., Balke, S., Arifi‑Müller, V., and Müller, M. (2020). MTD: A multimodal dataset of musical themes for MIR research. Transactions of the International Society for Music Information Retrieval (TISMIR), 3(1), 180–192. 10.5334/tismir.68.
Open DOI Search in Google Scholar Back to article
Zhang, Y., Wang, Z., Wang, D., and Xia, G. (2020). BUTTER: A representation learning framework for bi‑directional music‑sentence retrieval and generation. Online: S. Oramas, L. Espinosa‑Anke, E. Epure, R. Jones, M. Sordo, M. Quadrana, and K. Watanabe (Eds.), Proceedings of the 1st Workshop on NLP for Music and Audio (NLP4MusA) (pp. 54–58). Association for Computational Linguistics. https://aclanthology.org/2020.nlp4musa-1.11/.
Search in Google Scholar Back to article
Zhao, F., Chen, Y., Lu, W., Zhang, D., Yue, X., and Wei, J. (2026). HalluAudio: A comprehensive benchmark for hallucination detection in large audio‑language models. In M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens (Eds.), Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Long Papers) (Vol. 1, pp. 38797–38816). Association for Computational Linguistics. 10.18653/v1/2026.acl-long.1797.
Open DOI Search in Google Scholar Back to article
Zhao, H., Gan, C., Rouditchenko, A., Vondrick, C., McDermott, J., and Torralba, A. (2018). The sound of pixels. Berlin, Heidelberg, In 15th European Conference on Computer Vision (ECCV) (pp. 587–604). Springer‑Verlag. 10.1007/978-3-030-01246-5_35.
Open DOI Search in Google Scholar Back to article
Zhu, Y., Olszewski, K., Wu, Y., Achlioptas, P., Chai, M., Yan, Y., and Tulyakov, S. (2022). Quantized GAN for complex music generation from dance videos. In European Conference on Computer Vision (ECCV) (pp. 182–199). Springer. 10.1007/978-3-031-19836-6_11.
Open DOI Search in Google Scholar Back to article