Drumroll Please: Modeling Multi-Scale Rhythmic Gestures with Flexible Grids

Jon Gillick; Joshua Yang; Carmine-Emanuele Cella; David Bamman

doi:10.5334/tismir.98

Abstract

We present a new data representation for music modeling and generation called a Flexible Grid. This representation aims to balance flexibility with structure in order to encode all the musical events (notes or rhythmic onsets) in a dataset without quantizing or discarding any temporal information. In experiments with a dataset of MIDI drum performances, we find that when implemented in a Variational AutoEncoder (VAE) model, Flexible Grid representations can enable detailed generation of music performance data that includes multiple different gestures and articulations.

References

1Ames, C. (1989). The Markov process as a compositional model: A survey and tutorial. Leonardo, pages 175–187. DOI: 10.2307/1575226
Back to article
2Brunner, G., Konrad, A., Wang, Y., and Wattenhofer, R. (2018). MIDI-VAE: Modeling dynamics and instrumentation of music with applications to style transfer. arXiv preprint arXiv:1809.07600.
Back to article
3Cancino-Chacon, C. E., Grachten, M., Goebl, W., and Widmer, G. (2018). Computational models of expressive music performance: A comprehensive and critical review. Frontiers in Digital Humanities, 5: 25. DOI: 10.3389/fdigh.2018.00025
Back to article
4Danielsen, A., Nymoen, K., Anderson, E., Camara, G. S., Langerod, M. T., Thompson, M. R., and London, J. (2019). Where is the beat in that note? Effects of attack, duration, and frequency on the perceived timing of musical and quasi-musical sounds. Journal of Experimental Psychology: Human Perception and Performance, 45(3): 402. DOI: 10.1037/xhp0000611
Back to article
5Dixon, S., Goebl, W., and Cambouropoulos, E. (2006). Perceptual smoothness of tempo in expressively performed music. Music Perception, 23(3): 195–214. DOI: 10.1525/mp.2006.23.3.195
Back to article
6Eck, D., and Schmidhuber, J. (2002). Finding temporal structure in music: Blues improvisation with LSTM recurrent networks. In Proceedings of the 12th IEEE Workshop on Neural Networks for Signal Processing, pages 747–756. DOI: 10.1109/NNSP.2002.1030094
Back to article
7Fisher, M. (2012). Burial: Unedited transcript. The Wire.
Back to article
8Frane, A. V. (2017). Swing rhythm in classic drum breaks from hip-hop’s breakbeat canon. Music Perception: An Interdisciplinary Journal, 34(3): 291–302. DOI: 10.1525/mp.2017.34.3.291
Back to article
9Gillick, J., Roberts, A., Engel, J., Eck, D., and Bamman, D. (2019). Learning to groove with inverse sequence transformations. In International Conference on Machine Learning, pages 2269–2279.
Back to article
10Gillick, J., Tang, K., and Keller, R. M. (2010). Machine learning of jazz grammars. Computer Music Journal, 34(3): 56–66. DOI: 10.1162/COMJ_a_00006
Back to article
11Hawthorne, C., Stasyuk, A., Roberts, A., Simon, I., Huang, C.-Z. A., Dieleman, S., Elsen, E., Engel, J., and Eck, D. (2018). Enabling factorized piano music modeling and generation with the Maestro dataset. arXiv preprint arXiv:1810.12247.
Back to article
12Huang, C.-Z. A., Koops, H. V., Newton-Rex, E., Dinculescu, M., and Cai, C. J. (2020). AI Song Contest: Human-AI co-creation in songwriting. arXiv preprint arXiv:2010.05388.
Back to article
13Huang, C.-Z. A., Vaswani, A., Uszkoreit, J., Simon, I., Hawthorne, C., Shazeer, N., Dai, A. M., Hoffman, M. D., Dinculescu, M., and Eck, D. (2018). Music transformer. arXiv preprint arXiv:1809.04281.
Back to article
14Huang, Y.-S., and Yang, Y.-H. (2020). Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions. In Proceedings of the 28th ACM International Conference on Multimedia, MM ’20, page 1180–1188, New York, NY, USA. Association for Computing Machinery. DOI: 10.1145/3394171.3413671
Back to article
15Jeong, D., Kwon, T., Kim, Y., and Nam, J. (2019). Graph neural network for music score data and modeling expressive piano performance. In International Conference on Machine Learning, pages 3060–3070. PMLR.
Back to article
16Kumar, P., Sanju, H. K., and Nikhil, J. (2016). Temporal resolution and active auditory discrimination skill in vocal musicians. International Archives of Otorhinolaryngology, 20(4): 310–314. DOI: 10.1055/s-0035-1570312
Back to article
17Lee, M. K., Kusbit, D., Metsky, E., and Dabbish, L. (2015). Working with machines: The impact of algorithmic and data-driven management on human workers. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pages 1603–1612. DOI: 10.1145/2702123.2702548
Back to article
18Lerdahl, F., and Jackendoff, R. S. (1996). A Generative Theory of Tonal Music: Reissue, with a New Preface. MIT Press. DOI: 10.7551/mitpress/12513.001.0001
Back to article
19Mozer, M. C. (1994). Neural network music composition by prediction: Exploring the benefits of psychoacoustic constraints and multi-scale processing. Connection Science, 6(2–3): 247–280. DOI: 10.1080/09540099408915726
Back to article
20Muchnik, C., Hildesheimer, M., Rubinstein, M., Sadeh, M., Shegter, Y., and Shibolet, B. (1985). Minimal time interval in auditory temporal resolution. The Journal of Auditory Research, 25(4): 239–246.
Back to article
21Oore, S., Simon, I., Dieleman, S., Eck, D., and Simonyan, K. (2018). This time with feeling: Learning expressive musical performance. Neural Computing and Applications, pages 1–13. DOI: 10.1007/s00521-018-3758-9
Back to article
22Pati, A., Lerch, A., and Hadjeres, G. (2019). Learning to traverse latent spaces for musical score inpainting. arXiv preprint arXiv:1907.01164.
Back to article
23Radford, A., Metz, L., and Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434.
Back to article
24Roberts, A., Engel, J., Mann, Y., Gillick, J., Kayacik, C., Norly, S., Dinculescu, M., Radebaugh, C., Hawthorne, C., and Eck, D. (2019). Magenta Studio: Augmenting creativity with deep learning in Ableton Live. In Proceedings of the International Workshop on Musical Metacreation (MUME).
Back to article
25Roberts, A., Engel, J., Raffel, C., Hawthorne, C., and Eck, D. (2018). A hierarchical latent vector model for learning long-term structure in music. arXiv preprint arXiv:1803.05428.
Back to article
26Shi, Z., Cancino-Chacon, C., and Widmer, G. (2019). User curated shaping of expressive performances. arXiv preprint arXiv:1906.06428.
Back to article
27Sturm, B. L., Ben-Tal, O., Monaghan, U., Collins, N., Herremans, D., Chew, E., Hadjeres, G., Deruty, E., and Pachet, F. (2018). Machine learning research that matters for music creation: A case study. Journal of New Music Research, 48(1): 36–55. DOI: 10.1080/09298215.2018.1515233
Back to article
28Sturm, B. L., Santos, J. F., Ben-Tal, O., and Korshunova, I. (2016). Music transcription modelling and composition using deep learning. arXiv preprint arXiv:1604.08723.
Back to article
29Tokui, N. (2020). Towards democratizing music production with AI-design of variational autoencoderbased rhythm generator as a DAW plugin. arXiv preprint arXiv:2004.01525.
Back to article
30Vigliensoni, G., McCallum, L., and Fiebrink, R. (2020). Creating latent spaces for modern music genre rhythms using minimal training data. Proceedings of the 11th International Conference on Computational Creativity.
Back to article
31Yang, L.-C., Chou, S.-Y., and Yang, Y.-H. (2017). MidiNet: A convolutional generative adversarial network for symbolic-domain music generation. arXiv preprint arXiv:1703.10847.
Back to article

Drumroll Please: Modeling Multi-Scale Rhythmic Gestures with Flexible Grids

Abstract

Paradigm

My account