On Efficient Computational Methods for Transformer-Based Symbolic Music Generation

Abstract

Although Transformer models have shown particular promise for symbolic music generation, their quadratic computational complexity with respect to sequence length presents significant challenges for longer musical pieces. In this paper, we describe the goals and progress of an ongoing dissertation addressing these challenges through three interconnected research directions, aiming at the development of (i) novel tokenisation strategies that significantly reduce sequence lengths while maintaining generation quality, (ii) efficient methods for incorporating arbitrary musical information into attention mechanisms through both additive and multiplicative approaches, yielding statistically significant improvements over strong baselines, and (iii) a hierarchical attention architecture that explicitly models the multi-level structure of music across beats, bars, and larger segments using specialised block-sparse attention patterns. Results achieved so far support our central hypothesis that domain-aware architectural choices, informed by music theory, can yield significant improvements over generic sequence-modelling approaches.