Structured Linear Controlled Differential Equations: expressive and parallel sequence models

Speaker's Name: Lingyi Yang
Speaker's Affiliation: University of Nottingham
Speaker's Research Theme(s): Statistics and Probability,
Abstract:
When designing the architecture of deep sequence models, we want state-transition matrices that are expressive enough to capture complex patterns while maintaining the ability to be trained at scale. In this talk, I will introduce Structured Linear Controlled Differential Equations (SLiCEs), a unifying neural differential equation framework. SLiCEs with block-diagonal, sparse, and Walsh-Hadamard transition structures can retain ​ expressivity of dense models while being cheaper to compute. On benchmark tasks, SLiCEs solve the A5 state-tracking task with a single layer, achieve best-in-class generalisation on regular language tasks, and match state-of-the-art performance on time-series classification while cutting per-step training time by a factor of twenty.

Venue: Maths A17