Zero-Shot Extrapolation in State-Space Models for Long-Range Genomics

Long-range dependencies are crucial for interpreting genomic structure and function, yet conventional transformer-based genomics models often fail to generalize beyond their training window even when employing sophisticated positional embeddings. We show that State-Space Models (SSMs) can zero-shot extrapolate two orders of magnitude beyond their original context length, thus capturing distal regulatory interactions required for gene expressions without specialized fine-tuning. With our hidden-state transfer mechanism, we can efficiently process ultralong genomic sequences (1Mbp) on a single GPU—providing a scalable, generalizable, and resource-efficient alternative to transformers.