Poster
in
Workshop: Mathematical and Empirical Understanding of Foundation Models (ME-FoMo)
Towards Foundation Models with Mathematical Understanding
Peter Belcak · Roger Wattenhofer
Keywords: [ Transformer model ] [ mathematical concepts ] [ mathematical understanding ] [ integer sequences ] [ FACT ]
We investigate the ability of transformer models to build representations of integer sequences that are of utility to tasks where deeper mathematical understanding is needed.To that end, we train BERT-like transformer encoders to assess the impact of individual pre-training tasks on the quality of the resulting model, and evaluate them for sequence classification, continuation, unmasking, complexity prediction, and next sequence-part prediction.We find that the models both outperform benchmark baselines and provide reasonable estimates of the complexity of the mathematical rules behind the sequences.