Incentivizing Empirical Science in Machine Learning: Problems and Proposals

We introduce a proposal to help address a structural problem in ML publishing: the lack of community support and perceived lack of legitimacy for experimental scientific work that neither proves a mathematical theorem, nor improves a practical application. Such experimental work is the bedrock of many fields of science, yet is not well appreciated by top ML publication venues (e.g. NeurIPS, ICML, ICLR). The problem is twofold: reviewers are often unaware of the value of such work, and thus authors are disincentivized from producing and submitting such work. The result is a suffocation of a scientific methodology that has a long history of success in the natural sciences, and has recently been fruitful in machine learning.To address this, we propose introducing a Best Paper Award specifically for foundational experimental work in machine learning. The award targets scientific work that is missed by existing communities: we exclude primarily theoretical work and application-motivated work, both of which are well supported by existing venues (e.g. COLT, CVPR). We propose that ML venues include a subject-area for scientific aspects of machine learning'', and consider papers in this subject for the award. More ambitiously, it can be implemented as an endowed yearly award considering all papers in the prior year. We expect that establishing an award will help legitimize this research area, establish standards for such scientific work, and encourage authors to conduct this work with the support of the community.In this proposal, we first discuss the structural problems in ML publication which we hope to address. We then present a call-for-papers for thescience of ML'' subject area, describing the type of work we want to encourage. We argue that it is not only a scientifically legitimate type of work, but perhaps even a \emph{necessary} type of work. Finally, we discuss guidelines for how such papers may be evaluated by reviewers.