Does the Market of Citations Reward Reproducible Work?

The field of bibliometrics, studying citations and behavior, is critical to the the discussion of reproducibility as citations are one of the primary incentive and reward systems for academic work. Yet to the best of our knowledge, only one work has attempted to look at this combined space, concluding that non-reproducible work is more highly cited. We show that answering this question is more challenging than first proposed, and subtle issues can inhibit a robust conclusion. To make inferences with more robust behavior, we propose a hierarchical Bayesian model that incorporates the citation rate over time, rather than the total number of citations after a fixed amount of time. In doing so we show that, under current evidence the the answer is more likely that certain fields of study such as Medicine and Machine Learning (ML) do reward reproducible works with more citations, but other fields appear to have no relationship. Further, we find that making code available and thoroughly referencing prior works appear to also positively correlate with increased citations.