Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

Clemencia Siro ⋅ Guy Gur-Ari ⋅ Gaurav Mishra ⋅ Stuart Shieber ⋅ Jason Phang ⋅ Zijie Wang ⋅ Kory Mathewson ⋅ Giorgio Mariani ⋅ Allen Nie ⋅ James Y Zou ⋅ Behnam Neyshabur ⋅ Karl Krauth ⋅ Shixiang Gu ⋅ Pablo Antonio Moreno Casares ⋅ Maarten Sap ⋅ Mohit Tiwari ⋅ Bill Yuchen Lin ⋅ Aykut Erdem ⋅ Angelica Chen ⋅ Swaroop Mishra ⋅ Chenlin Meng ⋅ Ashish Sabharwal ⋅ James Simon ⋅ Louis-Philippe Morency ⋅ Kyle Richardson ⋅ Emanuele Rodolà ⋅ Adam Fisch ⋅ Simone Melzi ⋅ Kristen Chiafullo ⋅ Rif A. Saurous ⋅ Shubh Pachchigar ⋅ Siamak Shakeri ⋅ Aitor Lewkowycz ⋅ Yonatan Belinkov ⋅ Mihir Kale ⋅ Mantas Mazeika ⋅ Dar Gilboa ⋅ Hongming Zhang ⋅ Seung Jae Lee ⋅ Owain Evans ⋅ Ambrose Slone ⋅ David Dohan ⋅ Damien Sileo ⋅ Mor Geva ⋅ Cameron Diao ⋅ Christopher Potts ⋅ Jekaterina Novikova ⋅ Alicia Parrish ⋅ Debajyoti Datta ⋅ Chitta Baral ⋅ Maarten Bosma ⋅ Michael Strube ⋅ Jiacheng Xu ⋅ Trishala Neeraj ⋅ Colin Raffel ⋅ Leo Gao ⋅ Vishakh Padmakumar ⋅ Yu (Hope) Hou ⋅ Christopher Waites ⋅ Ellie Pavlick ⋅ Pouya Pezeshkpour ⋅ Nanyun (Violet) Peng ⋅ Gerard de Melo ⋅ Martin Potthast ⋅ Aarohi Srivastava ⋅ Abhinav Rastogi ⋅ Abu Awal Md Shoeb ⋅ Adam Brown ⋅ Adam Santoro ⋅ Aditya Gupta ⋅ Agnieszka Kluska ⋅ Diyi Yang ⋅ Akshat Agarwal ⋅ Alexander Kocurek ⋅ Ali Safaya ⋅ Ali Tazarv ⋅ Alice Xiang ⋅ Aman Hussain ⋅ Amanda Askell ⋅ Amanda Dsouza ⋅ Ameet Rahane ⋅ Anantharaman S. Iyer ⋅ Andrea Madotto ⋅ Andrea Santilli ⋅ Andreas Stuhlmüller ⋅ Andrew La ⋅ Ethan Dyer ⋅ Angela Jiang ⋅ Anh Vuong ⋅ Animesh Gupta ⋅ Anna Gottardi ⋅ Antonio Norelli ⋅ Anu Venkatesh ⋅ Arash Gholamidavoodi ⋅ Arfa Tabassum ⋅ Arul Menezes ⋅ Arun Kirubarajan ⋅ Asher Mullokandov ⋅ Austin Herrick ⋅ Avia Efrat ⋅ Ayla Karakaş ⋅ B. Roberts ⋅ Bao Loe ⋅ Bartłomiej Bojanowski ⋅ Benjamin Inden ⋅ Benno Stein ⋅ Batuhan Özyurt ⋅ Behnam Hedayatnia ⋅ Blake Howald ⋅ Bryan Orinion ⋅ Cameron Dour ⋅ Catherine Stinson ⋅ Cedrick Argueta ⋅ Cesar Ferri ⋅ Chandan Singh ⋅ Charles Rathkopf ⋅ Christian Voigt ⋅ Cindy Ramirez ⋅ Clara Rivera ⋅ Noah Fiedel ⋅ Courtney Ashcraft ⋅ Dan Garrette ⋅ Dan Kilman ⋅ C. Freeman ⋅ Daniel Levy ⋅ Daniel González ⋅ Danielle Perszyk ⋅ Danny Hernandez ⋅ David Jurgens ⋅ Deep Ganguli ⋅ Denis Emelin ⋅ Denis Kleyko ⋅ Deniz Yuret ⋅ Derek Chen ⋅ Mátyás Schubert ⋅ Derek Tam ⋅ Dilyar Buzan ⋅ Shyam Upadhyay ⋅ Dimitri Coelho Mollo ⋅ Dylan Schrader ⋅ Ekaterina Shutova ⋅ Elad Segal ⋅ Eleanor Hagerman ⋅ Elizabeth Barnes ⋅ Elizabeth Donoway ⋅ Emma Lam ⋅ Eric Tang ⋅ Ernie Chang ⋅ Ethan Chi ⋅ Ethan Jerzak ⋅ Ethan Kim ⋅ Eunice Manyasi ⋅ Evgenii Zheltonozhskii ⋅ Fanyue Xia ⋅ Fernando Martínez-Plumed ⋅ Francesca Happé ⋅ Gloria X Wang ⋅ Gonzalo Jaimovitch-Lopez ⋅ Gregor Betz ⋅ Hana Galijasevic ⋅ Hannah Kim ⋅ Hannah Rashkin ⋅ Hayden Bogar ⋅ Henry Shevlin ⋅ Hiromu Yakura ⋅ Hugh Wong ⋅ Kumar Shridhar ⋅ Ian Ng ⋅ Isaac Noble ⋅ Jaap Jumelet ⋅ Jack Geissinger ⋅ Jackson Kernion ⋅ James Zheng ⋅ Jan Kocon ⋅ Jana Thompson ⋅ Janelle Wingfield ⋅ Jared Kaplan ⋅ Jarema Radom ⋅ Jelle Bosscher ⋅ Jennifer Marsh ⋅ Jeremy Kim ⋅ Jeroen Taal ⋅ Jesujoba Alabi ⋅ Jillian Tang ⋅ Joan Waweru ⋅ John Burden ⋅ Dieuwke Hupkes ⋅ John Balis ⋅ Jonathan Batchelder ⋅ Jörg Frohberg ⋅ Jose Hernandez-Orallo ⋅ Joseph Boudeman ⋅ Joseph Guerr ⋅ Joseph Jones ⋅ Joshua Rule ⋅ Joyce Chua ⋅ Kamil Kanclerz ⋅ Karthik Gopalakrishnan ⋅ Katerina Ignatyeva ⋅ Li Zhang ⋅ Liam Dugan ⋅ Katja Markert ⋅ Kaustubh Dhole ⋅ Lucas Lam ⋅ Kevin Omondi ⋅ Kyle McDonell ⋅ Laria Reynolds ⋅ Lianhui Qin ⋅ Lidia Contreras-Ochando ⋅ Lucy Noble ⋅ Ludwig Schmidt ⋅ Luheng He ⋅ Luis Oliveros-Colón ⋅ Lütfi Kerem Senel ⋅ Maria Jose Ramirez-Quintana ⋅ Maartje Ter Hoeve ⋅ Mohit Bansal ⋅ Martha Lewis ⋅ Maheen Farooqi ⋅ Marco Baturan ⋅ Marco Marelli ⋅ Marco Maru ⋅ Marie Tolkiehn ⋅ Michael A. Yee ⋅ Mario Giulianelli ⋅ Michael Gu ⋅ Michael Ivanitskiy ⋅ Matthias Hagen ⋅ Medina Baitemirova ⋅ Mike Cain ⋅ Mimee Xu ⋅ Mitch Walker ⋅ Moin Aminnaseri ⋅ Mozhdeh Gheini ⋅ Nathan Chi ⋅ Michael Starritt ⋅ Michał Swędrowski ⋅ Michele Bevilacqua ⋅ Nayeon Lee ⋅ Neta Krakover ⋅ Nicholas Cameron ⋅ Nick Doiron ⋅ Nicole Martinez ⋅ Nikita Nangia ⋅ Niklas Deckers ⋅ Niveditha Iyer ⋅ Nuan Wen ⋅ Oliver Zhang ⋅ Omar Agha ⋅ Omar Elbaghdadi ⋅ Parth Doshi ⋅ Pascale Fung ⋅ Pegah Alipoormolabashi ⋅ Liao Peiyuan ⋅ Peter W Chang ⋅ Peter Eckersley ⋅ Phu Mon Htut ⋅ Pinyu Hwang ⋅ Piotr Miłkowski ⋅ Piyush Patil ⋅ Priti Oli ⋅ Qing Lyu ⋅ Qinlang Chen ⋅ Rabin Banjade ⋅ Rachel Rudolph ⋅ Raefer Gabriel ⋅ Rahel Habacker ⋅ Ramon Risco ⋅ Raphaël Millière ⋅ Rhythm Garg ⋅ Richard Barnes ⋅ Riku Arakawa ⋅ Robbe Raymaekers ⋅ Robert Frank ⋅ Rohan Sikand ⋅ Roman Novak ⋅ Paul Pu Liang ⋅ Rowan Jacobs ⋅ Ryan Stovall ⋅ Rylan Yang ⋅ Saif Mohammad ⋅ Sajant Anand ⋅ Sam Dillavou ⋅ Sam Wiseman ⋅ Samuel Gruetter ⋅ Sanghyun Han ⋅ Mukund Varma T ⋅ Sanjeev Kwatra ⋅ Sarah Rous ⋅ Sarik Ghazarian ⋅ Sean Casey ⋅ Sebastian Bischoff ⋅ Sebastian Gehrmann ⋅ Sepideh Sadeghi ⋅ Shadi Hamdan ⋅ Sherry Shi ⋅ Shikhar Singh ⋅ Daphne Ippolito ⋅ Shima Asaadi ⋅ Shyamolima Debnath ⋅ Simon Thormeyer ⋅ Sneha Makini ⋅ Soo-Hwan Lee ⋅ Spencer Torene ⋅ Stanislas Dehaene ⋅ Stefan Divic ⋅ Hanna Hajishirzi ⋅ Stephanie Lin ⋅ Stephen Prasad ⋅ Andrew Dai ⋅ Steven Piantadosi ⋅ Summer Misherghi ⋅ Svetlana Kiritchenko ⋅ Tao Li ⋅ Tariq Ali ⋅ Te-Lin Wu ⋅ Théo Desbordes ⋅ Theodore Rothschild ⋅ Thomas Phan ⋅ Tianle Wang ⋅ Adrià Garriga-Alonso ⋅ Tiberius Nkinyili ⋅ Timofei Kornev ⋅ Titus Tunduny ⋅ Trenton Chang ⋅ Tushar Khot ⋅ Tyler Shultz ⋅ Uri Shaham ⋅ Vedant Misra ⋅ Victoria Nyamai ⋅ Vikas Raunak ⋅ vinay prabhu ⋅ William Saunders ⋅ William Zhang ⋅ Wout Vossen ⋅ Xiaoyu Tong ⋅ Xinyi Wu ⋅ Yair Lakretz ⋅ Yichi Yang ⋅ Sophie Hao ⋅ Yifu Chen ⋅ Yufang Hou ⋅ Yuntao Bai ⋅ Zachary Seid ⋅ Cristina Garbacea ⋅ Ziyi Wu ⋅ Genta Winata ⋅ Shubham Toshniwal ⋅ Abubakar Abid ⋅ John Miller ⋅ Karen Livescu ⋅ Tatsunori Hashimoto ⋅ Ekin Cubuk ⋅ Sayan Ghosh ⋅ Harsh Mehta ⋅ Jacob Hilton ⋅ Yadollah Yaghoobzadeh ⋅ Jiaming Song ⋅ Siva Reddy ⋅ Stefano Ermon ⋅ Shashank Srivastava ⋅ Percy Liang ⋅ Chiyu Wu ⋅ James Koppel ⋅ Rui Zhang ⋅ David Drakard ⋅ Germàn Kruszewski ⋅ Dong-Ho Lee ⋅ Fatemeh Siar ⋅ Luke Metz ⋅ Roman Sitelew ⋅ Dan Hendrycks ⋅ Paul Vicol ⋅ Alexander Ray ⋅ Tobias Gerstenberg ⋅ Chris Callison-Burch ⋅ Sriharsha Hatwar ⋅ Xinran Zhao ⋅ Zijian Wang ⋅ Luca Moschella ⋅ Sam Bowman ⋅ Jaime Fernández Fisac ⋅ Danqi Chen ⋅ Stella R Biderman ⋅ Nitish Shirish Keskar ⋅ Eric Chu ⋅ Manaal Faruqui ⋅ Ksenia Shkaruta ⋅ Xudong Shen ⋅ Ryan Teehan ⋅ Vinay Ramasesh ⋅ Andy Zou ⋅ Jaehoon Lee ⋅ Hinrich Schuetze ⋅ Jesse Engel ⋅ Tal Schuster ⋅ Berk Ekmekci ⋅ Yangqiu Song ⋅ Andrew Lampinen ⋅ Dan Roth ⋅ Yasaman Bahri ⋅ Jascha Sohl-Dickstein ⋅ Jason Yosinski ⋅ Sebastian Schuster ⋅ Melody Arnaud ⋅ Russ Salakhutdinov ⋅ Nicholas Roberts ⋅ William Fedus ⋅ Sam Shleifer ⋅ Vivek Srikumar ⋅ Ronan Le Bras ⋅ Jos Rozen ⋅ Kevin Gimpel ⋅ Melvin McElrath ⋅ Omer Levy ⋅ Tal Linzen ⋅ Diganta Misra ⋅ Frieda Rong ⋅ Xiang Ren ⋅ Abhishek Rao ⋅ Mirac Suzgun ⋅ Yejin Choi ⋅ Michihiro Yasunaga ⋅ Sharon Zhou ⋅ Joshua B Tenenbaum ⋅ Sahib Singh ⋅ Michael Cohen ⋅ Tao Yu ⋅ Samuel Schoenholz ⋅ Rosanne Liu ⋅ Ryan Chi ⋅ Giambattista Parascandolo ⋅ Zhuoye Zhao ⋅ Erkut Erdem ⋅ Matthew Leavitt ⋅ Francois Chollet ⋅ Anders Andreassen ⋅ Timo Schick ⋅ Vera Demberg ⋅ Qiaozhu Mei ⋅ Daniel Khashabi ⋅ Jonathan Berant ⋅ Noah Constant ⋅ Alex Warstadt ⋅ Zirui Wang ⋅ Alethea Power ⋅ Niklas Muennighoff ⋅ Barret Zoph ⋅ Jason Wei ⋅ Christopher Manning