One major difficulty of programming is turning concepts into code, especially when dealing with the APIs of unfamiliar libraries. Recently, there has been a proliferation of machine learning methods for code generation and retrieval from natural language queries, but these have primarily been evaluated purely based on execution accuracy or overlap of generated code with developer-written code, and the actual effect of these methods on the developer workflow is surprisingly unattested. In this talk, I will describe a user study in which we performed a comprehensive investigation of the promise and challenges of using such technology inside the PyCharm IDE, asking questions such as “does it improve developer productivity or accuracy, how does it affect the developer experience, and what are the remaining gaps and challenges?”