Using GitHub Copilot to Solve Simple Programming ProblemsOnlineGlobal
The teaching and assessment of introductory programming involves writing code that solves a problem described by text. OpenAI Codex is a trained machine learning model that aims to do exactly that. Previous research found that Codex performs well on many programming problems, often generating correct and readable Python code. While Codex requires payment, GitHub Copilot, an IDE plugin that uses Codex, is freely available to students. This raises pedagogic and academic integrity concerns. Educators need to know what Copilot is capable of, in order to adapt their teaching to the age of AI-powered IDE plugins. Previous research investigated the most performant of the four Codex models and evaluated it quantitatively: how many tests does each generated code suggestion pass and how many problems have at least one correct suggestion that passes all tests. Here I evaluate Copilot instead, to see if and how it differs from Codex, and look qualitatively at the generated suggestions, to understand the limitations of Copilot. I also report on the experience of using Copilot for other activities asked of students in programming courses: explaining code, generating tests and fixing bugs. The paper concludes with a discussion of the implications of the observed capabilities for the teaching of programming.