Over the past few years, development teams working with artificial intelligence (A.I.) and machine learning have devoted more attention to building tools that will autocomplete your code. That’s potentially a huge development for all kinds of developers and engineers—those with experience can work faster, and those just starting out will have a helping hand when it comes to basic-level coding.
But these autocomplete tools have some downsides. For starters, programming is a complex art that involves a good deal of creativity. How can you build a system that accurately anticipates the coder’s intentions on the next line, much less a few hundred lines further on? While these tools may seem appealing in theory, in practice they might feel like a down payment on better versions that could emerge many years from now.
With all that in mind, GitHub has launched the technical preview of a new autocomplete tool, Copilot, that supposedly anticipates many of these concerns around using A.I. to code. Specifically, it relies on an algorithm called OpenAI Codex, adapted from the OpenAI’s GPT-3 natural-language generator, to accurately predict outputs. Like GPT-3, which was trained to mimic human language via a massive dataset of text, the OpenAI Codex’s creators fed tons of publicly available code into the system.
“GitHub Copilot draws context from the code you’re working on, suggesting whole lines or entire functions,” reads a note on GitHub’s corporate blog. “It helps you quickly discover alternative ways to solve problems, write tests, and explore new APIs without having to tediously tailor a search for answers on the internet.” In other words, the platform is trying to replace a partner or team who might help you carry out all those functions.
But how accurate is it? On the FAQ on the GitHub Copilot page, there’s a… complicated answer: “We recently benchmarked against a set of Python functions that have good test coverage in open source repos. We blanked out the function bodies and asked GitHub Copilot to fill them in. The model got this right 43% of the time on the first try, and 57% of the time when allowed 10 attempts. And it’s getting smarter all the time.”