ChatGPT, the AI chatbot developed by tech company OpenAI, can find and fix bugs in computer code as well as standard machine learning approaches 鈥� and does even better when engaged in conversation.
at Johannes Gutenberg University in Mainz, Germany, and his colleagues sought to see how well ChatGPT compared with other AI-powered coding support tools. A number of tools exist that use artificial intelligence to check programming code to ensure there are no mistakes.
鈥淐hatGPT came out and we thought it seems like it can also be capable of automatic bug fixing, but we don鈥檛 know how it works,鈥� says Sobania. One of the key things he and his colleagues wanted to test was whether engaging in dialogue with the chatbot would improve its ability to fix errors in code.
Advertisement
The researchers first asked ChatGPT to answer questions taken from the QuixBug benchmark dataset, which is a series of small but challenging programming questions. For example, they gave the AI a small snippet of code and asked : 鈥淒oes this program have a bug? How do you fix it?鈥� All the code used in the experiment was in the Python programming language.
ChatGPT managed to correctly answer 19 out of 40 queries put to it 鈥� comparable with two other deep learning-based code fixing approaches, called CoCoNuT and Codex. That roughly 50 per cent success rate was considered state-of-the-art for such tools before ChatGPT.
However, that was just part of the experiment. The researchers then utilised ChatGPT鈥檚 conversational interface to ask follow-up questions that a user would pose if they tried to insert the corrected text into a programming tool.
This approach highlighted where ChatGPT鈥檚 solution was incorrect and meant that in 31 of the 40 queries, ChatGPT solved the issue. 鈥淭hat was really surprising, because we haven鈥檛 seen it before,鈥� says Sobania. 鈥淭hat鈥檚 something new.鈥�
Sobania expects ChatGPT or similar systems to be deployed as an additional troubleshooting tool for programmers in the future.
鈥淎s an aid to programmers, it鈥檚 a good idea,鈥� says at the University of Surrey, UK. 鈥淚 can see it ultimately being used not just to check static code, but to help IDEs [software development programs] to make suggestions of how to correct or improve.鈥� However, Woodward warns that human oversight is still needed: 鈥淲e don鈥檛 want to rely totally on the AI as it is not infallible.鈥�
Reference: Arxiv,
