The prowess of the chatbot, powered by masses of data from the Internet and capable of writing texts in response to simple questions, has aroused since its launch at the end of last year the admiration but also the fears of many Internet users.
Some results were so convincing that teachers at several universities worried about a risk of widespread cheating and the end of traditional methods of teaching in the classroom.
Jonathan Choi, a professor at the University of Minnesota Law School, gave ChatGPT the same test that students take for graduation: 95 multiple-choice questions and 12 review questions in four disciplines.
In an academic paper published on Monday, Choi and his co-authors said the robot earned an overall grade of C+.
Although this score is sufficient for admission, the robot finished near the bottom of the class in most subjects, the academics point out.
“While writing its essays, ChatGPT demonstrated that it has mastered basic legal rules and that its organization and composition are consistently sound,” the authors write.
However, the chatbot "often struggled to spot problems when asked an open-ended question, which is an essential skill in law school exams," they nuance.
Authorities in New York and other jurisdictions have banned the use of ChatGPT in schools, but Choi believes the robot could be a great teaching aid.
"Overall, ChatGPT hasn't been a very good law student when acting alone," he wrote on Twitter.
“But we believe that by collaborating with humans, language models like ChatGPT can be very useful for law students taking exams and practicing lawyers,” he added.
To allay concerns about cheating, Choi also said two out of three proofreaders spotted the chatbot-written assignment.
"[They] had a hunch and their hunch was right, as ChatGPT had perfect grammar and was somewhat repetitive," he wrote.
© 2023 AFP