A group of law professors at the University of Minnesota used ChatGPT, a popular AI chatbot to answer exams in four courses, and graded them blindly with other students.
Jonathan Choi, a professor at Minnesota University Law School, gave ChatGPT the same test faced by students, consisting of 95 multiple-choice questions and 12 essay questions.
ChatGPT Is A Mediocre Law Student
In a white paper titled “ChatGPT goes to law school” published on Monday, the co-authors reported that the bot scored a C+ overall. While it is enough for a pass, it fell below the human average of B+. That would still be enough for the chatbot to earn a law degree, however.
“In writing essays, the AI chatbot displayed a strong grasp of basic legal rules and had consistently solid organization and composition,” the authors wrote. But the bot “often struggled to spot issues when given an open-ended prompt, a core skill on law school exams.”
While this sparked fear of students cheating using the bot, “the bigger potential for the profession here is that a lawyer could use ChatGPT to produce a rough first draft and just make their practice that much more effective,” Jonathan said.
He also wrote in reply to another Twitter user that two out of three markers had spotted the bot-written paper.
