Educational assessment might need a rethink in the wake of ChatGPT.Credit: Thomas Trutschel/Photothek/Getty
Between overwork, underpayment and the pressure to publish, academics have plenty to worry about. Now there’s a fresh concern: ChatGPT, an artificial intelligence (AI) powered chatbot that creates surprisingly intelligent-sounding text in response to user prompts, including homework assignments and exam-style questions. The replies are so lucid, well-researched and decently referenced that some academics are calling the bot the death knell for conventional forms of educational assessment. How worried should professors and lecturers be?
“At the moment, it’s looking a lot like the end of essays as an assignment for education,” says Lilian Edwards, who studies law, innovation and society at Newcastle University, UK. Dan Gillmor, a journalism scholar at Arizona State University in Tempe, told newspaper The Guardian that he had fed ChatGPT a homework question that he often assigns his students — and the article it produced in response would have earned a student a good grade.
ChatGPT is the brainchild of AI firm OpenAI, based in San Francisco, California. In 2020, the company unleashed GPT-3, a type of AI known as a large language model that creates text by trawling through billions of words of training data and learning how words and phrases relate to each other. GPT-3 is in the vanguard of a revolution in AI, sparking philosophical questions about its limits and prompting a host of potential applications, from summarizing legal documents to aiding computer programmers. ChatGPT is fine-tuned from an advanced version of GPT-3 and is optimized to engage in dialogue with users.
Avoiding the rabbit hole
Whereas GPT-3 was relatively cold and computer-like, ChatGPT can act almost as a collaborator off which users can bounce ideas. “From what I’ve seen of this, it’s so good because it doesn’t run off down a rabbit hole nearly as much as GPT-3 previously did,” says Edwards. “I just think essay assessment is dead, really.”
Others disagree that ChatGPT is such a game changer, noting that students have long been able to outsource essay writing to human third parties through ‘essay mills’. “It doesn’t necessarily add much functionality that wasn’t available to students already if they knew where to look,” says Thomas Lancaster, a computer scientist and academic-integrity researcher at Imperial College London.
Lancaster acknowledges that ChatGPT puts everything into a neat free package. But he thinks that ChatGPT-generated essays will out themselves more readily than the products of essay mills, by including quotes that weren’t actually said, incorrect information created through false assumptions, and irrelevant references.
“Despite the words ‘artificial intelligence’ being thrown about, really, these systems don’t have intelligence in the way we might think about as humans,” he says. “They’re trained to generate a pattern of words based on patterns of words they’ve seen before.”
Beyond the essay
Even if this is the end of essays as an assessment tool, that isn’t necessarily a bad thing, says Arvind Narayanan, a computer scientist at Princeton University in New Jersey. He says essays are used to test both a student’s knowledge and their writing skills. “ChatGPT is going to make it hard to combine these two into one form of written assignment,” he says. But academics could respond by reworking written assessments to prioritize critical thinking or reasoning that ChatGPT can’t yet do. This might ultimately encourage students to think for themselves more, rather than to try and answer essay prompts, he says.
How necessary that will be depends on how many people use the chatbot. More than one million people tried it out in its first week. But although the current version, which OpenAI calls a “research preview”, is available at no cost, it’s unlikely to be free forever, and some students might baulk at the idea of paying.
The situation both worries and excites Sandra Wachter, who studies technology and regulation at the Oxford Internet Institute, UK. “I’m really impressed by the capability,” she says. But she’s concerned about the potential effect on human knowledge and ability. If students start to use ChatGPT, they will be outsourcing not only their writing, but also their thinking.
She’s hopeful that education providers will adapt. “Whenever there’s a new technology, there’s a panic around it,” she says. “It’s the responsibility of academics to have a healthy amount of distrust — but I don’t feel like this is an insurmountable challenge.”
