Most readers can’t distinguish classic works by poets such as William Shakespeare and Emily Dickinson from imitations generated by artificial intelligence. And when asked which they prefer, they often chose the AI poetry.
“Over 78 per cent of our participants gave higher ratings on average to AI-generated poems than to human-written poems by famous poets,” says at the University of Pittsburgh in Pennsylvania.
Porter and his team prompted OpenAI’s ChatGPT-3.5 chatbot to generate five poems in the style of 10 famous English-language writers. Those poets, whose works span medieval to modern times, included Geoffrey Chaucer, William Shakespeare, Walt Whitman, Emily Dickinson, T.S. Eliot, Allen Ginsberg and Sylvia Plath.
Advertisement
The researchers randomly assigned a poet to each of their 1600 study participants and asked them to read 10 poems; five written by the human and five AI-generated fakes mimicking the poet’s style. The participants consistently failed to identify AI-generated versus human-written work, guessing correctly just 46 per cent of the time.
In a follow-up experiment, some of the participants rated the poetry based on factors such as originality, meaning and rhythm. On average, they gave lower ratings to human-written poetry and rated AI-generated poetry higher on factors such as beauty and emotion. However, if the participants were first told that a poem was written by a human poet – regardless of the poem’s actual authorship – they consistently rated those works more highly.
The fact that AI chatbots can imitate famous human poets may not be surprising, given that they were probably trained on the works of those writers, says Porter. “Essentially, ChatGPT has displayed its skill as a quasi-plagiarist,” says at the University of California, Los Angeles.
Despite participants’ apparent confusion, there are still indicators that can flag poems as something not written by a human. The AI-generated poetry featured obvious themes that made it “very straightforward and easy to understand on your first read”, says Porter. By comparison, human-written poems often require analysis and historical context to understand them.
The AI-generated poems also followed a more traditional structure with basic rhyme schemes and consistent stanzas, whereas human poets may break from convention. “Emily Dickinson sometimes breaks the expected rhyme scheme on purpose,” says Porter. “But the AI-generated poems generated in her style never did that once.”
Scientific Reports
