A HISTORICAL puzzle over the authorship of a 19th-century religious work has been solved with a paradoxical trick: throwing away what looks like the best data.
Moshe Koppel and Jonathon Schler from Bar-Ilan University in Ramat-Gan, Israel, were approached by historian Shlomo Havlin, also at Bar-Ilan, with a famous Jewish text called the Torah Lishmah. A rabbi called Ben Ish Chai published the collection of letters, ostensibly written by other rabbis, in Baghdad in the late 1800s. Ben Ish Chai claimed he had stumbled across them, but many historians think he wrote them himself.
Koppel and Schler鈥檚 task was to settle the question, one way or the other. They measured hundreds of features of the suspect text, such as the frequency of words, sentence length and grammatical patterns. When they compared the results with a similar analysis of a book known to have been written by Ben Ish Chai, they found differences that suggested the authors were indeed different people, as the rabbi had claimed.
Advertisement
But something wasn鈥檛 quite right. Closer inspection showed that only a few features separated the books. For example, one contained the phrase 鈥渢each us o master of righteousness鈥� while the other preferred simpler formulations, such as 鈥渢each me鈥�.
So the researchers came up with a novel idea: throw away the obvious differences, and examine the rest. Comparing the remaining results revealed how similar the books were. After three iterations, the differences between the two tomes began to disappear. When the researchers compared two texts by different authors, the differences fell away much more slowly.
It now looks as if Ben Ish Chai had, as suspected, written the Torah Lishmah letters. Discarding what seem like the best distinguishing features is a counter-intuitive strategy, but it seems to get the best results. 鈥淚 was astounded at how well it worked,鈥� Koppel says.
Koppel and Schler got equally dramatic results when they tested the technique, which they call 鈥渦nmasking鈥�, on works by English-language authors. So not only have they solved a long-standing mystery, they have also stumbled on a way of solving other literary conundrums.
