żěè¶ĚĘÓƵ

The database that is rewriting history to predict the future

A vast store of facts about the past aims to separate history from legend. It could pin down why civilisations rose and fell – and guide the evolution of our own
illustration world web
A database which crosses disciplinary boundaries
Beppe Giaobbe

“MY NAME is Ozymandias, king of kings: look on my works, ye mighty, and despair!” So run the famous lines of Percy Shelley’s poem about Ramses the Great, a pharaoh who ruled Egypt’s New Kingdom in the 13th century BC, when it was the world’s most sophisticated society. But the poem’s theme is the transience of glory. It describes the ruins of a giant statue to Ramses that lie scattered in the desert: “Round the decay of that colossal wreck, boundless and bare, the lone and level sands stretch far away.”

Why was the New Kingdom so advanced, and what triggered its downfall? There is no shortage of theories, and each has its champions. This is the way history tends to work: theories are put forward, their strengths and weaknesses are discussed, and then they line up alongside all the alternative ideas. Like old sailors, they never really die. , after the ancient Egyptian goddess of knowledge, aims to change that.

Seshat is a vast and growing database of historical and archaeological knowledge that can be explored using scientific techniques. That makes it a powerful tool for testing and ultimately discarding hypotheses. “A cemetery for theories,” is how Seshat co-founder Peter Turchin at the University of Connecticut in Storrs describes it. By making history more evidence-based, he and his colleagues hope it will become more relevant. They believe that understanding the forces that have shaped human societies in the past will give us more power to predict the future – and perhaps even to direct it by advising politicians and lawmakers on how to avoid the pitfalls of the past.

Historians may reasonably point out that their discipline has always been evidence-based. Certainly we are awash with information on past societies, but the Seshat team argues that this store of knowledge is now so huge and widely dispersed that no single human brain, or even team of brains, can hope to make sense of it all. What’s more, although some historical patterns are visible to the naked eye, many are not. Teasing out complex and subtle interactions that operate over long timescales is a task that suits computers to perfection, so perhaps it’s time more researchers invited them to collaborate? Gary Feinman, an anthropologist at the Field Museum in Chicago, certainly thinks so. He has watched how academia has responded to the deluge of data – with increasing specialisation and dwindling contact between knowledge “silos” – and sees Seshat as a precious counterweight to that. “We in historical and social scientific fields desperately need this kind of overarching, cooperative, comparative effort that crosses disciplinary boundaries,” he says.

So, how does it work? Seshat, conceived in 2010, is certainly ambitious. It divides up the world into 10 regions and selects three sample areas within each, according to natural geographic features and the level of social development. To capture as much social diversity as possible, the researchers considered when the first complex society emerged – defining a complex society as a chiefdom or state, with centralised control. They then chose one early, one intermediate and one late developer in each region.

, for now – but eventually the aim is to span the entire world. For each area, data input covers hundreds of historical variables, including the existence of a military hierarchy, the use of a calendar and the population of the largest settlement. The unit of information is a “fact”, which can take the form of a binary choice (presence/absence of writing, for example), a numerical value, or a range of values. Each fact is based on the consensus of specialist historians who meet periodically at workshops, first to decide which variables to consider, and then to validate the facts collected. Facts can be accompanied by text that expresses uncertainty or controversy and, critically, provides sources. The data spans 10,000 years from the dawn of agriculture to the eve of the modern era in 1900.

So far more than 130,000 facts have been input manually by a band of plucky research assistants. Earlier this year, they completed the first phase, which covers the broad themes of warfare, agriculture, ritual and social complexity. Now they are working on phase two, which focuses on internal collapse and encompasses the themes of well-being, institutions, equity and religion. Once in the database, the facts are machine-readable and interlinked along spatial, temporal and thematic dimensions, allowing researchers to explore a rich terrain of links and patterns (see “Under Seshat’s hood“).

“The Romans didn’t just lose territory, they lost different types of texts and the economy became simplified“

The first analysis using Seshat will be published soon. It’s an overview of the evolution of social complexity, identifying 53 variables involved, and using the database to compare the rise and fall of diverse societies across the globe since the Stone Age (see “The bumpy ride of progress”). It captures, for example, the precociousness of Egypt’s New Kingdom civilisation, with its monumental architecture, taxation and peace treaties. It also charts the heights of social sophistication reached by the Roman Empire, and its dramatic fall in the first half-millennium AD. The Romans didn’t just lose territory, “they lost different types of texts, the economy became simplified, and that complexity wasn’t recovered until the Renaissance in the 16th century”, says Turchin, who led the study. He sees it as “foundational” – more a description of general patterns than a test of alternative ideas – but the “massacre of theories” will begin soon with the publication of a slew of papers setting competing hypotheses against one another. The Seshat team refers to these as “horse races” – the goal being to eliminate the weakest each time – and they will tackle some of the thorniest issues in current debates about the past.

The bumpy ride of progress

For example, Tom Currie at the University of Exeter, UK, and others have been considering the origins of complex societies. They first emerged around 10,000 years ago, when many humans were shifting from hunter-gatherer lifestyles to farming. Nobody thinks this is a coincidence. But, says Currie, there are two main explanations. One idea is that farming supports the proliferation of specialised non-productive roles such as leaders, bureaucrats and bookkeepers because it produces more food than foraging. The second holds that, with more food, a farming population grows, giving it an edge in fights against smaller, non-farming communities. These then club together, the better to overcome the enemy, and the farmers follow suit in an arms race towards ever-greater social complexity.

To test these competing ideas, the team used the Seshat database alongside crop models originally designed to estimate productivity under different future climate scenarios. “The developers of these models alter the parameters and run them forward,” Currie explains. “We run them back.” They do this using what’s known about past climate and the timings of certain agricultural innovations. Once they have an estimate of productivity, they can calculate the proportion of a population that could fill non-farming roles without anyone going hungry, and find out how this fits with the recorded information on such things as numbers of bureaucrats and army size. The results should be published before the end of the year.

Using a similar approach, the researchers also hope to test theories that invoke climate as a major factor in the fall of societies. For example, did it trigger the collapse of the Mayan civilisation around the 9th century AD? This theory was made popular by Jared Diamond in his book Collapse. “The models are already suggesting that climate was not as big a factor in determining productivity as cultural innovations such as irrigation, manure, terracing, crop rotation and multicropping,” says Currie. The Maya were great innovators when it came to farming, so perhaps they were better able to withstand climatic change than Diamond and others have given them credit for – although, for the moment, this idea remains untested.

How to build an empire

Seshat should also be able to tell us whether agriculture alone was enough to drive the growth of empires, as some historians believe, or whether other factors were necessary too. Turchin’s own theory, for example, sees advances in military technology as an important driving force ratcheting up social complexity on either side of lines of conflict. Meanwhile, anthropologist and Seshat co-founder Harvey Whitehouse at the University of Oxford is using the database to test his ideas about rituals. These include a theory that the introduction of farming drove people to adopt new kinds of religious practices such as prayer, which in turn brought them together in larger coalitions of greater social complexity.

The team is excited to be able to put such ideas to the test, but also braced for disappointment. “If one of my theories falls by the wayside, so be it,” says Turchin. They are also hoping that their analyses will expose unsuspected interactions, perhaps in turn suggesting new theories. One person who thinks this is a real possibility is Ian Morris, a historian at Stanford University who has long argued that geography and human ingenuity interact in complex and surprising ways. To give an example, Britain’s mastery of shipbuilding and navigation put the Americas within reach and so helped overcome the geographical misfortune of being located at the extreme edge of Europe.

Seshat could help fill out the picture in other ways too. “Historians have long been aware that the sources available to them emphasise the importance of material elements over cultural ones,” says Morris. That’s because it is easier to describe or code an innovation, such as developments in pottery, than a cultural one. Yet something like a shift in religious thought or agricultural practice could be even more influential.

illustration Abraham Lincoln

Morris thinks Seshat will also help historians move beyond an emphasis on extraordinary individuals. The searchable database allows them to set, say, Abraham Lincoln in the context of his times and his contemporaries, and to explore to what extent his qualities accounted for events such as the emancipation of slaves and the preservation of the Union during the US civil war.

When the project began, there was a lot of suspicion and hostility from historians. But increasing numbers have become involved and now it’s hard to find any that object to it in principle. That’s not to say it doesn’t have its critics and limitations. Feinman, for example, points out that the current sample size of 30 societies is relatively small. So even if Seshat delivers a punch to a theory, he says, “I’m not sure it will be the knock-out blow”.

“Seshat could establish what conditions lead to stable and peaceful societies“

Others question Seshat’s objectivity. Jennifer Edmond at Trinity College Dublin in Ireland, who coordinates a digital historical research resource called , says coded data is open to interpretation. “Every time you use something like Seshat or CENDARI, there is the fingerprint of the scholars who put it together,” she says. “We can’t avoid that. The question is, how can we make it as minimal or as transparent as possible?” Computer scientist Kalev Leetaru of Georgetown University in Washington DC, who runs a digital inventory of the world’s media called , says that such initiatives will only become really powerful when machine coders replace biased humans.

Despite such qualms, the Seshat researchers believe their approach has real value. Already, relevant findings are being channelled to policy-makers, in line with the stated aims of the , a think tank in San Antonio, Florida, which supports the project. For instance, Turchin has been in dialogue with the Foreign Policy Research Institute in Philadelphia about demographic cycles and social breakdown. And Whitehouse has been talking to UK politicians with an interest in terrorism about factors that affect the structure, spread and survival of armed groups. In future, he says, Seshat could establish what conditions lead to more cohesive, stable and peaceful societies, as opposed to trying to impose “democracy” everywhere. And Turchin thinks it holds historical lessons that could inform decisions about .

Next February, when analysis of phase two data begins, Seshat researchers will focus their efforts on what causes societies to collapse. If one lesson is already clear from their analysis of the rise and fall of social complexity, it is that, in the long run, social evolution can’t happen without collapse. That is a message politicians may find hard to swallow.

Under Seshat’s hood

Seshat is the ancient Egyptian goddess of knowledge. It’s also the name of an ambitious database of historical information. Conceived in 2010, Seshat now contains more than 130,000 facts, all interlinked and machine-readable. This gives it some unique attributes. A computer can flag up contradictory entries. It can also exclude redundant variables using a statistical technique that picks out variables that correlate with one another.

Seshat’s real novelty, according to IT expert Kevin Feeney at Trinity College Dublin, Ireland, is its ability to accommodate uncertainty of all types, “unknown unknowns as well as known unknowns”. The historical record is notoriously patchy but it’s not always clear where the gaps in the information are. Seshat can highlight where details are missing or incomplete. It can even fill in some gaps using a statistical technique called multiple imputation, suggesting a range of possible answers based on what’s known about related variables.

To make all this possible, Feeney’s team built Seshat using the technology of the semantic web — a term coined by world wide web inventor Tim Berners-Lee to describe an extension of the regular web in which information is given well-defined meaning and linked in clever ways that only a computer can read. This makes the database flexible and able to learn by generalising, just as humans do. Feeney calls it “a Wikipedia for computers”. Researchers are now using Seshat to test competing historical theories and suggest ways we can learn from the past.

At the moment, a team is manually inputting the data, but in future, the process will start to become automated. Soon, for example, computers will search digital libraries such as for the best sources for a given fact. Software will be added so that data can be crowdsourced from the public and the core database will also be linked to other, more specialised historical and archaeological databases. At that point, says Feeney, Seshat will start to grow very fast.

This article appeared in print under the headline “History lessons”

Topics: History