Huddled round a screen showing an animation of a busy street, a group of researchers watch as a child emerges from behind a parked car and is mowed down by a truck. The simulation resets and the same scenario plays out again. And again. Each time, the pedestrian is hit, but in different weather conditions: heavy snow, dense fog or pouring rain.

The animation looks almost like a version of Grand Theft Auto (it is based on the same game engine) but this is no console title. The research team at CARISSMA, a leading research and test centre for vehicle safety in Ingolstadt, Southern Germany, is working on a significant problem for autonomous vehicles. How to make them safer in extreme weather, when their cameras and sensors are impaired by snow, fog or rain.

The CARISSMA team is part of a €9.7 million Europe-wide project called ROADVIEW— a 15-partner research consortium with members from seven countries including the UK, Germany, France and Sweden. All made possible with funding from the EU’s flagship €96 billion research and innovation programme, Horizon Europe.

Extreme weather

The 4-year project brings together some of the best expertise in Europe on autonomous vehicle safety, sensor technology and simulation modelling to achieve results that no single country could achieve on its own. The partners come from universities, research led-companies and car manufacturers, all with the overall goal of solving the extreme weather problem for self-driving vehicles.

“It is very important to bring different partners from different countries together. To bring different thinking together,” says Professor Werner Huber, Head of the CARISSMA Institute of Automated Driving.

Though companies such as Waymo and Tesla have already brought driverless taxis to public roads in the US, their roll-out in Europe has been slower because of a more cautious attitude to safety.

Experience from the US has thrown up situations where the cars have struggled in unusual conditions. In September last year, for example, torrential rain caused acute flooding in Arizona. Driverless Waymo taxis struggled in the conditions with . That led the company to suspend its operations temporarily.

“You don’t want a system that works only in ideal conditions,” says Professor Valentina Donzella, at Queen Mary University of London, whose team of engineers is part of the ROADVIEW consortium, “It’s important to be able to demonstrate safety in adverse weather or complex conditions.”

You need data, data, data and you can’t get enough from real world tests

That’s difficult because of the need to accumulate enough driving experience in extreme scenarios. “You need data, data, data and you can’t get enough [from real world tests] to find out what the car should do in a specific situation,” says Huber. That would be too dangerous on a public road.

The answer is to train the cars’ sensors and control systems in the lab or on the test track using data that simulates extreme weather conditions, something that Huber and Donzella’s teams are working on together. “The UK partners have great experience in this kind of modelling,” says Huber, “It’s not that easy to model bad weather situations into a simulation environment since you need a sensor model which … is really close to reality.”

In simple terms, it means taking real-vehicle motion and reproducing it in a simulation where the data from cameras, LiDAR and radar sensors is degraded to mimic what the car would see in heavy rain, snow or fog. That way, the researchers can take real-world motion and use it in extreme weather tests.

The geographical spread of the ROADVIEW partners confers an advantage, says Donzella. She points out that the nature of fog, snow and rain is not uniform across the continent and that impacts the way autonomous vehicles “see” the road.

“We realised that, depending on the country, we have different types of [fog, rain and snow],” says Donzella, “just discussing with the partners and understanding what we have to model in terms of adverse weather was really interesting.”

In the UK’s maritime climate, for example, snow tends to be wetter than in Finland or Scandinavia. “When it’s dry, it flies more [and] is more affected by wind,” says Donzella, “You have to take that into account [in the modelling].”

Critical scenarios

In Ingolstadt, Huber’s team uses the modelling expertise from Queen Mary to fool real-world cars on the institute’s test track into believing they are driving around in extreme weather conditions. That allows them to repeat critical scenarios like the child crossing the road — but in any weather.

Huber says the collaboration with the UK team has been “very fruitful” with each side bringing different expertise to the table. “UK partners have a special kind of thinking. They are critical and they are innovative and I think German partners are the same, maybe from another perspective,” he says, adding that both sides have long expertise with the automotive industry.

Donzella agrees. “It’s a matter of complementary skills, different expertise,” she says, “It’s been very, very productive.”

The project, which runs over four years, has also provided a structure for deep and long-term collaboration. “It gave us enough time to know each other, understand strengths and how to work together,” says Donzella. Members of her research team have spent time at the CARISSMA test track in Ingolstadt, Germany and researchers from that team have made the reciprocal journey to London.

That has made for a deep and effective partnership, she says. “Obviously we leverage complementary skills, but also complementary facilities, different viewpoints. Some things that can be more challenging here in the UK are maybe less challenging in Germany due to different road infrastructure and so on.”

That’s vital to building an autonomous driving system that will be safe in conditions right across the European continent. “Working together, we can bring this different cross-geographical perspective,” she says.

The ROADVIEW project comes to an end in August, but both sides intend to continue working together. “We have understood that environmental conditions are really challenging … and we need to consider everything together. So the complexity is very high and there are still a lot of things that we need to tackle,” says Donzella.

“I hope this isn’t the end. I hope this is the start of a long cooperation,” says Huber.

Watch to find out more: Inside the partnership helping driverless cars navigate extreme weather 

Find out ĚýĚý

A humanoid robot that can drive a car could one day be used as a chauffeur, though its creator concedes that this may take at least 50 years.

Most driverless cars work very differently to a human driver, using artificial intelligence and custom mechanical systems to directly move the steering wheel and pedals. This approach is much more efficient and simpler than using a humanoid robot to drive, but it is also bespoke for each particular car.

at the University of Tokyo and his colleagues have developed a humanoid robot, called Musashi, that can drive a car in the same way as a human. It has a human-like “skeleton” and “musculature”, as well as cameras in each of its eyes and force sensors in its hands and feet. Artificial intelligence systems work out what actions are needed to drive the car and react to events such as traffic lights changing colour or a person stepping in front of the car.

The robot can only perform a limited range of driving tasks at present, such as going forward in a straight line or taking a right-hand turn, moving at speeds of around 5 kilometres per hour on non-public roads. “The speed of the pedal or the velocity of the car is not high. Also the handling of the car is not fast compared to human beings,” says Kawaharazuka.

However, Kawaharazuka hopes that once the system improves, it will be able to work in any car, which could be useful for when humanoid robots are routinely produced. “I’m not looking 10 or 20 years in the future, but I’m looking 50 or 100 years away,” he says.

Free newsletter

Sign up to The Daily

Sign up to newsletter

“This study is potentially interesting for people developing humanoid robots, but doesn’t tell us much about autonomous driving,” says at University College London. “Self-driving cars don’t and shouldn’t drive like humans. The technology doesn’t have to rely on limbs and eyes so it can find other, safer, more useful ways to move through the world, relying on digital maps and dedicated infrastructures.”