Internet giants Google and Facebook have discovered they are experiencing computer chip failures that can corrupt data or make it difficult to unlock encrypted files. Facebook says hardware manufacturers must take notice of the problem, which has emerged due to the vast scale of computing resources the firms use.
The issue surfaced at Google when multiple teams of engineers reported problems with their computations, but the company鈥檚 usual diagnostic tools showed no problem. An investigation revealed that individual chips were responsible for repeated faults. In certain cases, researchers could prompt problems by changing a chip鈥檚 temperature.
These 鈥渟ilent errors鈥� are caused by bits on the chips flipping from 0 to 1 or vice versa. Cosmic radiation can cause bits to flip, so computers destined for space have to be specially designed to prevent this. The errors spotted by Google and Facebook manifest in a similarly sudden way, but are instead due to ever-shrinking chips exhibiting unpredictable behaviours.
Advertisement
Google says the issue is down to manufacturers squeezing an ever-higher density of components onto chips in a quest for higher performance, which leaves smaller margins for error. It says that problems can manifest months or years after installation as chips essentially wear out and develop issues.
An industry rule-of-thumb known as Moore鈥檚 law states that the number of transistors in a circuit will double roughly every two years, but the rate of miniaturisation has slowed as engineers reach circuit densities beyond which electrons cannot be reliably controlled.
These faults can lead to corruption of small amounts of data, but if that data is itself key to unlocking or finding other data, then the effects can quickly magnify. In one case, Google found that files encrypted by one machine could only be decrypted by that machine, even when the correct key was used on other computers.
While the flaws may be exceedingly rare for individual computer chips, companies like Google and Facebook rely on hundreds of thousands of servers, each containing numerous chips and devices that can individually contain a billion transistors. Google declined to speak to 快猫短视频 about the issue, but in a it pointed to finding hundreds of affected chips.
The company now tests chips throughout their lifespan to identify suspect chips as they age, and says it has devoted decades of cumulative engineer time to solving the problem.
Sriram Sankar at Facebook, , says that, in his experience, the issues could be found across all chip manufacturers. 鈥淭his problem of silent errors is going to happen across silicon devices. Whether CPUs or custom chips, you鈥檙e gonna have silent errors.鈥�
Sankar says the problem can occur during design, manufacturing or after months or years of operation. 鈥淭he reason we call them silent errors is because there is no error correction or error detection of these errors,鈥� he says. 鈥淭hat鈥檚 why this is an important area that the industry should focus on. I think all of the industry should take notice.鈥�
One solution is redundancy 鈥� critical systems such as aircraft computers might have three machines all running the same code and constantly comparing the output to check accuracy 鈥� but the cost of doing this across Google or Facebook鈥檚 vast operations would be prohibitive.
Sankar and his colleagues propose several cheaper fixes, including running automated tests on chips during maintenance or using less intensive testing applications that can run alongside a chip鈥檚 everyday tasks. Facebook has started doing this on the company鈥檚 servers over recent months, says Sankar.
Once these error-detection systems have found a problematic chip, engineers can simply turn it off and leave it in place, which is the most efficient solution, or replace it with a new chip. Sometimes, a faulty chip is left running in a server, but is virtually handed over to a team that investigates the nature of silent errors. This zoo of troublesome chips informs Facebook鈥檚 research and helps to develop even better tools for detecting them, says Sankar.
at the University of Sheffield, UK, says chips are now so complex that he would be surprised if any of them are designed and manufactured perfectly.
鈥淧rocessors do very strange things,鈥� he says. 鈥淭hey鈥檙e coming up with cleverer and cleverer ways of doing things, having many, many cores, and you get some really strange behaviours.鈥�
