We all are aware of the dangers that can bring down a data center like human error, fire, power issues (hello Delta Airlines), floodings, earthquakes and so on. However, you probably were not aware excessive noise can damage hard drives which as a result bring down a complete datacenter!
Technical people and customers of ING Bank in Romania found out the effect of sound on hard disks. ING Romania did a fire extinguishing test in the Bucharest datacenter on Saturday September 10 2016. Staff of ING opened the cylinders to dump the inert gas into the datacenter. However the noise of the gas dump was at such level that the vibration damaged many hard drives resulting in server being unavailable.
Over one million ING customers weren’t able to withdraw money from the bank’s ATMs and couldn’t pay by card on Saturday between 1PM and 11PM. The ING Bank online banking system was also down. See the ING press release. ING has a market share of 10% in Romania.
The datacenter issue also disabled the communication methods like email, website and SMS. So ING in the first hours could not explain to their customers what happened. It took the bank over four hours to post its first message on its Facebook account.
Motherboard reports about this incident here.
The bank monitored the sound and it was very loud, a source familiar with the system told us. “It was as high as their equipment could monitor, over 130dB”.
Vibrations caused by fire extinguishing equipment causing damage to hard disks is not the only danger of fire extinguishing equipment. Aerosol has a nasty side effect as well as it damages electronic equipment. See the post here for details.
It is remarkable ING suffered from this incident. Roughly around 2007, reports started to surface saying that hard disk drives (HDDs) were damaged during gas discharges from Inert Gas Fire Suppression System as Datacenterjournal.com reports. There are a couple of reports written by IBM and Siemens about this issue.
The Siemens White Paper indicates that various HDD fail after exposure to noise above 120dB. This noise level is reached during fire extinguishing by either Inergen or other extinguishing gas. Also the sound level of the alarms should be added to the source of noise.
In 2010 a datacenter of hosting provider WestHost was down for 6 days for the same reason as reported by Availabilitydigest.com. This is another case of damaged HDD’s.
In December 2015 the email system of Glasgow City Council was down for three days likely because of vibrations caused by fire suppression system.
However the manufacturer of Inergen (the gas used in the extinguishing equipment) found out the fire alarm siren (horn) can also damage hard disks.
In 2008 this engineer demonstrated the effect of sound on hard drives.
The problem is Acoustic noise. See Datacenterjournal:
When the gas is released from the pressurized cylinders it moves through the pipes at very high velocity. On exit through multiple nozzles in the data center, it generates high-level acoustic noise. The noise reaches the HDDs where it causes vibration, which in turn causes the read/write element to go off the data track. Current-generation HDDs have up to about 250,000 data tracks per inch on their disks. To read and write, the element must be within ±15% of the data track spacing. This means the HDD can tolerate less than 1/1,000,000 of an inch offset from the center of the data track—any more than that will halt reads and writes.
In this video a demonstration of a inert gas dump is given. The people inside the room got ear plugs to protect their ears.