On March 10, a major fire ravaged the Strasbourg data centers of OVH, the French leader in website hosting.
Since then, around a hundred technicians have been working day and night to save what can be saved and restart the servers.
But companies have lost vital data and OVH's image may suffer.
OVH on the relaunch.
Twelve days after the fire that ravaged its data centers in Strasbourg, the French leader in cloud and website hosting is restarting a large part of its services on Monday.
This extremely rare incident was triggered on the night of Tuesday to Wednesday March 10 in one of the four buildings of the Alsatian site, one of the most important of OVH in France.
The disaster, brought under control after several hours of struggle by a hundred firefighters, had affected more than 3.6 million websites.
Since then, OVH has worked hard to save what could be saved and to move forward.
>> Find the morning show of the day in replay and podcast here
Servers relaunched at no load ...
Once the flames were extinguished and the impressive column of smoke had taken off, the company's technicians entered the site to assess the damage.
Of the four Strasbourg datacenters, one was completely destroyed in the fire (SBG2).
Another (SBG1) suffered losses with four of its twelve rooms destroyed.
Finally, the last two buildings (SBG3 and SBG4) are miraculously intact.
© AFP PHOTO / SAPEURS-FOMPIERS DU BAS-RHIN
After clearing the ruins of the destroyed servers, the 130 or so OVH technicians dispatched to the site started operations to put the rescued data centers back on their feet.
First step: reconnect the electricity and connect the optical fiber.
The connections were made, with the agreement of the insurers, the week following the fire.
Second step: restart the servers.
It is painstaking work.
Each server must be inspected to see if it is in good condition.
The slightest deposit of soot can delay the restart for several days.
A colossal task when we know that there were 29,000 servers in Strasbourg before the fire.
... but extinguished after new suspicious fumes
Everything seemed to be on the right track, the SBG1, SBG3 and SBG4 servers had just been restarted, when a new incident brought operations to a halt.
On Friday evening, a release of smoke was reported in an unconnected battery room in the SBG1 building.
Applying the precautionary principle, OVH immediately cut the power supply to the data center, as well as to SBG4, the adjoining building.
All on-site activities were interrupted for a few hours.
"There was no fire, but a lot of smoke," assured OVH CEO Octave Klaba on Twitter.
Two employees present in the room were however "inconvenienced" and were taken care of by the emergency services.
An alert that led OVH to take a radical decision: building 1 will never be used again.
The intact servers will be moved to Roubaix and Gravelines and to the other Strasbourg datacenters spared by the flames.
Data permanently lost?
Everything that could be saved will therefore restart gradually during the week.
And this is where OVH will precisely measure the extent of the damage.
One thing is certain: all data from the burnt servers is lost.
For some customers, companies that did not have a backup of their data in another data center, there is a risk of losing everything.
And then there are also those who had backups but whose site has still not been restored.
OVH is virtually moving its customers to its Roubaix and Paris centers, but several business, local media or association sites, for example the National League against Obesity, are still inaccessible.
© PATRICK HERTZOG / AFP
"There is nothing that works. We lose tens of thousands of euros. Our own site accounts for 35% of our turnover", sighs David Monteiro, CEO of Homifab, a sales company of furniture online.
He regrets the lack of responsiveness of OVH: "We are in total vagueness, we have no information on the state of our back office. We have not received any personalized response, only the global communications sent. to all customers ".
Above all, the business manager wonders about the management of his data.
"We took out an offer with three automatic backups of our site in three different locations. Despite this, we still do not have access to our backups. OVH is unable to tell us where they are, which leads us to believe that all of our backups were actually on the Strasbourg site, ”explains David Monteiro.
"We feel cheated, the service is not provided. But the real problem is that our whole business can collapse overnight," he worries.
A brand image to rebuild
The survival of the French nugget is not at stake but this event is a severe halt to the ambitions of the group which hoped to go public soon.
Beyond its destroyed servers, it is also a brand image that will have to be rebuilt for OVH.
"We had chosen them because they had a good reputation and also because it is a French company. Now, it is clear that we are no longer going to trust OVH 100%. We are going to protect ourselves against this. kind of incident with backups on other servers than theirs, "says the CEO of Homifab.
To reassure angry customers, Octave Klaba, the founder of OVH, spares no effort.
Eyes marked by dark circles that betray a lack of sleep, he spoke twice in long videos posted on Twitter, his daily communication channel since the start of the crisis.
Detailed speeches to take stock of the situation of each damaged data center.
"We play total transparency", we know at OVH.
Octave Klaba also specified that affected customers would benefit from commercial gestures.
Update March 16 8:30 pm
A 10 minute summary of the situation following the fire in DC SBG2.
(french version asap) pic.twitter.com/pi5FNdpGpf
- Octave Klaba (@olesovhcom) March 16, 2021
The boss of OVH also announced the opening of a new production line in the Croix plant, near Roubaix, to replace the burnt servers.
"We have secured the supply of parts to manufacture 10,000 servers that will be produced in the next 2-3 weeks and we are working on securing 5,000 additional servers. The goal is to produce 15,000 servers in March and the first few days. April, "he detailed.
The fire investigation is progressing
At the same time, OVH is investigating to determine what triggered the fire on March 10.
The images of the 300 surveillance cameras at the Strasbourg site are currently being combed through and the company is committed to being fully transparent about the results of the investigations.
The currently preferred avenue is that of an inverter, an electric current amplifier, which would have caught fire.
However, this equipment had been the subject of a maintenance operation the day before the incident.
Human negligence cannot therefore be ruled out.
In addition, building 2, completely devastated by the flames, was built in 2011 with wooden floors.
Not ideal if a fire starts.
Finally, although the fire-fighting system was triggered in seconds, it did not stop the incipient fire, or even contain it, as it spread to Building 1. In response to criticism, Octave Klaba has already indicated that OVH will raise its safety standards.