Thread by @olesovhcom

Nous avons un souci d'alimentation de SBG1/SBG4. Les 2 arrivées électriques EDF sont down (!!) et les 2 chaines de groupes électrogènes se sont mis en défaut (!!!). L’ensemble de 4 arrivées elec n'alimentent plus la salle de routage. Nous sommes tous sur le problème.

En plus de souci sur SBG, nous avons le souci sur le réseau optique en Europe qui interconnecte RBX et GRA avec les POP. Il est down (!!).

SBG: ERDF is trying to find out the default. 2 separated 20kV lines are down. We are trying to restart 2 generators A+B for SBG1/SG4. 2 others generators A+B work in SBG2. 1 routing room is in SBG1, the second in SBG2. Both are down. #Murphy

SBG: 1 gen restarted.

RBX: all optical links 100G from RBX to TH2, GSW, LDN, BRU, FRA, AMS are down.

SBG: 2 routing rooms are UP powered by gen. EDRF still down. ETA: 15min
RBX: the DB of the optical node in RBX is corrupted. All links are still down. We upload the backup. ETA: 30min

RBX: restore of the setup in progress

RBX: the links are coming UP.

RBX: UP
SBG: in progress

SBG: ERDF repared 1 line 20KV. the second is still down. All Gens are UP. 2 routing rooms coming UP. SBG2 will be UP in 15-20min (boot time). SBG1/SBG4: 1h-2h

SBG: 2 routing rooms are UP. we are restarting the servers in SBG1/SBG4.

SBG: elec is UP. all racks are UP. we are checking which services/customers is still impacted.

Résumé des 2 incidents qui ont impacté #Ovh ce matin. #OVHGate
travaux.ovh.net/?do=details&id…

Summary of this morning 2 incidents #Ovh

Résumé Incident de RBX
travaux.ovh.net/?do=details&id…

Comment by Octave - Thursday, 09 November 2017, 17:53PM

Il reste encore en panne:
- 2100 serveurs dédié
- 1500 instances PCI
- 25000 VPS
- 300 hosts PCC

travaux.ovh.net/?do=details&id…

Résumé de la panne à SBG
travaux.ovh.net/?do=details&id…

all the details about SBG
travaux.ovh.net/?do=details&id…

SBG: 100% of the infrastructure has been UP since yesterday 11h00.
Since yesterday 11h00, we are working on the hardware issues: we are remplacing broken servers with the spares.
travaux.ovh.net/?do=details&id…

Port-mortem Incident RBX
travaux.ovh.net/?do=details&id…

Le suivi SBG
Comment by Octave (oles) - Saturday, 11 November 2017, 20:10PM
travaux.ovh.net/?do=details&id…

We've finished to fix the last PCI/VPS hosts. If your VPS/PCI is still down, please DM me the tech deatils (IP/Name of VPS/PCI)
On a fini de fixer le hardware du dernier host PCI/VPS. Si votre VPS/PCI est encore down, merci de m'envoyer DM avec les infos tech (IP/Nom)

today, we have 11 teams x 3 persons each to fix the last hardware issues.

Les cellules haute tension 20kV de SBG1 avec les cellules motorisées et l'automate NSM qui n'a pas marché. #Ovh

Les groupes électrogènes de SBG1 (2x2MVA), SBG2 (2x1.4MVA) et SBG3 (2x1.4MVA) #Ovh

Les navettes déposent à SBG les équipes fraiches qui viennent de Roubaix et on récupère ceux qui ont tout donné. #Ovh

It remains: 18 servers.

This night, we've fixed the last servers with the hardware issue. All serveurs are UP. Our apologies.

Cette nuit, nous avons fixé les derniers serveurs avec le problème hardware. Tous les serveurs sont UP. Toutes nos excuses.

SBG: >50000 servers. Usually, there is 1-2% of the servers that don't restart after a brutal electic shutdown, because of hardware issues. We had 0.84% of servers that we fully changed, new hardware. It took us <4 days last server.

We continue to work on the servers in SBG: rebuild spare stock, stock of servers in SBG, rebuild the servers in SBG3 (we took lot of pieces from the new servers prepared for SBG3), clean-up the operations, ..

Incident is closed.

If you have any issue, please DM me or contact the support. We have the software teams that will help you to fix your setup issues (firewall, etc).

Point suite aux deux incidents du 9 novembre 2017
ovh.com/fr/news/presse…

Following the two incidents on the 9th of November 2017
ovh.co.uk/news/press/cp2…