Skip to end of metadata
Go to start of metadata

Network service interruption

The interruption was caused primarily due to a design failure in the physical setup of the virtual infrastructure hardware.
The four machines operating the VMware vSphere system are redundantly connected to mains, one line is run over a UPS. All machines are connected redundantly to two network switches, which do not have the possibility of a redundant power supply connection, hence are connected to a UPS line.

This morning at around 7:15 a.m. the logging facility of our UPS states that a short outage occurred, which obviously did not do any more harm, apart from a tripped circuit breaker. The design fault was, that both network switches for the virtual infrastructure have been connected to the same breaker downstream of the UPS. As a result the virtual infrastructure was isolated from the outside.

Upon resetting the breaker at about 7:45 a.m. the switches connected the virtual hosts again.

The network switches are now connected to separate circuits to avoid this scenario in the future.