Skip to end of metadata
Go to start of metadata

After identifying the cause of the failure of two of our firewall cluster nodes, Internet access has been restored at 8:47 a.m. after a downtime of approx. 63 minutes.

We are sorry for the inconvenience caused.


Cause

While doing routine security updates on our high-available 4-node (two pairs of two nodes) firewall cluster a kernel update installed, which required reboots of the individual nodes. The architecture of our firewall cluster allows for individual outages of either one node of the two pairs, with just minimal interruption (typically < 20s) while switching the active role in each pair.

The kernel update today contained a driver update for the network adapters used for the interconnect between the inner and the outer pair of our firewalls. The new driver requires a newer firmware code compared to the old driver from the previous running kernel. Unfortunately the newer firmware code was not provided by the software distributor. Booting back the previous kernel solved the connection for now.

A bug report has been opened with the software distributor.