27th November 2019

Overall Cluster State Long Running Maintenance

During todays maintenance, we discovered an issue with the most recent kernel for RHEL systems. The issues causes systems to crash completely and reboot. We are currently investigating solutions, including a downgrade of all systems.

Update 03:00 We are downgrading all the kernels on RedHat and will inform here as soon as this has happen. The situation should start to improve and we will update about the progress in an hour.

Update 04:00 We are in the process of downgrading all affected systems. We expect to have all clusters up and running by 06:00.

Update 06:00 With the end of the official maintenance window, most clusters are up and running again. For a few we are still applying the workaround. We will inform these customers individually.

** Final ** Sorry for the inconvenience during this maintenance. We will prepare a Post mortem and publish it here.