19th May 2020

App Router OpenShift router memory usage unusually high

We're seeing an unusual increase in memory usage on the APPUiO application routers, we're currently investigating the situation. Applications may experience short interruptions.

2020-05-19 11:37: We're still investigating the situation, we haven't found the cause yet. Unfortunately we could not avoid the application routers being killed due to their excessive memory consumption.

2020-05-19 11:54: The memory usage has normalized for the moment, but we still have not found the root cause. Currently we believe that the increased memory usage was caused by an increased number of long-lived connections on both application routers. We're continuing to observe the situation.

2020-05-19 13:20: Both application routers are again showing unusually high memory usage. We are still investigating possible causes.

2020-05-19 14:25: Both application routers are still showing higher-than-normal memory usage. Connections may get interrupted briefly, as the application routers are consuming enough memory to be forcibly restarted by the system. We're still investigating possible causes.

2020-05-19 16:25: We're still seeing unusual memory usage pattern on both application routers. At this point, we believe that the increased memory usage stems from an increased number of open websocket connections. We will tune some configuration parameters for the application routers in tonight's maintenance window. We hope the tuned configuration parameters will improve the stability of the application routers.

2020-05-19 17:45: The memory usage has stabilized since 16:45. We're still going ahead and will adjust the router configuration to hopefully improve the stability of the routers.

2020-05-20 09:45: The configuration changes were applied to the application routers during the regular maintenance window. We're continuing to observe the application routers. Currently the application routers look stable.

2020-05-20 11:40: So far we don't see any spikes in memory usage on the application routers. We consider this issue resolved for the time being.