Update day 4
2nd patch seem to have solved it. Awaiting a RCA from vendor.
Update day 3
Services are working but are rendering some timeouts. We are working together with the software vendors developers to solve it.
Update day 2 11:00
System is still throwing a few exceptions. We are investigating and re-opening the case as it was accidentally closed by us.
Update 22.00
Fixed is installed on all nodes and services are restored.
Update 16.00
The patch is under development and will be punt into production within 1-2 hours.
Update 15:00
Service is still not responding as intended to some calls. Issue has been escalated to the software vendors development team and we are awaiting an ETA of the patch.
Update 13:00
Nothing to report on. We are still investigatin how to resolve the issue.
Update 11:54
The fix that we tried to apply did not seem to have any effect. Working together with software vendor trying to resolve the issue.
Update 11:22
We believed that we have found the cause to the incident and are about to try to apply a fix.
Our S3 storage is currently denying some connections. We are investigating and updating here.