Service Incident EU1 region - April 14th 2022 - Resolved

SUMMARY

• Date of incident: April 14th, 2022 
• Regions affected: EU1
• Customer Impact: Increased response time and increased number of errors in the Query API 

Thursday 14th April 08:15 UTC

This issue has now been mitigated and all response times and error rates should return to normal. We will provide an RCA once the root cause analysis investigation has been completed.

Should you continue to have issue please log a ticket with us.

Thursday 14th April 07:20 UTC

Our teams believe they have identified the cause of the issue and are working to resolve it as soon as possible.

Thursday 14th April 06:03 UTC

Since 04:00 AM UTC, we are experiencing high resource consumption on the EU load-balancing layer.
Customers in that region might observe spikes in response times and increased error rates.
Our teams are working to resolve the situation as soon as possible.

ROOT CAUSE ANALYSIS REPORT

On 2022-04-14 at 4 AM UTC, Attraqt received a number of alerts relating to an increased error rate across multiple customer accounts in the EU region. 6-7% of the query API traffic resulted in 503 statuses. The problem was caused by an unprecedented amount of simultaneous configuration requests to the external load balancing layer produced by over increased capacity for some cloud accounts. Attraqt readjusted the dedicated capacity and implemented a cap for such kinds of updates to the load-balancer layer to mitigate the incident.

 FOR MORE INFORMATION

All currently available information is included in this article. We will continue to provide updates on the issue here as we work to resolve the incident.

If you have logged a ticket with us we will provide the same information there as soon as possible.

The report of our root cause analysis investigation is usually posted here a few days after the incident has been resolved. If you have additional questions about this incident, please log a ticket with us.