Fredhopper Query API high error rate 13th March 2023 - Resolved

SUMMARY

• Regions affected: EU
• Customer Impact: Customers may have received HTTP status code 50* 

15/03/2023 

The RCA document is completed and is attached to this article.

The root cause is added below.

13/03/2023 - 04:15

The issue was resolved by our Cloud team. Once the root cause analysis is complete, it will be added to this article.

13/03/2023 - 00:30

Fredhopper started experiencing a high error rate on its Query API, which led to HTTP 500 status codes being sent. Our team started investigating the issue.

ROOT CAUSE ANALYSIS REPORT

On March 13th, 2023, at 00:00 UTC, The EU region Query API experienced a disruption when one of the authentication nodes entered a faulty state. The issue caused a portion of customer traffic to receive 500 status responses.

Attraqt determined that the faulty node was the primary reason behind the disruption and took steps to mitigate the issue.

To ensure the reliability and stability of our system, Attraqt removed the faulty node from the network and replaced it with a new one.

Improvement Options and Action Items

Attraqt will enhance monitoring and automation systems to prevent the recurrence of the incident.

 FOR MORE INFORMATION

All currently available information is included in this article. We will continue to provided updates on the issue here as we work to resolve the incident.

If you have logged a ticket with us we will provide the same information there as soon as possible.

The report of our root cause analysis investigation is usually posted here a few days after the incident has been resolved. If you have additional questions about this incident, please log a ticket with us.