XO Service Incident - June 15th 2022 - Resolved

SUMMARY

• Date of incident: June 15th, 2022 
• Customer Impact: All customers using the basic algorithm
  • Most Popular
  • Most Trendy
  • Random 
  • User Last View

Will return 0 products; If there is no ALS configured in the use case, it will probably result in empty blocks 

                                                                                                                                                                          

Update 16th June - PM

This issue has now been resolved. The root cause of the problem has been identified and fixed, and all algorithms are working as expected.

 

You may observe old data in the item list within the XO Console whilst the fix propagates, however, this will not impact Production, and all product data, tags, and APIs are working again.

                                                                                                                                                                          

Update 16th June - AM

The XO product experienced a reoccurrence of yesterday's issue, resulting in the basic algorithms not recommending any items. A widget using only basic algorithms, would not have returned any recommendations. This is now resolved.

Updates of catalogs or tags may still take longer than expected, full resolution is expected later this afternoon. We ask that until we have confirmed the issue has been fully resolved that you do not click on the 'Launch an import' or 'Recompute tags' button as this could cause further delays. 

                                                                                                                                                                          

Update 15th June

The problem has now been identified and fixed. The issue occurred due to a tag rule shared on all tenants, which was on all products, causing the number of tags to increase to a degree that impacted the service that uses the tags (Basic).

The problematic tag rule has been deactivated and this change is in the process of recomputing across all tenants, during this time updates of catalogs or tags may take longer than expected.

                                                                                                                                                                          

Wednesday 15th June

Since this morning, the XO product is experiencing an issue with the basic algorithms not recommending any items. A widget using only basic algos, won't return any recommendations/items returned.

Currently, the Attraqt team is working to resolve the situation and an update will be posted here.

                                                                                                                                                                          

ROOT CAUSE ANALYSIS REPORT

TBD

                                                                                                                                                                          

 FOR MORE INFORMATION

All currently available information is included in this article. We will continue to provide updates on the issue here as we work to resolve the incident.

If you have logged a ticket with us we will provide the same information there as soon as possible.

The report of our root cause analysis investigation is usually posted here a few days after the incident has been resolved. If you have additional questions about this incident, please log a ticket with us.