Service Incident with Insights reporting tool - August 17th - Resolved – Rezolve Ai Support Portal

SUMMARY

Date of incident: Issue started on August 5th 2022
Regions affected: All customers using analytics data for reporting
Customer Impact: Data not being processed because of a bug and reports are not accurate since 6th August

Tuesday 17th August 2:00 PM UTC

A rollback of the affected code was performed, and now the data should be processed as normal.

Tuesday 17th August 12:30 PM UTC

Attraqt has identified an issue with the analytics data that is sent to the Insights Reporting tool. The issue started on 6th August but was discovered today. The impact is that this data couldn't be processed and might be missing from the reports. The Development team is currently working to fix the issue, and we will provide an update once this is done.

They notified us that as this is an ingestion pipeline, it won't be possible to retrieve the missing data.

ROOT CAUSE ANALYSIS REPORT

On 5th August, a code change was deployed, which resulted in the processing of tracking data being slowed down, resulting in a backlog not being processed. As a result, all tracking data sent to Attraqt was lost between 05/08 & 17/08. Due to a fault in the monitoring methodology, the issue went unnoticed until August 15th.

Improvement Options and Action Items

After analyzing the situation, the following improvement options have been added.

Enhancing the Automated Monitoring and Alerting for FHR Insights.
Updated manual monitoring process to include more validations (we will continue performing and improving this process until the automated process is fully enhanced and functioning)
Updated deployment process for code changes to further minimize the possibility of faulty code being accidentally deployed.

Please find the whole RCA document attached below.

FOR MORE INFORMATION

All currently available information is included in this article. We will continue to provide updates on the issue here as we work to resolve the incident.

If you have logged a ticket with us we will provide the same information there as soon as possible.

The report of our root cause analysis investigation is usually posted here a few days after the incident has been resolved. If you have additional questions about this incident, please log a ticket with us.