Major Outage:
Incident Report for Intuiface

Post mortem analysis of the incident:

  • Immediately after the incident of 5-November between 8AM-11AM UTC, we needed to clean up the database and its storage on the disk.
  • We deleted old records contained within the largest table (removing records older than 16 months instead of 18 months). This deletion consumed a significant amount of journal storage (approx. 15GB per million of records), exhausting all available disk storage.
  • At around 1:30 PM UTC the database locked and entered automatic recovery mode.
  • We detected this issue three minutes later and immediately disabled the web server to limit the risk of paid Players and Composers stopping or refusing to launch. NB: Free Player and Composer remained unusable following remediation of the first incident
  • The database was repaired and placed back online after 30 minutes.
  • The web server was restarted and everything became fully operational.

NOTE: At no time was any customer data – experiences or analytics – ever at risk.

Remediation plans:

  • In the following days, the offending table will be carefully and slowly cleaned in small chunks to avoid the disk storage issue.
  • In the following weeks, we will migrate that table to a different database to avoid similar issues from occurring again.
Posted Nov 07, 2019 - 10:14 CET

Resolved is now available.
Free and paid Composers and Players can be launched again.

We will monitor our servers in the following hours and days to prevent a future occurrence of this issue.
Posted Nov 05, 2019 - 15:27 CET
Investigating is currently unavailable.
Free Player and Composer cannot be launched.

Paid Player and Composer continue to work properly.
Posted Nov 05, 2019 - 14:36 CET
This incident affected: Management Console.