Immediately after the incident of 5-November between 8AM-11AM UTC, we needed to clean up the database and its storage on the disk.
We deleted old records contained within the largest table (removing records older than 16 months instead of 18 months). This deletion consumed a significant amount of journal storage (approx. 15GB per million of records), exhausting all available disk storage.
At around 1:30 PM UTC the database locked and entered automatic recovery mode.
We detected this issue three minutes later and immediately disabled the web server to limit the risk of paid Players and Composers stopping or refusing to launch. NB: Free Player and Composer remained unusable following remediation of the first incident
The database was repaired and placed back online after 30 minutes.
The web server was restarted and everything became fully operational.
NOTE: At no time was any customer data – experiences or analytics – ever at risk.
Remediation plans:
In the following days, the offending table will be carefully and slowly cleaned in small chunks to avoid the disk storage issue.
In the following weeks, we will migrate that table to a different database to avoid similar issues from occurring again.
Posted Nov 07, 2019 - 10:14 CET
Resolved
my.intuiface.com is now available. Free and paid Composers and Players can be launched again.
We will monitor our servers in the following hours and days to prevent a future occurrence of this issue.
Posted Nov 05, 2019 - 15:27 CET
Investigating
my.intuiface.com is currently unavailable. Free Player and Composer cannot be launched.
Paid Player and Composer continue to work properly.