Mails sent by intuiface are delayed or lost
Incident Report for Intuiface
Postmortem

The issue

After deploying a server update, some minor errors were reported via email to the Intuiface staff. Due to a very high number of such notifications, our email servers were overloaded, causing us to reach the 24 hour email quota imposed by our email service provider, AWS. All emails were blocked, including all emails generated by the Hosted instance of Intuiface’s Share via Email Interface Asset.

In total, the blockage affected the following services:

  • Hosted Share via e-mail Interface Asset,
  • Emails sent to warn users that Intuiface Players are going online / offline
  • Emails sent when sharing an experience.
  • Registration confirmation emails
  • Forgotten password emails
  • Invoice emails

The affected e-mails were either delayed or lost.

The fix

We asked our cloud provider (AWS) to increase our email quota.
At around 2PM UTC on the 21st of November, our email buffer was emptied and emails were once again correctly sent.

Remediation

To avoid this email overload problem, we have separated our email servers. An independent server is now used to send (possibly very high volume) notifications to the Intuiface staff. As a result, internal notification emails will no longer interfere with the quota for emails created for and by our users.

Posted 18 days ago. Nov 22, 2018 - 16:09 CET

Resolved
The issue was solved. Emails created by the Hosted instance of the Share via Email Interface Asset are being sent again as of 2PM UTC.
Posted 19 days ago. Nov 21, 2018 - 15:30 CET
Identified
The issue has been identified and a fix is being implemented.
Posted 19 days ago. Nov 21, 2018 - 09:00 CET
Investigating
Emails are not being sent to our customers (payment-related emails, registration confirmation, password reset...) or to third parties when using the Hosted version of the Share via Email Interface Asset.
The issue has been diagnosed and is currently being corrected.
Posted 19 days ago. Nov 21, 2018 - 09:00 CET
This incident affected: Management Console.