On 13-December (a Friday), after midnight (UTC), an Intuiface employee using API Explorer reported an outage. The Intuiface Server Team also received automatic alerts about the lack of server response. These outages occurred several times until the morning of Monday, 16-December.
On 16-December, the Server Team performed root cause analysis, during which it was discovered that the outage issue mainly affected Intuiface employees. Further analysis identified the root cause of performance degradation as an unused but resource-costly feature only available to Intuiface employees. Non-employees encountered API Explorer outages as a side effect.
On the morning of 17-December, a plan was executed to remove the unused feature and rebuild the server. The rebuilt server was tested and ready at 2PM UTC. In addition, to proactively compensate for possible performance issues in the future, the host supporting the server was accelerated by a factor of two.
Moving forward, five optimization options have been identified, with implementation occurring over the coming weeks. These optimizations will tune request storage to accommodate an ever-growing database. These requests are used by the machine learning recommendation algorithm in API Explorer.