Resolved

We've now resolved the incident. Thanks for your patience.

The performance and timeout issues we experienced yesterday were caused by the interaction between a third party component/security update and a non-typical server load pattern that stressed external service calls (SCORM training registrations, custom inline email body resources, etc).

We have reverted the libraries that contributed to the issue and are continuing to research and experiment with long-term solutions.

Recovering

We are still seeing and working on slow requests for attachments and customer uploaded logos.

Recovering

Online Training courses are continuing to have sporadic timeouts, due to an ongoing outage at our SCORM hosting provider: https://status.cloud.scorm.com

Recovering

All customers are out of maintenance mode at this point. We are continuing to recover and some requests may continue to be slower than usual.

Investigating

We are continuing the recovery from this morning's outage. Customers who are currently unaffected may experience maintenance modes periodically as the environment is recovered.

Investigating

We're experiencing an elevated level of Web and API errors and are currently looking into the issue.

Began at:

Affected components
  • Web Site and Native Apps
  • API