Resolved
After 1 week, 23 hours, and 35 minutes

We have implemented some remedial changes to our platform to assist with some underlying infrastructure issues within Microsoft Azure. These changes have stabilized the platform including background jobs.

Pending infrastructure updates within Azure (in the next couple of weeks), we will be continuing to monitor for any new issues.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 1 day, 4 hours, and 24 minutes

We are continuing to work with our hosting partner to make the overall platform performance more consistent after some reports of short periods of longer page loading times.

We will continue to post updates to this page as we continue to make progress with this issue.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 10 hours and 28 minutes

End-of-day update: Queued background jobs have been caught up and processed. We are still monitoring some reports of delays in loading some functions of the platform (form manager, form builder, etc.)

Avatar for
Recovering
After 3 hours and 56 minutes

An update to the platform has been made and we are beginning to see some improvements to performance. Background jobs that have been requested are also starting to process again.

We'll be leaving this incident open to monitor for any additional issues and will post updates as we continue to see improvements to the platform.

Avatar for
{:closed=>"Closed", :complete=>"Complete", :false_alarm=>"False Alarm", :identified=>"Identified", :investigating=>"Investigating", :open=>"Open", :recovering=>"Recovering", :resolved=>"Resolved", :scheduled=>"Scheduled", :underway=>"Underway"}
After 1 hour and 36 minutes

Our support team has identified the core cause of the issues and are working on an update to the application that will be deployed shortly.

Avatar for
Identified

We are currently experiencing some performance issues that are affecting the overall platform including the time required for background jobs to process.

Our support team is aware of the issues and are working on a resolution plan.

Updates will be posted here as soon as they are available.

Avatar for
Began at:

Affected components
  • Response Management
  • Reporting
  • Email Delivery
  • One Roster Integration
  • Background Jobs