roommaster - Audit Process Freezing Issue – Incident details

Resolved: Night Audit Issue Fully Resolved

The issue impacting night audits has now been fully resolved.

If you still have not yet performed your night audit for last night, please do so at this time. All audits should now complete successfully, and system performance has returned to normal.

We sincerely appreciate your patience and understanding as we worked through this issue.

Update: Night Audits Available

If you still have not completed your night audit for today, please go ahead and try again now.

The system has stabilized, and recent attempts have been completing successfully.

⚠️ Please do not stop or restart the process once it has begun, even if it appears slow. Allow the audit to finish completely to avoid further delays.

Thank you for your patience as we worked through this issue.

If you have not yet been able to run your night audit and are seeing one of the following messages:

⚠️ “Problem: Backup of files could not be done before the audit…”
⚠️ “Problem: Night audit is currently disabled for a maintenance window…”

Please know that we are aware of the issue.

At the moment, when we re-enable the night audit process, a very large number of customers attempt to run it at once. This sudden surge places heavy demand on the system, requiring us to temporarily disable night audits until the current workload clears. Once audits in progress are completed, we re-enable the process, allow more audits to finish, and then repeat this cycle.

We understand how disruptive this is and sincerely appreciate your patience as we work through this staged recovery. Our priority is to ensure every customer can complete their night audit successfully as system load balances out.

👉 Important: If you were able to start your night audit but it is taking longer than 30 minutes to complete, please contact our support team for assistance.

Update: Night Audit Availability

If you have not yet been able to run your night audit and are seeing one of the following messages:

• ⚠️ “Problem: Backup of files could not be done before the audit…”

• ⚠️ “Problem: Night audit is currently disabled for a maintenance window…”

please hang in there.

Each time we re-enable the night audit process, a very large number of customers attempt to run it at the same time. This sudden surge overwhelms the system, forcing us to temporarily disable night audits until the bottleneck clears and existing runs are completed. We then turn the process back on again for a short window, allow more audits to finish, and repeat.

We understand this is frustrating and truly appreciate your patience as we work through this staged recovery. Our goal is to ensure all customers can complete their night audit successfully as system load balances out.

👉 Note: If you were able to run the night audit but it is taking an extremely long time (over 30 minutes) to complete, please reach out to our support team for assistance.

While the underlying issue from very early this morning that prevented night audits from running has been resolved, we are continuing to experience periods of heavy demand as many customers attempt to perform their night audits at the same time. This surge can cause system performance to slow, and in order to protect overall stability, we have had to temporarily disable the night audit at intervals.

When this happens, you may see one of the following warning messages:

• “Problem: Backup of files could not be done before the audit…”

• “Problem: Night audit is currently disabled for a maintenance window…”

These safeguards are intentional and are designed to reduce system strain and allow processing to catch up.

👉 If you encounter either of these warnings, please wait and try again later. The night audit will complete successfully once system performance has stabilized.

We know how important this process is for your operations, and we sincerely apologize for the disruption and extra effort this has caused.

As we shared earlier, performance is expected to return to normal as the system catches up on processing. However, we want to provide an important update regarding night audit availability.

When we re-enabled the ability to run night audits this morning, a large number of customers attempted to run them all at once. This immediate surge caused performance to slow again, and in order to protect the system, we were forced to temporarily disable night audits.

⚠️ As a result, you may still see one of the following messages when attempting to start the night audit:

• “Problem: Backup of files could not be done before the audit…”

• “Problem: Night audit is currently disabled for a maintenance window…”

These safeguards remain in place intentionally to prevent additional system strain.

👉 We ask for your patience as we carefully reintroduce night audit access in phases to allow processing to stabilize. If you receive either of these warnings, please wait and try again a little later.

We sincerely apologize for the disruption and understand how critical the night audit process is to your operations. Our team is actively monitoring and working to ensure full performance is restored as quickly as possible.

Thank you for your patience and cooperation.

Earlier this morning, we experienced a system issue that temporarily prevented customers from running the night audit. That issue has now been resolved, and night audits can be processed successfully.

Because the system became available again at the same time for everyone, many customers are now attempting to run the night audit simultaneously. This heavy load is causing the process to run more slowly than normal, and in some cases, it may take several hours to complete.

⚠️ Important: Please do not restart the restore process multiple times if it seems slow. The audit must be allowed to fully complete. Restarting it will only add to delays and can cause additional issues.

Performance is expected to return to normal later this morning as the system catches up on processing.

You may also see one of the following warning messages when attempting to start the night audit:

• ⚠️ “Problem: Night audit is currently disabled for a maintenance window…”

• ⚠️ “Problem: Backup of files could not be done before the audit…”

Both of these messages have been intentionally put in place to prevent additional customers from starting the night audit and further overwhelming the system.

👉 If you receive either of these warnings, please refrain from running the night audit until after 10:00 AM EST.

We sincerely apologize for the disruption this has caused and appreciate your patience as the system fully recovers. If you encounter any further issues beyond slow performance, please don’t hesitate to reach out to our support team.

Earlier this evening, we experienced a system issue that temporarily prevented customers from running the night audit. That issue has now been resolved, and night audits can be processed successfully.

Because the system became available again at the same time for everyone, many customers are now attempting to run the night audit simultaneously. This heavy load is causing the process to run more slowly than normal, and in some cases, it may take several hours to complete.

⚠️ Important: Please do not restart the restore process multiple times if it seems slow. The audit must be allowed to fully complete. Restarting it will only add to delays and can cause additional issues.

Performance is expected to return to normal later this morning as the system catches up on processing.

You may also see the following error message appear when attempting to start the night audit:

⚠️ Warning  "Problem: Backup of files could not be done before the audit…"

This message has been put in place intentionally to prevent additional customers from starting the night audit and further overwhelming the system. If you receive this error, please refrain from running the night audit until after 10:00 AM EST.

We sincerely apologize for the disruption this has caused and appreciate your patience as the system fully recovers. If you encounter any further issues beyond slow performance, please don’t hesitate to reach out to our support team.

Notice: Night Audit Procedure Running Slowly

We want to make customers aware that the night audit procedure may take significantly longer than usual to complete.

⚠️ Important: Please do not stop or restart the night audit if it appears to be running slowly. It must be allowed to fully complete, even if the process takes longer than expected.

Interrupting and restarting the audit can cause further delays and issues. Performance will continue to stabilize as the system completes its processing.

Notice: Night Audit Procedure Running Slowly

We want to make customers aware that the night audit procedure may take significantly longer than usual to complete.

⚠️ Important: Please do not stop or restart the night audit if it appears to be running slowly. It must be allowed to fully complete, even if the process takes longer than expected.

Interrupting and restarting the audit can cause further delays and issues. Performance will continue to stabilize as the system completes its processing.

⚠️ Important: Please do not restart the restore process multiple times if it seems slow. Allow the system to complete the process.

Resolved: Night Audits Restored – Currently Running Slowly

The issue preventing night audits from running has been resolved, and night audits can now be processed successfully.

At this time, please note that night audits may run more slowly than usual while system processing fully catches up. Performance is expected to return to normal very shortly.

⚠️ Important: Please do not restart the restore process multiple times if it seems slow. Allow the system to complete the process.