Email Notifications Delayed

We've now resolved the incident. Thanks for your patience.

Email notifications from EHS Insight are currently delayed.

Cause:

The 20.9 release of EHS Insight included a fix for an issue that prevented reminders for a specific type of overdue Compliance Tasks from being sent.

After 20.9 was deployed this morning, most customers using the Compliance Task module were sent a very small number of "catch up" emails. Two customers were sent several hundred "catch up" emails and one customer was sent hundreds-of-thousands of "catch up" emails for overdue Compliance Tasks. To that customer, especially, we apologize for the excessive emails today and are implementing automated limitations to guard against that volume of excessive email from recurring in the future.

Unfortunately, catching up on all backlogged reminder/overdue emails today caused us to exceed our 200K daily email quota limit with AWS SES (our email provider), which has now throttled sending emails for our entire account.

Resolutions:

1. No emails have been lost, only delayed.

2. Any email addresses suspended due to bounced emails today will be cleared

3. The most impacted customers have had any queued unsent past due Compliance Task emails purged. This will, however, not prevent further per-day incremental Compliance Task reminders/alerts from being triggered going forward for any Compliance Task forms, including forms in Sandboxes that are assigned but not actually being completed.

4. Once we either are granted a temporary quota increase or as the quota beings to rollover in a few hours, delayed emails will be sent. We have elected not to fail-over to our backup email provider because we suspect many of our customer's email filtering will take more time to adjust to the change of providers than it will take to resolve the throttling with our primary provider.