Resolved -
Summary:
On April 10, 2025, Clearfly experienced an interruption in email delivery due to an unexpected IP address change by our email provider, resulting in rate-limiting. The issue was resolved by migrating traffic to an IP address with a shared reputation, and all services were restored by 10:30 AM MDT.
Incident Description:
On the morning of April 10, 2025, at approximately 8:30 AM MDT, Clearfly customers began reporting an inability to receive emails from our services. Upon investigation, we observed that all reported failures were specifically related to Microsoft servers, and the returned error code indicated rate-limiting of traffic originating from our sending IP address. Reviewing logs, we identified initial rejections starting the evening of April 9.
We were surprised to discover that our emails were originating from a new and unannounced IP address, as our previous IP address had a long-established sending reputation. A support ticket with our email provider revealed that they had retired our old IP address without prior notification and assigned us a new IP address lacking a sending reputation. Due to our email volume, this new IP address was quickly flagged and rate-limited by Microsoft's servers.
We’ve been using the same IP address with an established reputation for years, so at this point we were surprised to see that our emails were now coming from a different IP address. A support ticket with our email provider revealed the fact that they had retired our old IP address without notification and had assigned us a new IP address without a sending reputation. Due to our email volume, this IP address was quickly flagged and rate-limited by Microsoft’s servers.
Once the root cause was identified, we immediately took steps to mitigate the impact. We successfully migrated our email traffic to a different IP address with a shared sending reputation, which restored deliverability. By 10:30 AM MDT, all emails from the Clearfly Portal and Clearfly services were delivering successfully. Subsequently, by 12:30 PM MDT, we had re-queued all previously failed messages for redelivery.
Moving Forward (Preventative Measures):
1) IP Address Redundancy and Prewarming: Clearfly has proactively acquired multiple IP addresses across separate network blocks. We have begun the process of "prewarming" these IP addresses, which involves gradually building a positive sending reputation for each. This redundancy will allow us to quickly isolate and remove any single IP address experiencing deliverability issues in the future, ensuring minimal impact on overall email delivery.
2) Enhanced Deliverability Monitoring and Alerting: We have implemented more in-depth monitoring of key deliverability statistics, such as bounce rates and rejection rates. We are actively working on tuning the alerting thresholds for these metrics to proactively identify and address potential issues, ideally before any customer traffic is affected.
By implementing these measures, Clearfly is committed to preventing similar email delivery interruptions in the future and ensuring the reliability of our communication services.
Apr 10, 15:19 MDT
Monitoring -
All emails are now delivering as expected. Any emails that failed have been requeued for delivery.
Apr 10, 12:32 MDT
Identified -
We have a workaround in place while we continue to troubleshoot and are seeing emails successfully delivered again. We're continuing to work on a permanent solution and are planning changes to mitigate the changes of a similar issue re-occurring, as well as additional monitoring to ensure that we are proactively alerted to similar deliverability issues in the future.
Apr 10, 10:42 MDT
Update -
We believe we've identified the cause of an issue that is resulting in emails being throttled to any Microsoft email servers. We're escalating with our email provider and hope to have an update soon.
Apr 10, 09:21 MDT
Investigating -
We are currently investigating reports that customers are unable to receive emails from Clearfly services.
Apr 10, 08:56 MDT