Incident Summary
On Wednesday October 29th, 2025 between 4:13 PM and 4:42 PM EDT, https://app.movableink.com was inaccessible to customers. During a routine upgrade, a previously unmet bug condition in the deployment system was met which resulted in the automatic removal of a network dependency configuration. This caused the dependency to become inaccessible by resources, which resulted in a service disruption for app.movableink.com. Note, this disruption did not have any impact on the content serving path. No campaign traffic or image rendering was affected.
Resolution
On-call engineers were engaged via automated alerting at 4:13 PM EDT and immediately began to diagnose the situation. At 4:39 PM EDT the issue was identified and a mitigation was put in place to restore app.movableink.com availability. At 4:42 PM EDT it was confirmed by on-call engineers and automated alerting that service accessibility was restored. Following this, the original network dependency configuration was restored and the temporary mitigation was removed. The team continued to monitor and verified that the issue was fully resolved.
Corrective Actions
A fix to prevent this condition along with additional, automated, pre-flight guardrails have been identified and will be integrated into the continuous deployment system.