Incident Summary
On Monday October 20th, 2025 between 3:10 AM and 6:40 AM EDT, the Studio platform experienced degraded performance during which customers could have experienced slow loading or failure to load dynamic campaign assets out of our US East datacenter. Separately, between 8:28 AM and 11:03 PM EDT, reporting and analytics features within the Studio platform were unavailable.
Resolution
On-call engineers were engaged immediately and it was determined that degraded performance was due to a combination of regional and global service unavailability within our cloud service provider US East datacenter. Once this issue was identified, engineers attempted to shift all traffic away from our US East datacenter to our US West datacenter and raised a high priority issue with our cloud service provider’s technical support team. Due to the underlying issues within our cloud service provider, repeated attempts failed until 6:30 AM EDT with a full failover of US East traffic to US West completed at 6:40 AM EDT. Once this failover was completed, error rates within the platform dropped off and content serving was restored to normal performance.
As service provider issues persisted, on-call engineers were again engaged regarding availability of reporting and analytics. A separate high priority issue was raised to cloud service provider support, who determined that this issue was directly related to the ongoing service provider issues. The data and site reliability teams continued to monitor the situation with the service provider throughout the day and as the situation gradually resolved it was determined at 5:15 PM EDT that the site reliability team could begin sending a portion of traffic back to US East. US East was again serving normal traffic load by 5:45 PM EDT, restoring normal operations of load shared between US datacenters.
Reporting and analytics underlying systems were not restored until 11:03 PM EDT at which point historical data was available, but the prior day's data had not been successfully processed and remained unavailable. Reporting systems continued to process backlogs through EOD Thursday October 23rd, 2025 at which point the backlog was fully processed.
Corrective Actions
This was a result of a one-off, global service unavailability disruption within our cloud service provider in the US East datacenter, and there are no corrective actions to be taken to prevent recurrence.