Between the hours of 6:12PM and 6:51PM ET on Tuesday, January 6th, Studio users may have experienced issues with CSV active integrations (data sources). A change was deployed to production intended to improve CSV ingestion performance which inadvertently exposed a previously unidentified permissioning issue in production storage. This caused CSV ingestion failures. Once the change was fully reverted normal CSV processing resumed. No data loss occurred during this time, and no other data ingestion paths were affected.
The deployed change introduced new file handling behavior that interacted with production storage permissions in an unexpected way. These permission constraints were not present or enforced in local development environments, preventing detection during pre-deployment testing. Once CSV ingestion errors were detected in production, Engineers were able to trace the issue to storage permission constraints added as a result of the deployment. Once the deployment was reverted to the previous stable version, expected CSV ingestion behavior was immediately restored.
In order to prevent similar events from occurring in the future, our Engineering team is working to bring our staging environment to functional parity with production.