24 minutes of API outage was caused by a failed database migration. The outage affected all Increase users.

At 16:00 UTC, we executed a migration against Increase’s database. The migration partially failed, resulting in an inconsistency between the deployed code and database schema. We were immediately alerted to the failure.

By 16:08 UTC, the scope of the issue had been confirmed and the engineering team gathered in a Zoom room to debug the issue and restore service.

At 16:18 UTC, the engineering team performed a data migration and deployed a new version of the API server to be in sync with the database schema. Full API service was restored by 16:24 UTC.

Increase is making changes to our Continuous Integration pipeline to ensure that database migrations of this type cannot result in data being out-of-sync with the code.

Our alerting and escalation paths worked as expected.

We apologize for the outage and appreciate your patience as we worked to restore service. For help with any specific transaction, please contact support@increase.com.