This incident has been resolved and error rates returned to normal.
What happened: a database schema migration, executed as part of an overnight
deployment, held a database lock for longer than expected. Between 00:04 and
00:25 CEST, a portion of verification requests (up to ~30% at peak) returned
errors. Completed verifications were not affected and no data was lost; failed
requests could be safely retried.
We are updating our deployment process so schema migrations run in a
non-blocking manner to prevent this from happening again.
Resolved
This incident has been resolved and error rates returned to normal.
What happened: a database schema migration, executed as part of an overnight
deployment, held a database lock for longer than expected. Between 00:04 and
00:25 CEST, a portion of verification requests (up to ~30% at peak) returned
errors. Completed verifications were not affected and no data was lost; failed
requests could be safely retried.
We are updating our deployment process so schema migrations run in a
non-blocking manner to prevent this from happening again.
Identified
We identified elevated error rates affecting identity verification requests.
Some verification sessions may have failed or responded slowly during this
period. Our team was working on restoring normal service.
