Issue resolved

Incident Report: Picturepark DAM Unexpected downtime - March 31st, 2024

Initial cause

On Sunday, March 31st, 2024, our monitoring systems detected inconsistencies within our databases. Subsequent investigation revealed that our SQL server experienced repeated crashes and restarts, stemming from a host failure within our cloud hosting provider’s infrastructure.

Resolution

In response to the outage, our emergency reaction team doubled its efforts to mitigate the impact on our services. One team focused on restoring the existing system, while another simultaneously built a new environment from scratch. By Tuesday, April 2nd, we successfully deployed the new system and restored data from the most recent backups, (early morning of March 30th, 2024).

Impact

Initially, the platform experienced instability, progressively worsening until it became entirely inaccessible. Upon recovery, data was reverted to the state captured by the March 30th backups. Regrettably, any changes made between March 30th and April 2nd were lost during this process.

We understand the frustration and inconvenience this disruption may have caused and sincerely apologize for that.

Follow-up actions

• Engaging with our cloud hosting provider to implement measures aimed at preventing similar incidents in the future.
• Continuously reinforcing our backup protocols to enhance data resilience and recovery capabilities.
• Notably, our Content Platform remains unaffected as it operates on a separate infrastructure provided by Azure.

Next Steps

Should you have any inquiries or require further assistance, please do not hesitate to reach out to your Key Account Manager or contact our support team at support@picturepark.com.

We appreciate your understanding and patience as we work diligently to ensure the stability and reliability of our services.

The issue has been resolved. The platform is available again as expected.
An incident report will follow.

We truly apologize for any inconvenience this may have caused and thank you for your patience and understanding.

We’re currently performing emergency maintenance on Picturepark and the platform is unavailable. We’re striving to complete this maintenance swiftly, but at the moment, we can’t provide an exact timeframe for when services will be restored. We truly apologize for any inconvenience this may cause and thank you for your patience and understanding.

The platform is currently experiencing issues. We are investigating this and will update you here. Apologies for any inconvenience.