Degraded performance with Link Creation

Incident Report for Linkfire

Postmortem

On Saturday 16th of September, an outage in our cloud server provider caused our primary links database to become inaccessible, which affected the links module in the Suite. As a result, users were unable to see their links inside the app, as well as creating and editing links.

Existing Links and landingpages remained unaffected. No tracking data from the links was lost during the incident.

We sincerely apologies for any inconvenience this has caused. We are re-evaluating our cloud server setup, and will be building in more redundancy to combat situations like this going forward.

Incident Duration: 11 hours and 18 minutes (7:26 AM UTC to 6:44 PM UTC)

Impact: No user was able to see, create, edit or delete links in the Suite.

Posted Sep 18, 2023 - 13:53 CEST

Resolved

The incident has been fully resolved and all systems are now running normally. Althought there was connectivity issues to an internal database, no data was lost during this period.

Posted Sep 16, 2023 - 21:15 CEST

Monitoring

The issue has been mitigated and all functionalities in the Suite have returned to normal. We will continue to monitor the issue for some time to ensure that everything remains normal.

Posted Sep 16, 2023 - 20:48 CEST

Update

Although the issue persists, we are making good progress to mitigate it. The initial tests have shown positive results, and now we will proceed with its implementation. We remain extremely cautious with every action we take to ensure that other systems are not affected.

Posted Sep 16, 2023 - 19:53 CEST

Update

Work continues to address the connectivity issues. Our focus remains on migrating the database to a different region without losing data. We are being extremely cautious with every action taken to ensure no further issues arise, hence the long recovery time we are experiencing.

Posted Sep 16, 2023 - 18:11 CEST

Update

The connectivity issue to the internal databases persist, therefore no visible changes can be seen in the Suite.

We're currently working on restoring the database in a different region, which should enable a workaround to the connectivity issue, without loosing any other functionality or data.

Posted Sep 16, 2023 - 17:07 CEST

Update

Links module is still experiencing issues. We're currently looking into different ways of mitating this issue as soon as possible.

Posted Sep 16, 2023 - 15:45 CEST

Update

Although the Suite remains usable, we continue to experience issues with link the link creation functionality. Updates will be issued as events warrant.

Posted Sep 16, 2023 - 12:40 CEST

Identified

Due to connection issues to our internal databases, we're experiencing a degraded performance with the link creation functionality in the Suite.

No other system is affected by this outage.

Posted Sep 16, 2023 - 10:33 CEST

This incident affected: 🎶 Application (🖥 app.linkfire.com).