WebDesigner Posted December 24, 2022 Share Posted December 24, 2022 As all of you most likely know, all sites went down 3 days ago. Since then we have worked hard to get the server back online along with data had been corrupted restored to the expected state... Unfortunately during our technical issues we found that the issues were larger than originally expected. During the restoration process the company (whom makes the software the site runs on), despite me telling them explicitly not to, restored all databases to the last known good backup in August. Despite my best efforts to restore the current database, they had overwritten the files with a very old backup. This company is Plesk... Why did this happen? A perfect storm of sorts, read on for more technical jargon... I had set up an automatic notification system to email me had anything gone wrong. The company who makes the software, updated the software which broke my notification system. After performing a routine reboot of the server, the software moved many of the files and broke almost everything... It was at this point it was apparent to me, backups began to fail after August 5th. Me and a couple Technical Support "Engineers" at Plesk started working on the issue, it took us almost a day and a half to restore the files back to the proper state. Had the support agents listened to my instruction, we would not have lost almost 6 months of data. So what are we going to do to prevent this going forward? Firstly we are ditching the company that caused most of this, I am going to temporarily move all sites to a partnering hosting provider. There we will not use Plesk. While the site is hosted on partners servers, I am going to personally wipe and rebuild the network typography to be a proper High-Availability Server Cluster. We will not be using any one companies software, this way if one server gets unplugged the site stays up. This will also affect the backups, in which will be stored at 3 separate locations at all times. At this point we will be moving the sites back to our own locally hosted servers. From here we will improve the efficiency of the site, in addition to expanding the site and adding features. I do apologize for the inconvenience this has caused, and we will be preventing this from happening again in the future. Thank you to those who are sticking around and helping... Any comments or concerns you can post here. Quote Link to comment Share on other sites More sharing options...
Coop Posted December 24, 2022 Share Posted December 24, 2022 Thank you for your hard work. Lesson learned. We will continue to enjoy this site with a fresh start! 3 Quote Link to comment Share on other sites More sharing options...
Bushwacked Posted December 24, 2022 Share Posted December 24, 2022 wow! what terrible support from Plesk!! using 6mo old backups in the first place is a rookie move .. they should have seen the date and asked a question before going forward. That is a joke! This gives me flashbacks of when I ran the DBA team ... they had some screwups, but dang, nothing this bad. Thanks for the hard work, I know this is such a time sink when crap like this happens! 3 Quote Link to comment Share on other sites More sharing options...
Kev Posted December 28, 2022 Share Posted December 28, 2022 To say I was unhappy about this would be the understatement of a lifetime! Even though we took it in the shorts by Plesk, it's still my name and face here so, accept responsibility. My sincerest apologies for the loss of data! I exhausted a great number of hours (and offer of money) to retrieve the data without success. As we sort through our transition, things here should be pretty much normal. We're installing even more failsafes and redundancies to ensure that this does not happen in the future. Thanks to all for your understanding! Kev 4 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.