Today I received an email from Neil over at KohlerCreated notifying me that his website’s database was down. I checked and some other blogs were effected too, so that meant that something was up with one of the database servers :-(. We also had a VM server being removed from our rack at the datacenter today.
After being unable to connect, ping or RDP into the database server I figured that it might have been inadvertently powered down or disconnected when the VM server was being taken out.
Luckily I was able to call the datacenter admin before he left the datacenter (so he didn’t have to make a trip back) and he determined that one of the ethernet network cables had been inadvertently snagged and unplugged during the removal.
After he reconnected the ethernet, we were back in business 🙂
While we regularly backup our databases, this experience has motivated me to work toward seamless failover capability for our database servers. Within the next few months I’ll be in Tulsa to install new switches and a new backup firewall, so now I will also work on the DB server redundancy and failover.
Thank you to our bloggers for your patience, we appreciate you using our service!
PS thanks to Neil for notifying us about the issue in the first place!