Customer Appreciation Month is always planned to be a time of fun and showing my appreciation for you the customer – but sadly this year hit some server outage today with Koinonia or to the folks who more commonly use it “Church Hosting” – and so today my focus was more on restoration of services vs. posting videos and goodies, because Church Hosting while free, is incredibly important to Me! Plus we can party all the rest of this week!
So for those of you wondering “What happened?!” well drive failure. All servers run on what’s referred to as “RAID” basically improving drive performance and redundancy by using many drives – in this case the RAID setup is part of a device called a SAN (think BIG hard drive) which we lease access on, high redundancy and too good availability – but like all things computer, failures happen.
When noticed some files were mixing where they shouldn’t and restoring backups worked for short while but even after repaired things continued to go wrong. I contacted the datacenter and they too began working on what appeared to be a larger issue on their SAN, so they replaced the effected drives and began rebuilding the RAID so it could return to a healthy state, as it can tolerate several lost drives, but in this case it hit our data spot on causing the hiccups, clean data from working areas would try to populate, failed areas in the active server would hiccup it.
With the help of the guys in the datacenter we moved things around, ran some repair which took some time, and got things back online and running smoothly – and while I am grateful of the hardwork by the DC guys, I always want to keep downtime to a minimum.
So whats the plan? How do we prevent this from happening ever again? Sadly we can’t always — I could move to another company, I could build a new server, but bottom line is the DC worked to the best of their ability – but already in talks about further redundancy to help reduce the overall downtime – but I do thank everyone who lifted the matter up in prayer – and the patience of churches as things were in and out, and still a little slow – but even more tonight tuning things to see if I can get a little more performance out of things.