The Outage… Oi!

Where to begin? Firs things first: All Services are now 100% online.

Where we stand:

Here’s where we are with things right now:

  • Chuck is Incredibly Tired (and a little bit hungry)

The outage has brought about some interesting things, though:

  • Increased followers on Twitter, Facebook and Google+
  • Huge increase to the trouble ticket list (largest ever in 10+ years!)
  • Clients were updating their details.

So one bad thing, three positive things. God’s got it all figured out.

Where we are going:

Outages are never fun, and you can plan and plan and plan some more. However, things like a data center wide power outage aren’t something planned for. The current provider has had problems with power in the past (back in 2014) but since has been stable.

Now I’m not a fan of just running when something goes wrong. I mean we serve God. He’s all about forgiveness and redemption! So forgive and forget right? Right. But that does not mean I will stay at the full capacity I have been.

Grace and Church are the two remaining stragglers, it’s been part of my plan for a while now to move them, and I’ve already got server hardware delivered. Biggest servers yet.

The Plan:

Migration. I’ll post more once I have a tad more rest under my belt. The long-term goal is to move Grace and Church sooner than later. Now with next to nothing in sleep for well over a week I need rest. So if I can wing some rest soon, I’ll work likely one or the other this weekend or next.

Apology:

I’ve been blessed with amazing customers. Go ahead, high five yourself from Me. While I do understand those customers who are frustrated (you are no less awesome) are upset because they are using hosting services and they are down, I appreciate all of you for sticking it out.

As always, I own up to this, as a one man show I could say “Oh the datacenter dropped the ball! Wasn’t me!” but let’s be honest. I chose this company. I placed your sites on these servers, they failed. That’s my bad. I chose to forgive and forget but I must build things back up and keep them stable.

I am going to do more, and do some contests with some prizes as well as my way of saying thanks – the cool thing is all active and current customers are entered! So hang tight, by FlockUpdate time in a few weeks I’ll have more on that and more about stabilizing everything.

Thank you all for sticking with me! Appreciate everyone’s support! Hopefully we’ll have all things cleared up very shortly!

Trouble Ticket Wisdom: Is my email/site down?

knowledgeA common trouble ticket that finds its way to the support desk is

“I can’t seem to get email, my website is down! Why is everything offline?”

Well there are a few reasons as to why things could actually be offline:

  • Scheduled Maintenance
  • Unplanned Server Outage
  • IP Blocked by Firewall

So what do you do when these things occur? How do you know? Well let’s go over a few ways that your ticket would be cleared up:

Scheduled Maintenance

It happens sometimes, there are software updates which require a little downtime, some are quickly scheduled for safety’s sake, others planned weeks in advanced to address bigger issues, or even just general housekeeping! So how do you find out about these sorts of outages? Email! First and foremost I email customers about LARGE outages – so be sure your email on file is to an address NOT associated with your service.

Now for quick and priority always be sure to check Twitter or Facebook, chances are if it’s quick you’ll see a notice, and sometimes a rundown of what’s going on.

Unplanned Server Outage

This happens! I’m sorry, sometimes things outside of my control, hardware failure, power issues, datacenter server attacks, you name it! So when things go completely wrong and my hands are tied waiting for the datacenter I tend to keep updates on Twitter and Facebook, if it’s something major it is posted here at the blog too!

IP Blocked by Firewall

You set up a new email address and the password won’t take – after about 3 tries because you may of just hit S instead of D you realize that it is giving you a new error – chances are you got blocked by the firewall which likely thinks with all this password failures you are up to no good!

Previously I’d of sent you to another website, but now I send you to FlockHosting’s website to find your IP address give the number in green to me in a ticket, and ask to see if you have been blocked, if it’s not a firewall block the IP still helps me determine what is going on!

There you have it!

You’ve just read up on several scenarios related to outages and being unable to access your website, do remember you can check on the fleet server status by checking in the customer zone – but usually the above will help you stay up to date!

Have a question? Trouble Ticket Wisdom will come from tickets but too if you have a question about how to do something, open a ticket and let me lend you a hand! Who knows your issue may appear in one of these posts!

Outage: Praise & Koinonia

Two servers in the same datacenter are being effected by a network outage, the datacenter currently does not have an ETA as they are still investigating the outage – I’ll be updating this Post and Facebook as I know more! Sorry for the outage, hopefully it will quickly repaired.

Update: All services have been restored, network is once again functionally normally.

RFO: Praise

Not to take away from Customer Appreciation Month – But today’s extended outage did throw some customers for a loop, and the Reason For Outage (RFO) is better laid out in a post vs. Twitter/Facebook posts.

Earlier this AM the server praise had a brief outage, due to a drive failure in the storage array the exact response from the datacenter was:

One of RAID disks failed. We are working on its replacing.

Easy peasy – it was replaced and only a minor hiccup as the storage system went briefly into read-only mode. Fast forward to later in the day, to rebuild the redundancy and get the new drive properly in the RAID array, it must be re-sync’d into the system, thus what we experienced this afternoon, tackled with extremely high load on websites & a storm of SPAM hitting the servers filters, it was a combo of just bad timing.

The server went down, and in coming back up ran a FSCK (for Windows Users, think Scan Disk) it checked the disk to make sure everything was working correctly and this can take some time – but things leveled out and came back online with some handy help from the datacenter crew, things are back online.

Now please be aware there is still a sync process on-going and its syncing a LOT of data, so sluggish response may continue – please bear with me on this, I wish I could speed it up, but redundancy isn’t something I take lightly.

The server will continue to be closely monitored and backups still are on-going so your data is safely backed up – but if you have any questions or concerns about the outage today, please do not hesitate to contact me via Email, Ticket, Forums or Live Chat.

Thanks again for your patience and prayers!

Titus to Solomon

Earlier this year Titus was setup, it was a move in how hosting “could” be done using a 64bit platform – for the most part things have gone smoothly but in the realm of upkeep, security, and issues like modules, backend support etc. have made me have to re-focus and stick to a 32-bit setup.

So to that end Titus will be de-comissioned this Saturday after a migration of data to Solomon, all customers on Titus should have received a email with this information – but for those who may not of, this is being cross posted here on the blog, as well as Facebook and Twitter. Read the email that was sent out below.