Server Migrations – The Plan

work-n-manI have finally had a chance to get some sleep in, and finished up the plan to move some things around and it should be rather quick and painless – I am making some changes to your DNS if you are managing your own DNS keep an eye out for a ticket regarding your site(s) move, as it will contain new information on how to make adjustments to point to new IP’s – there will be 2 migrations, one for Praise and another for Koinonia, and here is the plan!

Praise will be making a migration this next week on Wednesday March 20th at 10PM (Pacific) – I will be sending each of you information about the move – and nothing will change resource wise, you’ll still have the resources promised in the original giveaway all those many moons ago!

Koinonia will be making a migration over the next weekend starting Friday March 22nd at 10PM (Pacific) – As this is Church Hosting, I am doing a few modifications as things are *slightly* different on Church Hosting accounts, and need to get things all moved around and sitting pretty with the new hardware.

Now this was going to happen much sooner however on top of everything else on-going, cPanel released a new version which I wanted to settle in first before making a HUGE jump between servers and having more issues, so I do appreciate your patience!

The goal in this move is speedy-move! I am making some adjustments in how frequently servers look for updates, this is called TTL (Time to Live) – making some adjustments to this will help speed along the DNS changes to your visitors! If your managing your own DNS the ticket will let you know what and when to make adjustments to your Zone Files to speed along with the rest of us.

This process should be painless, as this isn’t my first round of migrations – over the years the process has gotten better and better, so hoping for a quick data transfer, and a quick DNS update to get things moved over and much more stable!

If you have any questions in the mean time please don’t hesitate to contact me with ANY questions you may have! Hope everyone has a GREAT weekend in the mean time, and I’ll be in touch!

Outages (Boo!)

No one is a fan of outages, especially me! I do my best to avoid them if I can, and much of the time it’s smooth sailing – but recently as customers on Koinonia and Praise have noticed, it’s not been a good few days! Outage, then back online, then down – currently still working on the on-going issue with Praise – but as some aren’t big users of Twitter or Facebook I wanted to post a big update as I sit and wait on the Datacenter.

Koinonia had a big outage lasting over 7 hours after some datacenter issues, they have assured me it was an “isolated” issue, and is resolved, but it doesn’t bolster confidence that days later the similar issue hit Praise – and Praise is still as of this writing offline, they are working to get things back online – but they are using the words now “RESTORE” from a previous backup – so I am patiently waiting to see where we stand as backups are kept, but rolling back to a old version always runs the risk you don’t always have your latest posts/files/emails – and that’s not my desire – but if data corruption did occur, something is better than 100% loss, but still not what I’d desire.

This is not something I am going to stand for though, I need each and every customer to understand that I too agree this is unacceptable, and I am working on a migration plan to another datacenter as the track record in recent months is not one I feel reflects the quality I strive to provide with FlockHosting – so I am working with another company I’ve done business with over the years to deploy new hardware and once things are back online schedule a migration to new servers.

I know some have told me “It’s okay Chuck, this happens!” and yes, it does. But I strive to be a good stewart of what God’s given me, and that’s all of you as customers – I desire for the most uptime, fewest problems, and the best deal for your dollar with services here! I am very saddened that it’s hit Church Hosting & Praise which is a handful of clients who’ve been with FlockHosting for over 10 years, so I appreciate all of your patience, but I want to make sure everyone is stable, safe, and secure – and I can’t turn a blind eye to the recent events which included a server belonging to a customer too having a multi-hour outage.

So once this calms down today (I’ll post some updates to the end of this post with more info as it comes along regarding Praise) – I’ll begin making the plan, and by the mid-month FlockUpdate I’ll have a time-table and what to expect. So please be on the look out for that – I’ll email blast & usual social networks about it too.

Again my deepest apologies, I hope things are completely stable here shortly – actually just had remote monitoring text that things are “UP” so I’ll begin reviewing and touching base with the L3 tech support team and see where we stand! I’ll post updates in this post shortly!

Update (1:08PM): Services are once again responding and looking over all services – data appears to be possibly as old as the 3rd (4 days old) trying to get confirmation on this as I continue to verify services. As soon as I have more I’ll update this post with additional information.

Update (1:29PM): And were down again 🙁 I’m very sorry for this extended outage everyone, hoping to get things squared away ASAP talking to the DC to see what happened and why this KEEPS happening. Prayers are appreciated!

Update (2:07PM): Still awaiting news from the datacenter, they are working on the problem but no updates but I see things in motion, I’m keeping an eye on things and wish I had more to report, but I wanted to let you guys know things are still being worked on!

Update (2:29PM): Things are back online again, still waiting for final word from DC guys about where we stand – thanks again to everyone for your patience in all of this, I’ll be keeping an eye on things!

Update (6:30PM – Post Nap): Just got word from the datacenter on the date of the data used to restore the system, they used an image from “March 04, 2013 00:33” – other images showed damage – so you may have some files added since the 4th missing, and I apologize for that, but we have a fully working system, no user reported issues, general audit by me shows things working A-OK. If you notice anything out of the ordinary, please open a trouble ticket ASAP!

Update (8:05PM): Got a report that things were funky, and appears they were indeed – server appears to of run out of memory it seems (not confirmed but familiar feel) but rebooting the server, seeing if we can’t get things stable – if these persist, going to pull an all nighter and customers will be relocated to Faith, Solomon, and Shepherd over the course of the evening as I can migrate your data off – I do apologize again for these issues – hopefully we can wrap them up tonight one way or another.

Update (10:26PM): Sorry for the slow reply, waiting literally for a reply – datacenter reports there is again an issue with the storage system as the original problem that started this all 🙁 It is being “worked on” – I wish I had better news, but I have also been using this time to talk to another company about some options – thankfully the contact there is easier to get ahold of even after 10PM, so not wasting time but working on solutions to get things working smoothly for you all again. I hope to have some good news shortly! Hang in there all prayers are appreciated, going to be a long night!

Update (10:56PM): Latest update, the storage system is being replaced and data migrated, DC teams are pushing us to the front of the pack, so we’ll get migrated to the new system, and hopefully if that remains stable I’ll let things roll as-is for the evening -but- I will still continue to create a plan for migration – and be in touch with each of you regarding the move – but I do want you guys to be stable for a while if at all possible before shifting accounts all over the place. More updates to follow, thanks again to everyone’s patience and prayers!

Update (11:56PM): We got to the front of the line, still meant all our data being migrated to a new storage system, and things feel a lot more stable, but I’m not going to bed anytime soon, seeing how stable things are and monitoring for an hour or two, till I’m sure I’m not going to bed only to find out 5 minutes after I turned in things went sideways, and getting a copy of data moved off-site for ease of access if things get bad again. So in the mean time feel free to check out things too on your side – if you see anything odd, either add to a ticket you already have open or email me what’s up. Thanks You All Sooooo much for your patience and kindness thru all of this!

Update (1:15AM): Into the evening I go! I am currently packaging all accounts in cPanel’s format for migrating accounts between servers, load is low on the server currently but this process can slow things down some, but its in motion now, data will be off-network just in case anything funky happens again and I need to relocate accounts to another DC – hopefully this early hour won’t effect anyone negatively – so far it’s been stable and speedy, but I want to get this set moved over just in case I need to light it up elsewhere! Will be adjusting DNS too so if needed changes are quick! Lots to do, and times like this I consider giving coffee another try 🙂 More updates to follow!

Update (2:35AM): Only a half dozen more accounts to package and get moved over, moving along quite quickly which is nice in the realm of not being up the entire night, but too let’s me stress the current configuration and ensure its going to be ok! God is Good! I’m even multi-tasking and picking/practicing songs to play on Sunday 🙂 So making good use of the time as I monitor things! More updates soon!

Update (3:05AM): All accounts packaged and re-located for safe keeping, server has been quite stable – Currently 3 hours and 30 minutes of uptime! So going to leave things as-is for the evening, and I’ll draft up a official maintenance night – I don’t want to rush anything and too make there be additional downtime as sites propagate to another server – so I’ll continue to make the needed adjustments to get things ready to roll, I hope to have a plan up by Monday and we’ll make it happen! But again I want to say thank you to everyone for their patience and grace! It was not an easy day overall, and I’m sorry that the outage lasted so long! Praying for stability, but if needs be I’ll be awake and make the move happen sooner if needed.

Your continued prayers are appreciated! I am going to wrap up some final checks and head to bed and try to get a solid 8 hours if possible, but I’d settle for 6! 🙂

Koinonia Migration Update

So last night around 2am the migration took place – the outage lasted 30 minutes according to monitoring – but all things are up and running, things are a tad snappier – and looking good!

Still going to monitor things and see if a server migration is needed, but as it stands things are looking very good! God’s been sending his IT Angels to help and just glad everything has migrated smoothly for now, but I do want to ensure everyone’s services are safe, secure, and stable – so I’ll continue monitoring, but for now – things are looking much much better!

Thank you for your continued prayers & patience in this process! Much appreciated! I’m going to go sneak in a nap 🙂

Koinonia – Migration

I received an email from the datacenter regarding the storage array we utilize, they are migrating it to a newer unit which should help further stabilize services and hopefully improve performance while were at it! But the catch 22 is, since we simply lease a large chunk of storage, they can’t give us a ETA on when this will happen, only a time frame for how long it will take to migrate data.

Starting as of right now thru Friday there will be a blip on the server for 15-30 minutes, hopefully much less if all goes well, but I would say please be prepared for up to 1 hour of outage just in case, but sadly again I can’t say “Tomorrow at 2pm” it may start in 20 minutes, may be starting as I type this! But the goal is to get things to happen as such:

  • Clean Server Shutdown
  • Data Migration
  • Data Verification
  • Startup Server
  • Smiles return to Churches!

I do again appreciate all churches and their patience in this set of issues, please know I do share the frustrations of email outages, website blips and downtime – so I am hoping this does continue to improve service for all of you – but in the mean time if there is anything I can do to help your church website wise, prayer requests, etc. please just let me know!

Church Hosting (Koinonia) Update

So over the last 5 days Koinonia has had a rough patch, some issues with the storage array created a few more problems than just some downtime – email and a few other sub-systems were effected by some data corruption which wasn’t at first apparent as some a large portion of folks utilize Google Apps and there is not a huge amount of use on subsystem.

However a few folks reported issues and I began digging in and found some errors – most of the problems are fixed now, but I am still monitoring for 2 more days very closely to ensure we are 100% back up to par, as I have always said Church Hosting just because it is free doesn’t mean it gets less support – to that end a plan is in place.

If I notice anything in the next two days (or at all this week for that matter) I’ll be deploying a fresh server – 100% from the ground up ready to go, and I’ll begin migrating customers to the new server – all of your data, email accounts, configuration, etc. will all be backed up, moved, and DNS adjusted to a new Koinonia, a fresh install ready to get going – the process should take roughly 1 hour as I’ll be making adjustments to the DNS refresh rates so that it will roll over quicker and once things are sorted it’ll go back to standard.

But if this happens I will be sending out a email blast to all churches on Koinonia with a time frame of when all this will happen, my goal will be a 24 hour notice – granted if something crops up which is dangerous or puts data at further risk with delay, I’ll be escalating this plan – but as it is, things are much much more stable and running a lot smoother – so hopefully we’ll be good to go, and I’ll schedule a migration none the less at a later date and time with more advanced warning.

I do apologize for the outages and hiccups with email – I am working on making sure everything is running smoothly and that plans are in place to ensure things are working at 100% and your sites are functioning properly but your prayers for this matter are appreciated, as I do like to get some sleep, and when things aren’t working I don’t tend to sleep till its functional again, thus why at nearly 11pm things are running smoother and I can get a few hours in again before I get back to monitoring and fine tuning things again.

Stay tuned, more blog posts to follow!