Newsgator is down for “maintenance.” I first noticed at around 7 am, it’s now 9:10 am EST and it’s still down. Bummer.

There may be maintenance going on, but there’s no way I’ll buy that it was scheduled or planned. Something broke, they’re fixing it. I refuse to believe that any online service would schedule maintenance that would pull the service down at 9 am on a Tuesday morning. It’s like shutting down the highway to fix potholes at 4 pm on the Wednesday before Thanksgiving. You just don’t do it unless it’s an emergency.

Update: 10:30 am – still down. Nothing on Greg Reinacker’s blog. The #1 search on Technorati is, you guessed it, “Newsgator”. I’ve been a NG subscriber for a few years now, and this is the first time that i can recall that everything is offline. Sure, I’ve had trouble getting to my feeds now and again, but then I’ve always been able to get to the support forums or the main page of the site. Now, no dice. If it’s anywhere in the Newsgator domain it’s unreachable. Must be a pretty bad fire to put out. Hope the hamsters are working overtime.

I typically start my day with a quick scan of my feeds to get an overview of what’s happening in the world…going right to work without it almost feels like I skipped my morning cup o’Joe.

Ah, found someone who got an email from NG Enterprise support which estimated that NG will be back online mid/late morning. What timezone?  

Hey NG folks…at this point, you should drop the “currently down for maintenance” line and truthfully state that you’re having a problem and you’re working on it, with an estimate of when things should be back.

Oh well…back to work…

Update noon EST: Still down, but the message has been improved:

NewsGator Online is currently down. We expect to be back online today by 12:00pm MST.

Better. NG can’t forget that their customers are for the most part, tech savvy. The "we’re down for maintainance" line doesn’t fly…and after a few hours it becomes an insult. Much better to go with honesty. Get the database back online and then blog about exactly what went wrong, what steps were taken to fix the problem and most important: what actions are now being taken to learn from the mistake and make sure it doesn’t happen in the same way again. Weeks from now people will forgive Newsgator for having a bad morning. All servers crash. All hardware and software fail eventually. Anyone expecting 100% uptime needs to come visit planet Earth sometime because it just doesn’t happen. What Newsgator will be judged on is how they responded to the crisis. 

Update 12:30 pm EST: Newsgator is back online. 90 minutes ahead of their prediction. I have no idea how up-to-date the feeds are..I’m not seeing that they have this entry, so it may take some time to catch up.

Now let the ‘splainin’ begin.  

Update 4:30 pm: …and ‘splainin’ they did. Greg Reinacker posted a thorough, detailed explanation of what went wrong, including steps that are being taken to make sure an outtage like this doesn’t happen again:

Moving forward, we’re doing a number of things. Specifically, we still need to get to the root cause of this particular problem, but in general, we’re working a lot on our back-end systems to build additional redundancy, scalability, performance, and resiliency.  We’re going to partition data in different ways, so a failure like this will only affect a portion of the user base, a portion of the content, or a portion of the functionality.  Our back-end cache servers will expand to be able to serve data while underlying data store is offline.