Read and/or listen to Niall Kennedy's exclusive interview with SixApart's Anil Dash about TypePad's recent outage and what it means to the future of business blog hosting - and to SixApart. (I'll listen when I get a chance. Still writing... ) Niall works for Technorati, BTW, which is down the street from 6A in San Francisco.
Also, read Steve Rubel's provocative "this too shall pass" post about this latest TypePad episode, yet another in a string of frustrating experiences with the service over the past few months. Steve says The Day is Darkest Before Dawn and uses eBay's early scaling problems as a parallel.
For the record, I tend to agree. Like Steve, I remain a SixApart fan and hope they figure out how to make TypePad bullet-proof.
And yes I was one of those leading the cry for an explanation of the problems when I wrote Listen up SixApart: some of your TypePad customers may switch.
I'm not giving up on TP yet. But for heaven's sakes guys, throw a ton of that venture money you got (how much was it??) at this problem and get it fixed.
Related:
Silence of the Blogs (Forbes' Technology)
Thank you for your patience Debbie, I surely understand those who have lost it.
We are investing brains, money and time against this problem and it has generally paid off. Performance in November was not perfect but was better than October, December was even better than November until this down time. Unfortunately each time you try to make a system more reliable you also create risk. In this case the risk was assumed to be small, we assumed wrong.
Over the last few weeks we have successfully moved the entire service from San Jose to San Francisco (there are still a few network changes to make to be totally moved). We are on new machines including an expensive, reliable and fast file server. We moved terabytes of data, put it on this new system and moved tons of servers without a service failure. Thursday we were completing the instillation of a redundant front end to the file server and this installation required a reboot. For some yet to be determined reason the reboot corrupted the directory of the file system and led to the longest application down time in TypePad’s two year service history.
Apologizing at this point sounds so hollow that I am almost afraid that people will take it as an insult. But we are sorry, in fact I’m mortified. We don’t really have the right to ask for our customers continued trust, but that is what I must do and honestly believe we can earn it.
We will complete restoring data this weekend. We should finish the remaining details of the move by the first half of January. And I truly believe that risk, while already low, will be greatly reduced by then. No big new changes in hardware to make, no data to move, no networks to reconfigure, that is all behind us and we can focus on continuing to smoothly growing capacity to keep up with demand. We will keep making the system better, but what we have today is many times more bullet proof than what we had three months ago.
All this is to say that I am optimistic that Steve is right, this is the darkness before the dawn and we will be providing you a rock solid stable service for all of next year and well on into the future.
Posted by: Barak Berkowitz | December 17, 2005 at 08:55 PM