More on the January 30-31 #twittout

Perhaps you've seen my earlier post that detailed how last night's scheduled Twitter outage went into some unscheduled territory, which ended up influencing the way that I do things.

I thought I'd collect some other perspectives on what happened last night...and this morning.

Let's start with Dave Winer:

Over the last 24 hours Twitter has been down as much as it's been up. As always this gives us a reason to think about what the world would be like without Twitter and then those of us who are engineers or would-be engineers, start thinking about ways to fix the problem, whatever it is....

I'd like to really understand what's going on behind the scenes at Twitter, Inc. They say they're confident the new infrastructure will hold up better, I'd like to understand why. Can we have a meeting, with a few people from the tech community who actively use Twitter and a few people from the company, to be briefed on what's going on....

PS: I was going to post a link to this on Twitter, but arrrrgh!

I'll give Twitter two voices in this matter. Let's start with Ev, on the official Twitter blog:

I just left the office, at 6am, with most of our engineering and technical operations team. That's only a handful of people, but we were all there all night.

You may have noticed we had an outage last night/stretching into this morning. This was a planned maintenance project, not a surprise happening for us. But we did go far beyond our planned time window, which sucks. What sucks more is the series of outages (planned and not) we've had lately. We know this makes Twitter frustrating to use, and we want you to know, we hear (and share) your frustration and are working really, really hard to fix it.

The good news is we finished a major infrastructure project tonight, which we've been working on for months and that we think is going to help a lot. I'm sure there will be some glitches in the coming weeks, but we've laid some important foundation that's going to help us build toward the future.

Thank you for your support and patience.

Update (8:30a PT): We're still experiencing patches of slowness service-wide. Some of our systems are not cleanly restarting. We're watching closely, bringing things back up manually, and fixing our automation tools accordingly!

Now let's turn to Biz Stone in his personal blog, in response to Dave Winer's post:

Twitter service is sometimes interrupted because we're working on it. We're concentrating our efforts on creating a reliable, global communication utility that can be depended on 99.99% of the time. Our team made a significant achievement last night but but it's important to note that our goal will not be achieved in just one night.

Reliability will be achieved by sustained effort and strategic decisions like hiring Lee and building out our technical team which is currently just four brilliant programmers and two awesome operations folks. The all-nighter Twitter pulled last night was part of a major project to get us closer to our goal.

We'll be able to share more of the specifics of our latest infrastructure project when we've successfully completed it and can write about it with some perspective.

We don't get into deep, technical specifics on our main blog mostly because I'm the one who usually updates it and I'm not really that deep or technical. For more technical discussions we hope people will visit our developers group and developer blog.

Which raises the question - is Dave Winer's request reasonable?

I'll give you the somewhat anonymized example from the bowels of MegaCorp (my otherwise unidentified employer). Several years ago, MegaCorp did not perform well on a certain external test, and our existing customers were naturally concerned and wondering if they would need to find a new vendor. At a previously scheduled meeting with our customers, we put one of our research and development guys on the stage (i.e. a "shirt" instead of a "suit") who acknowledged that we did not perform well, and assured the users that we would take steps to perform better in the future. However, we did not give specifics about what we planned to do, and I think it's fair to say that we didn't do this for two reasons: (1) we didn't want our competitors to know our plans, and (2) since this was a research exercise, we couldn't precisely say what we WERE going to do. The story has a happy ending, because we did do some stuff and are now kicking butt. (Rah rah.)

Perhaps Twitter has similar reasons for not opening the kimono, worried about stuff leaking to its competitors, and worried about the reaction if they were to tell their entire customer base "We have an idea, but we don't know whether it will work as we envision it. We may have to change." While a certain segment of Twitter users would understand this, some may react negatively - "What? You don't know if something is gonna work? Why don't you talk to my friend in the garage who wrote a Visual Basic program that manages our entire ham radio network!"

But there's another part to the MegaCorp story that I haven't discussed yet. While we do not provide a ton of confidential information to our entire user base, we do have a small group of users under non-disclosure who are privy to confidential information, including future release plans and internal technical accomplishments.

Does Twitter have a customer advisory board that is under non-disclosure and can provide an external evaluation of Twitter's plans, both for scalability and for monetization?

This is especially critical (or, you could say, obvious) if Twitter has such a small staff. In that case, outside perspective is necessary; don't think that you can do it all yourself.

[mrontemp business] | [mrontemp politics] | [mrontemp technology] | [mrontemp del.icio.us tags]

Sphere: Related Content