General Musing

blaze your trail

Fail whale solution for Twitter #failwhale @twitter

with 2 comments

Many years ago while working for a large ISP we had occasional outages due to bugging software or being over capacity. Mail was the biggest problem, with many people – even on the dial-up network – POP-ing[1] their mail every 5 minutes. We had rolled our own mailserver and we were constantly fixing our infrastructure to give our customers the highest quality. Yet we still had the occasional outage which caused our helpdesk to be flooded with calls from people whose mail client gave a pop-up message with an error message.

To solve this we implemented 2 systems, the first being dynamic rate limiting: depending on capacity if you pop messages more than ever X minutes we will simple tell you you have no mail every X numbers of the time. Previously we would send an error message telling the user that their mail client was configured to POP mail more often than is recommended, obviously the helpdesk was less than happy about this.

Secondly we created what we called: “Lying POP3”. It simply tells the user they have no mail when we are experiencing outages. This means that in the case of outages, which might last from 5 to 30 minutes – temporary outages – we didn’t need to have a status page telling people that we were temporarily down, their experience was not that we were down. Obviously we had the advantage of that the majority of our users weren’t tech savvy and that mail wasn’t experienced as an instant technology. And in the case of real outages or upgrades, we obviously updated our users and helpdesk within this buffer or ahead of time.

So what am I trying to tell Twitter?

Experience is the key to service! You are down when I perceive you as down, you are up when I perceive you as up.

So how would I do it?

I’d do exactly the same as I did with mail for the API, dynamically rate limit and lie for a short period of time about tweets from the twips a user follows. For tweets a user sends I would have a FIFO queue which takes all the tweets send and processes them when it is again possible to process them, this would naturally have the same rate limiting as normal. For the website I would use a cache copy of 1 page of tweets, preferably with a failwhale notice at the top so people know there is an outage not to expect updates, but the experience of the people is that they can see tweets.

In the terms of the book “Authenticity” my experience of Twitter

Fake-fake: is not true to itself; is not what it says it is[2]

  1. Using POP3 to retrieve mail.
  2. Real-Fake

Written by Daniël W. Crompton (webhat)

July 2, 2010 at 5:51 pm

Posted in mail, social

Tagged with , , , ,

2 Responses

Subscribe to comments with RSS.

  1. The author is the best, I selected works will add it.

    Bond Gray

    July 12, 2010 at 3:17 am

  2. Thanks very much for your fantastic job; this is the kind of thing that keeps me on track during my day. I’ve been searching around for these posts after I heard about them from a colleague and was delighted when I found it after searching for some time. Being a blogger myself, I’m glad to see others taking initiative and contributing to the community. I wanted to show my appreciation for your work as it’s actually very encouraging, and many writers do not get the acknowledgment they deserve. I am sure I’ll be back and will refer some of my friends here.

    Hertha Akiereisen

    July 18, 2010 at 7:38 pm

Please Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: