Here’s what I think happened – there was a bug in a forced ugprade of Skype 5 for Windows that brought down nearly all the supernodes.  Here’s how I figured it out…
Yesterday our Director of Conversations, Dan York, wrote a great article on the fundamentals of how Skype works and why Skype was down for nearly two days.  You can read it at:

http://www.disruptivetelephony.com/2010/12/understanding-todays-skype-outage-explaining-supernodes.html

Dan does a great job describing how Skype works, but doesn’t get into why it may have crashed in the first place.

I may have some insight.

Voxeo uses Skype – specifically Skype instant messaging – extensively within our business.  Skype is an amazingly good product.  Voxeo would not work nearly as well without it.  In fact we haven’t worked nearly as well the last two days.  Fortunately they were two slow days, with many of our employees out on holiday already.  After yesterday we quickly got a backup Jabber/XMPP solution in place.  That was fun but frustrating.  A story for another time.

As Voxeo’s CEO I travel fairly often.  About two years ago I also started using Skype video calls to talk with my kids at home while I was traveling. The video quality was often excellent. However at times the frame rate of the video would get quite low.  So I started looking into why that was.

I started by turning on the Skype setting to show technical information during calls.  I discovered was that the video frame rate changed significantly depending on the type of NAT (Network Address Translation) in use at the various hotels and airport WiFi networks I was on while traveling.  At times Skype would proxy all my video traffic from my house, through another computer on the Internet, and then to my well-traveled notebook computer.  That’s when video quality would go down.  It made sense – as those of us who work in the VoIP industry know, the worst-case for streaming audio/video over IP is when both sides of the connection have restrictive NAT configurations in place.

I couldn’t control the NAT that occurred where I was traveling – but I knew I could control the NAT at my house.  So I spent a small amount of time analyzing how the Skype client was working at my house and how it interacted with my Firewall.  I changed some Firewall settings to use one of my public IP addresses to specifically forward Skype traffic nicely… and it worked great.  On my next trip I had top-notch video quality wherever I went, because Skype no longer had to proxy my video calls through any other servers.

Guess what else happened?  My home computer seems to have become a Skype supernode.  Hundreds if not thousands of IP interactions were flowing in and out of this particular Skype client.  I couldn’t read any of them – all of the Skype protocol, messages, audio and video are encrypted.  While I couldn’t tell the specifics, It was obvious my local Skype was quite busy.

The combination of great bandwidth plus a Skype client and firewall configured to work great together for video and audio seem to be the exact attributes Skype looks for in Supernodes.   At first I thought it was amusing.  Then I noticed it was using over half my bandwidth.  So I put a rule in place to restrict Skype to about 20% of my bandwidth at the most and forgot about it for over a year.

Fast forward to yesterday…

I mostly use Mac’s, in fact Voxeo is a 100% Mac environment, but I do have one Windows Machine at my house.  I mentioned above how Voxeo really uses Skype IM as our primary form of communication, especially considering we have people on the East Coast and West Coast of the US, in the EU, and in China.  So I got up early yesterday to check on the latest Voxeo messages.

The Skype client was crashed.  I thought that’s unusual, and I restarted it.

Within five minutes it crashed again.  Strange, I started it again.

And it crashed… again.  What the heck was going on?

Then I remember a post I saw from Chris Pirillo on my Facebook account:

“Skype 5 Forcing Updates”

http://chris.pirillo.com/skype-5-forcing-updates/

I didn’t pay much attention at the time, but sure enough I checked my Skype version – Skype had done a forced upgrade to the latest version 5 client.  It never asked me, never advise me, just did a forced upgrade to the newest Skype version.  (I have a theory about why Skype took this bold step, which I’ll also share another time).

I was real annoyed for a few minutes.  As I mentioned Voxeo really depends on Skype to get our work done.  I depended on Skype and they forced a buggy upgrade on me.   Then I thought about it again.

Wait a minute.  Skype depends on it’s supernodes even more than Voxeo depends on Skype.  Supernodes like my machine… which keeps crashing.

I literally made an astonished out loud curse at that moment.  And quickly pulled up Twitter search.

Sure enough their were hundreds of messages about Skype being down.

Now I can’t say for certain this is what happened, but I really think it’s the most likely possibility:

Skype pushed a forced upgrade of all Windows clients out in the last few weeks.  By yesterday *all* the Windows Skype supernodes – and given Windows market share, that probably means nearly all of the Supernodes – were on the same version of the client.  It’s probably the first and only time the vast majority of the Supernodes were running the exact same Skype client software.

And that software has a bad, bad bug in it.

Originally from Voxeo Blogs<