Pingdom Home

US + international: +1-212-796-6890

SE + international: +46-21-480-0920

Business hours 3 am-11:30 am EST (Mon-Fri).

Do you know if your website is up right now? We do! LEARN MORE

Is Digg digging its own grave?

Kill us hereDigg has a problem. That problem is that the more users Digg gets, an increasing amount of the stories that reach the Digg front page are bound to be unavailable, brought down by the storm of visitors from Digg.

If Digg keeps growing it will automatically kill almost any site that reaches the front page.

Digg can’t keep using the same model they are now. It simply doesn’t scale. It will reach a point where it has so many users that the amount of traffic that comes from being on the front page of Digg is more than any normal site can handle. (You could argue that this is already happening.) As it grows, Digg will be knocking out larger and larger sites.

We (i.e. Pingdom) monitor the uptime and response time of a lot of websites, and there are numerous occasions where we have seen pages become either very slow or completely unavailable in connection with hitting the front page of Digg. We perform more than 7 million tests per day, so we have a significant amount of data to look at. The problem tends to be the initial spike of traffic when a news item first hits the front page.

A visual example

Digg can't keep growing

As you can see we divided sites into three main categories:

    1. Sites on shared hosting accounts. Can handle a normal amount of traffic. The majority of sites on the internet are hosted this way, and a lot of these will have serious problems (i.e. crash) if they end up on the Digg front page.
    2. Sites on a dedicated server. Can handle a lot of traffic, but can, and often do, run into trouble during the initial peak of traffic from the Digg front page.
    3. Sites on multi-server setups. Handle anything you throw at them, but are basically only used by media portals, large news sites (think CNN.com) and large enterprises. Digg isn’t a problem here.

 

Sites with multi-server setups only make up a tiny part of the content available on the internet. Isn’t one of the main points of Digg to show niche content and find news before the big media finds it? This content is often driven by people who earn no or very little money and host their sites on modest setups.

Mirroring content isn’t the answer (legally speaking)

The so-called Digg effect is so common that it has given rise to a separate mirror service (Duggmirror) that makes copies of downed sites. The legality of this practice is highly dubious from a copyright standpoint, and isn’t run by Digg. There is even a statement by Digg-founder Kevin Rose from 2006 regarding the caching of content, which is quite interesting in this context:

“Digg should never take away traffic or cache content, should always push to site creator.”

Summary

The conclusion here has to be that if Digg wants to keep growing, and we assume they want to considering all the attention they have received from VCs, they need to somehow change their model. If they just keep going the following will happen:

    1. Most of the websites on the front page of Digg will be unavailable, which means that…
    2. The quality of Digg will suffer and it will be much less useful.
    3. Some sites may not even WANT to be on Digg. It may simply be more trouble than its worth.

 

Perhaps the Digg crew is working on some solution to this problem. We don’t know. What do you think?

A small note on Digg’s users and visitor numbers

According to an article by John Graham-Cumming, Digg currently has about 2.7 million registered users, which is almost three times as much compared to a year ago. But of course, the majority of visitors to Digg will not be registered users, but simply people there to find interesting news. According to data from comScore, the Digg website had about 12 million unique visitors worldwide in December.

As an example, last time we had a post on the front page of Digg, the dedicated server for this blog couldn’t keep up with the initial burst of visitors from Digg. This is kind of ironic considering we’re an uptime monitoring company… :)



6 comments
Gabriele Pierfederici
Gabriele Pierfederici

As new technologies evolve and prices drop, highly scalable web hosting facilities should become more and more affordable and widespread. Today most shared hosting plan are based on the concept of overselling single server's hard disk space, CPUs and network bandwidth. That has obvious limits and doesn't scale. However, more flexible solutions are spreading rapidly. Some example are: - cluster based shared hosting (like MT GS, not as common as single server hosting but already quite cheap), - complex application platforms like Mosso and Joyent Accelerator (not so cheap yet, but very scalable) - grid based on demand infrastructures and utility computing, like Amazon Web Services, Sun's Grid, 3Tera (pay just for what you use, almost limitless). Definitely, being hit by a lot of traffic shouldn't be anything to worry about. From my perspective, Digg's steady growth should be taken seriously into consideration by every hosting company that should adequate rapidly if they don't want to loose customers.

Andrew Baron
Andrew Baron

I have noticed the opposite. It seems like Digg stories do not lead to as much traffic as they once did because all of the people on Digg are spread out more around the site. Thus, it's not like everyone goes there and clicks on the top stories. They are saturated throughout all of the stories. The more people, the more stories and thus the more saturation. Rocketboom does not get a lot of traffic from Digg but for any story with x number of Diggs, we used to get a lot more for the same number. We have seen a lot of traffic from various sites and over the last few months there is no site that has come close to throwing us as much dense, sudden traffic as Stumble Upon. I dont know whats going on over there, but we are waking up big time to that site.

Royal Pingdom
Royal Pingdom

Sal Cangeloso > Yes, there are many things to take into consideration but we still think the post tells what is happening. One of the main things with Digg is that the people choose the content and because of this it can have news and interesting information before any of the big news sites pick it up. And this shouldn't be lost or Digg will be a aggregator for all the big news sites. Hosting is indeed changing and getting better, but like you said this is happening slowly. Also, did you see the note about Diggs growth in the end of the post? Growing their userbase almost three times during a year is pretty good :) But of course it has to stop sometime. Andrew > Yes, we know about Slashdot, which was the original source of this phenomenon. But it seems Slashdot isn't growing any more and it has lost some of it users to Digg. lestranger > You are right, there are several other sites out there besides Digg. But Digg is the largest one and the one with the biggest issues. Think of getting on all of these sites at once though.

lestranger
lestranger

I would argue that StumbleUpon, as Firefox spreads, will become more and more difficult for web hosts to handle. It is not necessary to read reviews of sites linked, only one click and the database directs you to a random site based upon your preferences. While it is very friendly for the end user (I found this site through StumbleUpon), I've found that an increasing amount of sites Stumbled return "CPU quota exceeded" or "Bandwith quota exceeded" announcements, which is alarming. It's a great social bookmarking service, and the idea is great, but it does take a great toll on web hosts, as there is no guarantee the content is needed or wanted by the end user. I would say it is not uncommon to simply click the StumbleUpon button after another "unsuccessful" Stumble without voting it out (I'm afraid I have done it too often too), which leads to increased unwanted traffic to sites that necessarily wouldn't get as much load.

Andrew
Andrew

This effect already exists with Slashdot (the term is called "Slashdotted", quite original, really). Thing is, Slashdot hasn't killed itself yet. It's unlikely to happen with Digg, I suspect.

Sal Cangeloso
Sal Cangeloso

That's an interesting analysis, but I think some things have to be taken into consideration. 1) As digg grows and becomes more commercialized it tend to focus links more on large, popular sites that can handle the hosting. For example, sites like Engadget and Techcrunch get lots of Diggs, so they expect the load and build hosting systems that can handle it. 2) Traditional shared hosting is being (slowly) replaced by grid and slice systems. These are going to offer a lot more power to the individual account when necessary. Media Temple's Grid is only $20 a month and can handle a Digg. It has it's own issues, but it can handle a heavy load. 3) If a site gets on Digg and it goes down, it gets buried and pulled from the front page. So the impact of the issue is limited. 4) From what I understand Digg's readership is already beginning to plateau. So while it may still grow, we are not going to see continued explosive growth. So your trend line of continued arithmetic growth doesn't make sense. Interesting post. Good food for thought...