When your website gets indexed by Google it will be added to Google’s regular crawling. This means that Google will load pages from your site at a regular interval. If your website is popular (high traffic and many inbound links) it will be crawled more often than if it is seen as a less important site.

Google tries to adjust the crawling to hours to when it’s easier to access your site. If your site is seen as unreliable by Google you can be punished by losing positions in the search engine result pages. “Unreliable” can mean that your website is slow or that Google has had several issues with indexing your website.

If your site is completely down or responds with a server error page, Google will come back later and try again. After several attempts they will however give up and remove you from their index. In other words, you lose all the traffic you get from searches in Google.

Above: Graph showing how organic search traffic from Google can be affected if your site is dropped from the search index due to downtime.

Matt Cutts, a Google employee, and Vanessa Fox, a former Google employee, have said the following regarding Google indexing and downtime:

“Googlebot will try a few times before the pages drop from the index.” – Vanessa Fox

“As for how long it takes for a page to get back in once the site is back up, that really depends on a number of factors, such as how often the site is crawled in general.” – Vanessa Fox

“Sometimes temporarily down pages turn into truly-gone-forever pages, so we have to drop those pages at some point. But it’s also true that we go back and revisit those pages pretty often and try to recrawl them in case the site comes back up.” – Matt Cutts

It seems that depending on the size of your website, luck (!) and the type of issues you are having, anything can happen between nothing and being completely removed from Google’s index.

From all the discussions about this topic you can only come to one conclusion: uptime is very important if you don’t want to rely on pure luck.

What can I do?

1. Monitor your uptime and quickly take action if your site goes down. You can use any uptime monitoring service that can send you alerts, for example Pingdom.
2. Monitor and test your response time at regular intervals. In addition to monitoring services, you can use our free Pingdom Tools to test your site’s total load time.
3. If you need to take down your site, use the right server error codes. Don’t just put up a regular page explaining the maintenance since this might get indexed instead of your real site.
4. Use Google Webmaster Tools and check that no pages are listed in the crawl errors section.
5. Beware of expiring domains, hosts that show informational pages if you exceed your bandwidth or any other occurrence that will alter your website in an unwanted way.

In 2010, there were just over 1 million secure Internet websites worldwide. Almost half of those, or 446,992 to be exact, were located in the United States.

But in which country can we find the most secure websites in relation to population? The answer may surprise you.

Read more

The New England Patriots held what seemed to be a commanding lead (17-15) with five minutes left of Super Bowl XLVI last night. But the New York Giants came back and managed to win with 21-17.

As exciting as the game sounds, we missed the whole thing, instead spending our time watching the Superbowl.com website.

It turned out to be a rather dull thing to do because the site held up well and there was no downtime at all. The response time also didn’t give away anything significant in terms of online Super Bowl traffic.

Read more

As Super Bowl 46 is approaching, fans will flock to the Lucas Oil Stadium in Indianapolis, Indiana, and to TV sets around the world to follow the New York Giants battle it out with the New England Patriots.

Kickoff is scheduled for 6:30EST on Sunday, February 5, and we’re already monitoring Superbowl.com to see how the site will handle the event.

What team will win Super Bowl 46? How will the site cope? We can only wait to find out.

Read more

Every Friday we bring you a collection of links to places on the web that we find particularly newsworthy, interesting, entertaining, and topical. We try to focus on some particular area or topic each week, but in general we will cover Internet, web development, networking, performance, and other geeky topics.h

This week we bring you a collection of articles focusing on cloud, with a few other topics thrown in to boot.

Read more

Out of the 59 US-based e-commerce sites we monitored during the holiday season last year 28 scored a perfect 100% uptime for December.

Whether this helped spur on the booming sales in the US, we don’t know, but retail e-commerce spending in the US reached $37.2 billion for the November to December 2011 period. That was an increase of 15% from the same period in 2010.

We decided to dig into the numbers for these e-commerce sites to see how well they did in terms of uptime and performance. After massaging the data coming from our Pingdom probes, it turns out that the sites overall performed well during December 2011 in terms of uptime, but response time was an issue for several sites.

Read more