US + international: +1-212-796-6890
SE + international: +46-21-480-0920
Business hours 3 am-11:30 am EST (Mon-Fri).
It has probably escaped no one that Amazon had several days of serious issues with its cloud hosting service last week, which took a large number of sites either fully or partly offline, including sites like Reddit, Foursquare and Quora.
Big data has become one the new buzzwords on the Internet. It refers to the massive amounts of data that many modern web services deal with. This post will list some of the more useful software available to web developers for working with big data.
Amazon recently published some numbers showing how Amazon S3 (Amazon’s “cloud-based” storage service) has grown over the years. They even included a chart, which you can see here below, which clearly shows how the number of objects stored in S3 has exploded.
These days if you try to find news involving the word “cloud,” you’re more likely to get an article about cloud computing than you are finding a weather report. If the amount of news referencing “cloud” is anything to go by, the media has embraced this new terminology with open arms, starting in 2008.
Google made a huge splash when it announced its plans for the Chrome operating system, a web-centric OS where essentially everything is run through a web browser. One great promise of Google’s Chrome OS is the arrival of low-cost, lightweight hardware, since most of the storage and other data handling is done in the cloud. Perhaps that 100-dollar computer will finally become a reality.
But there is a problem. A rather big one. The strength of the Chrome OS, that it makes maximum use of online resources, also limits its potential adoption. To have any real use of the OS you need a decent Internet connection, and that has some significant implications we need to look at.
Last week Google finally unveiled their much-talked-about Chrome OS, and subsequently worked the tech community into a frenzy. The operating system certainly lived up to Google’s initial promises of being browser-centric – it is basically just the Chrome web browser atop a custom Linux kernel.
Chrome OS is a momentous step towards making the fuzzy concepts of cloud computing more of a distinct reality. What follows are a few reasons why I think it matters, and how it will change the computing landscape by bringing us closer to the cloud than ever before.
The combination of high bandwidth and low-cost hard drives has created a small revolution in online storage. Web-based storage like DropBox, Mozy, Data Deposit Box, and Amazon S3 offer individual computer users and firms of various sizes a fast, convenient, and flexible way to store their data.
But before you or your company jump in credit card first, you should consider the following five factors before you choose an online storage solution.
How many servers does Google have? Nobody outside Google knows exactly how many servers the company has, but there have been a number of estimates through the years. One of the most quoted ones is from 2006, when it was estimated that Google had approximately 450,000 servers. And that was three years ago.
Another estimate showed up in 2007, this time from the analyst firm Gartner, estimating the number of Google servers to one million.
Considering that both these estimates are from a long time ago and Google has grown its data centers significantly since then, it’s not unreasonable to assume that Google today has at least one million servers worldwide.
In November, Google wrote in their official blog that they had done an experiment where they had sorted 1 PB (1,000 TB) of data with MapReduce. The information about the sorting itself was impressive, but one thing that stuck in our minds was the following (emphasis added by us):
An interesting question came up while running experiments at such a scale: Where do you put 1PB of sorted data? We were writing it to 48,000 hard drives (we did not use the full capacity of these disks, though), and every time we ran our sort, at least one of our disks managed to break (this is not surprising at all given the duration of the test, the number of disks involved, and the expected lifetime of hard disks).
Each of these sorting runs that Google did lasted six hours. So that would mean that hard drives would be breaking at least 4 times a day for every 48,000 hard drives that a data center is using.
Gmail could be unavailable for more than 21 hours in a day, and Google could still tell you that according to their SLA, the service has had 100% uptime.
It sounds impossible, but it’s a direct consequence of how Google has written its SLA for Google Apps (which includes Gmail, Google Docs, Google Calendar and more). We will explain this in detail further down, but let’s first look at what the SLA actually says.
