Pingdom Home

US + international: +1-212-796-6890

SE + international: +46-21-480-0920

Business hours 3 am-11:30 am EST (Mon-Fri).

Do you know if your website is up right now? We do! LEARN MORE

Peeking behind the scenes of the world’s largest sites

Nuts and boltsDo you want to know more about how big websites like Twitter, Facebook, Hotmail and others handle the technical challenges of dealing with massive amounts of visitors?

Well, you’re in luck, because many of those sites and services have engineering and/or developer blogs that share plenty of information about the challenges they have to deal with and the tools they use. This is an insider’s view that you usually can’t get anywhere else, giving us a unique view of what’s going on behind the scenes of some of the world’s largest web services.

Yes, you’ll have to be a tech geek to find them interesting, so consider yourself warned. ;) But we here at Pingdom are geeks, and we are really into web tech so we love these kinds of blogs. They are blogs that web developers and sysadmins can learn from and be inspired by.

Let’s get right to it.

Twitter’s engineering blog

Twitter’s engineering blog gives insight into what makes Twitter tick and what technical solutions they are using to solve their scalability challenges.


A few recent posts:

  • Hadoop at Twitter
  • Introducing Gizzard, a framework for creating distributed datastores
  • The Anatomy of a Whale

Facebook’s engineering notes

Want to learn more about the technical ins and outs of the world’s largest social network? Look no further than Facebook’s engineering page.


A few recent posts:

  • Site Reliability Engineering at Facebook
  • Intro to Facebook Search
  • Working with Zuck

Digg’s technology blog

Whether you use Digg or not this is interesting. Self-described as “a peek under the Digg hood for API developers, partners and technology geeks.”


A few recent posts:

  • Saying Yes to NoSQL; Going Steady with Cassandra
  • Testing at Digg
  • Database Capabilities in a High-Volume Environment

Inside Windows Live

A blog by the engineering team behind Hotmail, Messenger and Windows Live. It has a large variety of articles with behind-the-scenes information and data analysis.


A few recent posts:

  • Hotmail tips the scales
  • Are you a piler, filer, or purger?
  • Keeping the Messenger service running—on a massive scale

Flickr’s developer blog

A blog by Flickr’s development team that dives into the nuts and bolts of the photo-sharing site and working with the Flickr API.


A few recent posts:

  • Using, Abusing and Scaling MySQL at Flickr
  • A Chinese puzzle: Unicode and EXIF metadata parsing
  • Language Detection: A Witch’s Brew?

Meebo’s developer blog

Interesting developer-oriented articles about the inner workings of Meebo’s instant messaging service.


A few recent posts:

  • Performance Case Study: The Meebo Bar (Part 1 of 3)
  • Keeping your passwords safe, part 3
  • Dev tips and tricks

Amazon Web Services blog(s)

You have Amazon’s official AWS developer blog which has plenty of announcements and great advice for using Amazon EC2, S3, and so on, but the real nugget of gold is the insightful All Things Distributed, a blog written by Amazon’s CTO, Werner Vogels. So, with apologies to the official AWS blog, that’s the one we’ll list here below.


A few recent posts:

  • Choosing Consistency
  • 82 Billion Objects in Amazon S3
  • Good Advice on Keeping Your Database Simple and Fast

There’s more, much more…

What we find interesting with these blogs is that they give unique insights into the nitty-gritty technical details of running sites that are much, much larger and more demanding than your average site.

There are a bunch of other sites you might also like to check out depending on your interests. The behind-the-scenes articles may be a bit sparser on these, but they still come up with the occasional gem. Here are some to get you started.

  • Google Code blog
  • Google App Engine blog
  • Yahoo’s developer network blog
  • Signal vs. Noise, the blog of 37signals, makers of Basecamp and other web apps.
  • Wikimedia’s techblog (the people behind Wikipedia)

Then on top of these you can add all those great blogs that don’t actually belong to a company/service but still offer great articles about how the big sites are run. A great example is the High Scalability which is a blog dedicated to how sites deal with scalability and performance issues.

We’re sure we’ve missed several excellent blogs, so if you know of other services whose engineers or developers blog about their technology, let us know in the comments. We can build a great list here!


dive into information techology