Pingdom Home

US + international: +1-212-796-6890

SE + international: +46-21-480-0920

Business hours 3 am-11:30 am EST (Mon-Fri).

Do you know if your website is up right now? We do! LEARN MORE

The “top secret” room where 260 Internet Service Providers connect

We came across a very interesting article in Wired called A Lesson in Internet Anatomy: The World’s Densest Meet-Me Room. Over 260 ISPs, including major network providers like AT&T, Sprint and Verizon, all cross-connect in a single data center in an office building in downtown LA.

Big no no
Scissors and other sharp objects cannot be welcome here…

This has been going on for 20 years. So much for not having a single point of failure:

If this facility went down, most of California and parts of the rest of the world would not be able to connect to the internet. Tour one of the web’s largest nerve centers, hidden in an otherwise nondescript office building.

This is where 260 ISPs connect to each other.
Yes, that’s the building. This is where 260 ISPs connect to each other. One Wilshire.

Let’s hope that no one stumbles over the wires in that data center. That could cause some serious downtime.

Note: The photos are all by Xeni Jardin (the link leads to a Flickr photo set with more images from One Wilshire).



No related content found.

No Comments

“an office building in downtime LA.”

Either some clever wording or a wee slip of the keyboard!

David, a wee slip of the keyboard, I’m afraid. Fixed now.

I guess when you spend all day working with downtime-related issues, this is what happens… :)

Was wondering why all my web browsing goes through LA. Even though I live in Canada.

I have this huge urge to run rampant through that place with a chainsaw.

That’s really fascinating. However, I doubt that individuals would actually lose complete connection. I would find it surprising that so many homes would have to go to that building. More likely other connections would crash from the additional traffic.

Thanks for posting that. Now, rather than just some random terrorist organization knowing just where to put a device, any goofy moron with one also knows where to put it.

moss

> Was wondering why all my web browsing goes through LA. Even though I live in Canada.

Because Canada sucks and can’t last 3 days without US
LOL

This is the kind of information I would prefer not to find on the Internet. Whoever published this information is trying to give a clue to some crazy terrorist about a target that would disrupt free world communications?

Hi

Actually, if managed correctly this does make sense. In the UK we have a system called LINX – the London Internet Exchange. The majority of ISPs all link through LINX, the purpose being to ensure that web requests for sites hosted in the UK are only routed in the UK. This can cut out a significant number of router hops which subsequently strips latency out of the issue.

My worst “route” was for a visit to the Microsoft site hosted in Seattle. My trip left the UK and was routed across the Atlantic, in the US arounf MAryland / DC, across the US and out the otherside, across the Pacific and in to Asia where it was then routed up therough Turkey and some of the western european countries (former soviet states) then in to Germany and France – acrosss the English Channel in to Engalnd whence it left again, crossed the Atlantic,making landfall in Baltimore, across the US and finally making its way to the destination.

A US equivalent of LINX could have “trapped” the request and ensured that the traffic stayed routed in the US.

I am so sick and tired of the worn out terrorism article. Hugo Hernandez, you sir, are an asshole.

There are bundles of wires in every building and this looks like one of countless demarcation closets. Sorry to tell you whoever gave you this information was perhaps drunk or uninformed. The “web” is built upon root servers in secure facilities attached to countless other servers around the world. Notice there is no large, (OC48 or higher), fiber in this picture and I am sorry to tell you the ISPs in question do not run their networks on CAT5.

Nice try, April 1st is still a ways away!

OK I went back and read the article…. seems absurd but yes, I agree, why the heck are you publishing this? Do we really need to put this knowledge in the public domain? I hope you guys don’t get in trouble.

yeah, a US equivalent of LINX would be useful for traffic like that, but then you have the danger of it being misused by the government to regulate the internet. and with how things are going in the US right now, i don’t think we need to make things any easier for the government to expand its “national security” operations

dude, I would bet my life savings that there’s a homeland security dongle that just sits there and mills up captured packets all day. I’m surprised nobody’s talked about that yet. The real terrorists are the ones in charge.

I’d kill power on the largest piece of interconnected-looking hardware I could. I’m sorry guys, I’d have to. The chance to bring down the *entire* internet for a short time? Surely you jest. That’s like having the means to drop your pants and moon out of the back window of the coach but with the ENTIRE WORLD as your audience. Wrong? Hell yes, but you know what they say – power corrupts, absolute power corrupts absolutely. ;)

Saheeb Longadong

March 13th, 2008 at 2:50 pm


Why are we a nation so quick to show vital information at this time of war and terrorists, not to mention idiots trying to make a name for themselves. I see shows telling how many new Hummers are going to Iraq, where the troops take a dump, and on and on. We are our biggest enemy. When the lights go out we can shake our fists at ourselves in the mirror, in the morning of course, when the sun comes up and we can see. I think mirrors will still works. Maybe not. We need a bigger gene pool. Or we need to invade Canada and spread out the blast zones so some of us survive. Just kidding Canada, we would never attack you. If we were going to you could just read about 6 months in advance in 27 newspapers and TV shows. So we would have to call it off.

The people commenting here really don’t understand how the Internet works and think they can fix it.

Henry and Mossgard have a problem with you publishing this information because they fear the terrists. Well this information is already public domain. It’s not hard to track down the physical location of a device on the internet and unless your network is behind a firewall all the information (content and registration info) is in the public domain.

Many people brought up LINX… this is not unique to just London. There are many internet exchanges throughout the world. They don’t need to be an entire facility, some are as small as a broom closet. Any place where providers can physically connect to each others networks is considered an internet exchange really.

http://en.wikipedia.org/wiki/Internet_exchange

You Americans are living in a perpetual state of fear!

Is this kind of discussion commonplace in the US of A? Ie; all issues are looked at under the facet of terrorism?

No, this is not a commonplace discussion. Just a few ill-informed geeks talking crap. We fear drive-bye shootings in Compton or Watts a lot more than worrying about a few terrorists strolling around downtown LA with explosives taped to their testicals. Actually we think about surfing and pretty girls a lot more than terrorists.

Sure it is. See the picture? Roughly one huge mess of cables intersecting another huge bunch of cables.? That makes perfect sense, right?

260 ISP’S using cat 5 and other cables spliced together in a building in downtown LA. You see the internet is nothing more than a bunch of soup cans and tightly pulled string.

People are so damn gullible.

Henry, do you really think an OC48 is *physically* a larger cable?!? It’s just an multimode fiber cable. Visually, it’s gonna look identical to a fiber Ethernet cable. So, yeah, any of these cables *could* be OC48 (or 10Gig, etc). Most network cabling looks a hell of a lot alike. It’s really not possible to tell from a picture of a cable tray whether that’s a key internet meet-me point or just a moderately large datacenter.

BTW.. for anyone interested, here’s a picture of the connector ends for an OC48 cable (http://www.cisco.com/en/US/docs/routers/10700/10720_uplink_cards_install_config/13113i.html#wp1039180)

Well, you are forgetting the Redundant locations, disaster recovery locations, etc.

It would be pretty stupid for a Company to put all of his major infrastructure in a single building….. well, true is I have seen cases…

Cheers

You guys are paranoid about the wrong thing. Some mystery terrorist from whatever country you guys are supposed to hate this month won’t do a damn thing to that building. What I’m worried about (since I live in North America and most of my traffic goes through that room at some point as well) is all the black-suits from various 3-letter government organizations getting a nice convenient place to snoop just about all data that flows through teh pipes.

Never mind the terrorists, I’m more concerned that this is located right on top of the damn San Andreas fault line!

Not to mention at one corner of one of the most busiest streets in down-town L.A… all it’d take is one idiot in a truck and the whole building could be put out of commission.

screw scissors, screw runaway trucks.

A disgruntled office worker and
just 1 M-80 firecracker shoved between some wires would do it.

That would be damn funny…….hey my screen blanked out OH CRAP MY PORN!!!!

That’s just one of MANY cross connect sites in the world. We have a building in St. Louis that houses all of the major internet backbones also. It’s no big secret that there are locations that do nothing but host the pipes and data centers, ya’ll think this stuff runs out of the cable companies alone? I’d wager that most web hosting companies run out of buildings like this that have major access to the pipes. They’re all setup with redundant power, generators and backbone pipes, it just makes sense.

Trip over wires…

“sorry it took us a year folks, noone told us on the nets so we didnt fix :)

Um… a couple of you guys are using the phrase “public domain” … I know what you mean, but it’s worth pointing out that this is a legal phrase, not a statement of availability.

In actual fact, this information is -not- in the public domain. It’s copyrighted. The photo is copyrighted to the photographer, the article is copyrighted to the blogger, and I don’t know who the comments belong to….

you do realize that this info is already public if he knew it. he’s just telling about it. If I had a blog and I knew about this I would post it too.

NotASelfAbsorbedAmerican

May 6th, 2009 at 2:56 pm


>Because Canada sucks and can’t last 3 days without US
LOL

Do you realize that Canada has a smoother economy and Health Care for their citizens that our Country the United States could never equal? So many people bag on our Northern Neighbor it truly is pathetic. The U.S. is just a glorified spoiled brat in too much debt and big guns…sounds like a global bully to me. Yes I am a natural born citizen here but I am far from pleased at the continuous downfall of the government and those who ‘run’ this Country. If I were blessed with the means to bail from this place, sadly I would. It is hard to watch a place you so wish to love fall apart around you and take you with it.

There are a lot of comments being made here that are innaccurate. I have been inside one wilshire, my company (backbone internet provider has some equipment there)

There are several datacenters in the building, one is operated by CRG West. We’re there participating in some peering arrangements with other ISPs. This can be done through a peering exchange which is essentially a set of large network switches. Ports on an exchange are expensive, so many providers arrange for private peering where they connect to eachother. Besides the large grandfather type ISPs, most of this is done via gigabit or 10 gigabit ethernet, both mostly over single mode fiber. The large granddaddy ISPs like to use things like OC-48 and OC-192 which perform the same function but cost 10 times as much.

I’m not sure which facility in one wilshire this is, but I don’t recall seeing a mess this big. Some datacenters are old and nasty like this, some are new and shiny. I’ve seen these pictures float around the internet for years. If there was a mess like this in my datacenter, I’d get fired, and I spent two months designing cable management systems to prevent a mess like that from ever happening. We have a tech right now who does wiring like that an I have to clean it up. Although he doesn’t realize it, he’s very close to getting fired or transferred.

In the picture I see lots of cat5, some 25 pair wire, a few fibers, and some fiber tubes. Typically datacenters use separate cable trays for fiber and copper wire, so it’s entirely possible this is inside a carrier hotel or peering point, and we’re just not seeing the fiber tray.

As far as the comments about terrorism go. Let’s say some nutcase took out this building. We would re-route traffic around it, end users would notice a hiccup. There is another building right down the street where we have even more infrastructure. Users in the LA region would definitely feel a loss of service as one of their major peering points is disrupted. Beyond the local area, some fiber routes across the pacific would go down since they eventually land at one wilshire for routing purposes. The traffic levels on other trans-pacific fibers would go up, such as our circuits landing in SFO and down the street. The internet would be quite congested but would still function ok, depending on who your ISP is and how they operate.

Our company runs with very high redundancy, we carry a lot of financial data and our customers scream to high heaven about any latency, delay, or packet loss. When a circuit hits 80%, we pre-emptively order another (it’s hard to make money like that) and our users would probably not even notice the loss of one wilshire unless they had traffic specifically destined for an ISP taken offline there.

Many of our competitors constantly run their network at full capacity and are totally behind the curve on upgrades, so they do things like limit download speed and use other QoS to handle the load rather than just having a good network with free flowing traffic. As a result, we are a bit more expensive, but there’s nothing wrong with being on the higher end of a market I guess.

In any major city there are a number of these places, in smaller markets, there are one or two. Some of the major peering points and carrier hotels include one wilshire, 529 Bryant in SFO, 350 E. Cermak in Chicago, Infomart in Dallas, 111 8th and 60 Hudson in NY, lots of datacenters in Ashburn, VA (DC) and Global Switch in Amsterdam.

Most of these buildings house multiple competing datacenters such as CRG West, TelX, Switch and Data, and Equinix. These are all publicly traded companies you can research. Many peering exchanges (the switch) are operated as non-profit to avoid conflicts of interest with the huge number of ISPs they deal with.

Some of the major backbone providers include Cogent, Level3, and Global Crossing. The grandfather type ISPs do a lot of retail traffic to the home, and the backbone providers wholesale IP to large websites, and then peer the traffic off to the ISPs.

FearBotFiftyFiveThousand

May 10th, 2009 at 6:46 pm


@NotA: Don’t feed the trolls.

@USA folks: People like you are the reason that children get expelled for pointing chicken fingers at their teachers in the lunchroom.(http://everything2.com/title/Boy%2520suspended%2520for%2520pointing%2520chicken%2520finger%2520like%2520gun)

“So much for not having a single point of failure:”

There are similar meetme rooms in every datacenter around the world, due to the nature of resilient networks these do not pose a threat to connectivity as much as say, trans-atlantic fibre.

might as well have put a bulls-eye on this building…nice work.

We should hope no cyber-terroists find out about this place. If they do, it would probably be near the top of their hit list. Stay hidden, One Wilshire, stay hidden.

This is not unusual; the major hubs all have a facility that looks similar. Atlanta, Chicago, LA, Seattle, NYC and Dallas (where I worked). There are some others but those are the major, and thus the biggest, ones.

There is no “tripping over” the cables. They are in overhead racks. Also, everything you see in that photo appears to all be CAT5 or CAT6. None of it looks like fiber so this is likely an old photo.