# Server Colo Pictures - Courtesy of Net-Cat



## Dragoneer (Dec 9, 2009)

Images from our new colo facility. Net-cat, Dax and myself were there to get things installed, setup and configured. We were there from 5:00pm 'til around 1:00am. Long day, but fun and productive.



All pictures taken by Net-Cat.


----------



## verias (Dec 9, 2009)

So....pretty.....


----------



## Cotoncandie (Dec 9, 2009)

Yay


----------



## ZentratheFox (Dec 9, 2009)

Very nice, gang. Very nice.


----------



## Grrxyn (Dec 9, 2009)

Awesomeness!

I like how clean rack installations look when its actually done right! Kudos to your hard work! ^^

Now I want to see pictures of the damage to net-cat's vehicle... =/  That is some BS!


----------



## Carenath (Dec 9, 2009)

*want*

I'd toss up a picture of the forum server.. but I don't have one.


----------



## keegan (Dec 9, 2009)

Nice work guys, thanks!


----------



## yoshi000 (Dec 9, 2009)

make sure to clean them, so they wouldn't get dirty. *start to clean off my PC*


----------



## Teh Emo Penguin (Dec 9, 2009)

It's the most beatiful thing I've ever seen...   :3
*is happy*


----------



## calaverx11 (Dec 9, 2009)

What are each of the servers, hardware wise? I recognize the Dell on the bottom (is that Trogdor?), we use a lot of those at my workplace.


----------



## RunningCat (Dec 9, 2009)

So that is where all my Furry porn comes from


----------



## yak (Dec 9, 2009)

Oh hey and actual _cable management_, moreso, _color coded cable management_/ Now that's neat and tidy. 
I love now lonely and subpar the uplink cable looks compared to all that.

And what's with the robotron-900 looking PDU


----------



## Kangamutt (Dec 9, 2009)

Very nice! You guys deserve a gold trophy full of steak-umms.


----------



## Armaetus (Dec 9, 2009)

No ending photo of you three?


----------



## DragonTrew (Dec 9, 2009)

Nice wiring job you guys have done! Congratulations.


----------



## KogentaSama (Dec 9, 2009)

Nice job on the wiring.  Wish our server closet was that clean


----------



## quentinwolf (Dec 9, 2009)

Dragoneer said:


> Images from our new colo facility. Net-cat, Dax and myself were there to get things installed, setup and configured. We were there from 5:00pm 'til around 1:00am. Long day, but fun and productive.
> 
> 
> 
> All pictures taken by Net-Cat.



Just curious... Is it really a good thing to have all the ethernet cables bundled up with all the power cables?   I've heard it can degrade performance... Although I guess its a fairly short run so it shouldn't effect too much.


----------



## Devious Bane (Dec 9, 2009)

Awesome.

Potential hazard noted in first image.


----------



## Bruce (Dec 9, 2009)

Does the Colo provide UPS'ed power, or just a generator?  I don't see any 4U boxen in that pile marked APC or Tripp-Lite or Minuteman...

If the Colo can't get the generators started in thirty seconds it might be nice to do orderly shutdowns on everything power-hungry, and leave one box up with an 'away message' while it pages the Admins and sends a canned message to the Forums server hosted elsewhere.  (Help me! Heeeelllllpp Meeee!)


----------



## TheSnowedOne (Dec 9, 2009)

Lovely setup guys - I'll have one to go thanks.  =)


----------



## bear-paws (Dec 9, 2009)

Looks just like Sili Valley in the springtime. Ah, memories. When they were pricier. And less dense.


----------



## Dragoneer (Dec 9, 2009)

calaverx11 said:


> What are each of the servers, hardware wise? I recognize the Dell on the bottom (is that Trogdor?), we use a lot of those at my workplace.


http://en.wikifur.com/wiki/Fur_Affinity#Hardware

Wikifur has an accurate write-up on our hardware loadout. 



yak said:


> Oh hey and actual _cable management_, moreso, _color coded cable management_/ Now that's neat and tidy.
> I love now lonely and subpar the uplink cable looks compared to all that.
> 
> And what's with the robotron-900 looking PDU


I'd planned to do some cable management in there, but Dax beat us to the punch. Dax did all the custom cabling, management and more on our boxes, and it turned out pretty spiffy.


----------



## john8792 (Dec 9, 2009)

"If the Colo can't get the generators started in thirty seconds it might be nice to do orderly shutdowns on everything power-hungry, and leave one box up with an 'away message' while it pages the Admins and sends a canned message to the Forums server hosted elsewhere. (Help me! Heeeelllllpp Meeee!)"


lol


----------



## pikachu_electricmouse (Dec 9, 2009)

That thing at the top is a switch right?  Kinda like a router?

And those things at the bottom, those are like individual PC's, right?

So what's up with each of the things on the bottom having multiple lan connections going into and coming out of them?


----------



## Dragoneer (Dec 9, 2009)

pikachu_electricmouse said:


> That thing at the top is a switch right?  Kinda like a router?
> 
> And those things at the bottom, those are like individual PC's, right?
> 
> So what's up with each of the things on the bottom having multiple lan connections going into and coming out of them?


It's a Cisco 3750, a big ass switch. Our switch we had before was a very basic one we used for our private LAN, nothing more.

And each box is a PC/Server. They have multiple connections for a variety of reason. Yellow connections are internal, that allow us to send data internally that doesn't hit the internet. Blue are external connections that send data to/from the internet. Some of them have multiple blue connections - this lets us remote into some of the servers to manage them long distance.


----------



## yak (Dec 9, 2009)

Each server has two network connections:
* 100Mbit connection to WAN, a.k.a the internet
* 1Gbit connection to a private LAN

LAN connection is used for fast, low latency "side-band" data interchange between servers. More specifically, for one server to connect to the database running on the other, or to transfer backups - all without using up our limited bandwidth on the WAN interface, which is dedicated to serving user requests.

Additionally, two of the servers have Remote Access Cards, or how Sun calls them "Lights-out Management" cards. These things have an interface of their own that one can connect to and perform power cycling, monitoring and configuration of the server itself without having to physically be in the colo and doing it manually.


----------



## calaverx11 (Dec 9, 2009)

Dragoneer said:


> http://en.wikifur.com/wiki/Fur_Affinity#HardwareWikifur has an accurate write-up on our hardware loadout.


What does SirKain do? Is that just a backup server? That's the only thing not listed on that page.


----------



## Carenath (Dec 9, 2009)

Ask Sirkain himself 



KogentaSama said:


> Nice job on the wiring.  Wish our server closet was that clean


I wish I had a server-closet to start with...



Bruce said:


> If the Colo can't get the generators started in thirty seconds it might be nice to do orderly shutdowns on everything power-hungry, and leave one box up with an 'away message' while it pages the Admins and sends a canned message to the Forums server hosted elsewhere.  (Help me! Heeeelllllpp Meeee!)


Most DC's I've ever checked out, provide power to the servers through an Online UPS to provide instant fail-over protection.. usually up to 20 minutes, to the entire setup in the event of minor interruption to mains power.. for longer interruptions, a diesel generator (which will take about 2-3 minutes to start.. and stabilise power outpu).
This is probably not a given for every datacenter, but I'd imagine Switch&Data have this, as would OC3 Networks.


----------



## Dragoneer (Dec 10, 2009)

calaverx11 said:


> What does SirKain do? Is that just a backup server? That's the only thing not listed on that page.


Sirkain is a website we've been hosting for a while. Not just FA is in our racks.  We can (and have been) hosting his server amongst our racks for a while now. The more you host, the cheap it gets.



Carenath said:


> Most DC's I've ever checked out, provide power to the servers through an Online UPS to provide instant fail-over protection.. usually up to 20 minutes, to the entire setup in the event of minor interruption to mains power.. for longer interruptions, a diesel generator (which will take about 2-3 minutes to start.. and stabilise power outpu).


This is correct, and because of that they don't permit you to host your own UPS in the system. Now, the only down side is if a fuse pops...


----------



## Bruce (Dec 12, 2009)

Dragoneer said:


> Sirkain is a website we've been hosting for a while. Not just FA is in our racks.  We can (and have been) hosting his server amongst our racks for a while now. The more you host, the cheap it gets.
> 
> This is correct, and because of that they don't permit you to host your own UPS in the system. Now, the only down side is if a fuse pops...


Been there, Dealt with that - both the fuse part and the generator part, because I did the CO power.  Trust me, -48V @ 4,000A (500KA fault minimum right across the main battery buss) there are places you do NOT want to drop a wrench, you won't get it back - except as flash burns.

Someone tried to level the bussbars with an aluminum level once - Just like Houdini {Poof!} level disappeared.  I was on the other end of the building and I was seeing blue spots.

One constant - No matter how often you test, there were times when the generator fell on it's face.  The power would go out the same day the starting battery went open-cell.  Or it would run for a half hour on the daytank, and then {Koff - Koff - Dedd} you'd find out that the feed pump from the main underground fuel tank had died...

No pressure, you have 6 hours of batteries - but with GTD-5 switches in 100+F weather, you have about two before the switch starts to die from the heat.  Crunch time...

Haven't been by my local switchroom lately (at&t Canoga Park CA) when they were testing - last I heard (sic) they still had the old International Harvester Solar gas turbine.  Sounds like a jet helicopter parked on the roof.


----------



## Dragoneer (Dec 12, 2009)

Bruce said:


> Haven't been by my local switchroom lately (at&t Canoga Park CA) when they were testing - last I heard (sic) they still had the old International Harvester Solar gas turbine.  Sounds like a jet helicopter parked on the roof.


What I love about our new colo is it feels alive. There's this definite pulse in the air and in the building itself. About every second or two, you can feel this gentle vibration from all the power systems...

It's kinda nice!


----------



## Rehka (Dec 12, 2009)

Carenath said:


> ... for longer interruptions, a diesel generator (which will take about 2-3 minutes to start.. and stabilise power outpu).



I wish more people would realize this, when the power goes out at one of our jails, were getting calls within minutes saying "OMG the generator hasn't come on!", or the next day saying "it took our generator like 5 minutes to start!" its annoying, and we always have to send a tech, and they always check it out and its always resolved as "its normal" -_- though  can imagine being in a jail with no power for any period of time can be a bit ....freaky.


----------



## anthroguy101 (Jan 28, 2010)

@Dragoneer
How much storage do these servers have?  Also, somebody mentioned on-site backups on their twitter.  Does this mean that outages are less likely to occur?


----------



## Dragoneer (Jan 28, 2010)

anthroguy101 said:


> @Dragoneer
> How much storage do these servers have?  Also, somebody mentioned on-site backups on their twitter.  Does this mean that outages are less likely to occur?


Trogdor = 6X 1TB Western Digital Blacks
Bahamat = 4X 1TB Western Digital Blacks
Tiamat = 4X 74GB WD Raptors
Novastorm = 2X 74GB Seagate Savvio, 6X 146GB Seagate Savvio
Ghostlight = 2X 250GB 
Amberfall =  TBA soon. =)

And outages are less likely to occur, but anything can happen. Catastrophic outages should be faster to recover from. We'll soon be deploying off-site backups as well.


----------



## anthroguy101 (Jan 29, 2010)

Thanks Dragoneer.  So that's a total of 11.8 Terabytes (with at least ten for art).  Yikes.  That's a lot of picture capacity.  Glad to know we're covered.


----------



## DragonTrew (Mar 2, 2010)

anthroguy101 said:


> Thanks Dragoneer.  So that's a total of 11.8 Terabytes (with at least ten for art).  Yikes.  That's a lot of picture capacity.  Glad to know we're covered.



Not quite, it all depends on the RAID configuration on each server, for instance if we have two 500GB in RAID 1 it means all the data is duplicated so actually you only have 500GB of usable space, but you get the fail-safest configuration...

On the other hand, if you have two 500GB in RAID 1 the data is divided in two and each part goes to a different HD, so you have 1TB of space but if one HD fails all your data is lost...

In RAID 5 you have 3 or more HDs that are divided, each HD sacrifices a fraction of it's space to store part of the data on the entire set, thus you 'lose' one HD in terms of space (3x 500GB in RAID 5 means 1TB) but if one HD fail, you just replace it and everything is ok, you don't even need to shut down the server same occurs on the RAID 1, that is sometimes called hot-swap or hot-plug of HD...

The other RAIDs you get are combination of those like RAID 10 is a set of RAID 0 in a RAID 1 configuration, RAID 50 is two RAID 5 in RAID 0 config.. and so forth...

That was a pretty fast explanation, so forgive me if i get something wrong... Also, no time to talk about RAID 6 so sorry for that as well...

To end this reply, I want to say sorry for my bad English


----------



## DragonTrew (Mar 2, 2010)

On the second paragraph, I meant RAID 0... sorry...


----------



## Ben (Mar 2, 2010)

DragonTrew said:


> On the second paragraph, I meant RAID 0... sorry...



There's an edit button at the bottom of all your posts. Just throwing that out there. :V


----------

