# FYI: Hardware Failure Discussion



## Dragoneer (Jun 19, 2008)

As was posted on the Site Status by Yak it appears FA has suffered a hardware failure and is suffering degraded performance. Because of the generous support and donations by the userbase we have been able to order a new hard drive, and have overnighted it to down to Virginia. If it arrives on time we will attempt to install it sometime later tomorrow night.

We'll be keeping you informed as to the status.

In the mean time pleas try to consolidate your discussion here.


----------



## STrRedWolf (Jun 19, 2008)

Um, what does this have to do with 404's on every page (including the front page) of the site?  Did someone forget to put in a fake page in the directory that is usually used to mount the RAID?


----------



## Dragoneer (Jun 19, 2008)

STrRedWolf said:


> Um, what does this have to do with 404's on every page (including the front page) of the site?  Did someone forget to put in a fake page in the directory that is usually used to mount the RAID?


FA had a hardware failure... and... uh, the site is currently experiencing problems because of it? We're working on getting something setup a bit more proper for it.


----------



## xxdarkdemonessxx911 (Jun 19, 2008)

so if things go right it should be back up by tomorrow night or saturday right? or is it saturday or sunday?


----------



## Dragoneer (Jun 19, 2008)

xxdarkdemonessxx911 said:


> so if things go right it should be back up by tomorrow night or saturday right? or is it saturday or sunday?


That's the idea... 

It's in Newegg's corner right now. I overnighted a new Raptor via FedEx. If they're able to get it to us tomorrow... I'll go to the colo myself and swap it out. If not one of the other FA server techs will swap out the drive over this weekend.

It *WILL* be fixed no later than Monday.


----------



## STrRedWolf (Jun 19, 2008)

Um... yeah.  I don't see the connection between server software and server hardware in that message (nor in Site Status).

Keenspot's Comic Genesis has a similar set up for their forums (being the admin there, I know this).  Unmounting the drive that has the forum site on it will start 404'ing the entire site (in the usual case).

So since you say it's connected in some shape or form that you can't describe, I'm feeling rather free to guess that the failed hardware has been unmounted from a directory where the web server software was told to find it, and is thus 404'ing like mad.  

Am I right?  Besides, I find users more understanding when they know that this is a side effect of the hardware failures and it comes from an administrator's mouth.


----------



## xxdarkdemonessxx911 (Jun 19, 2008)

Dragoneer said:


> That's the idea...
> 
> It's in Newegg's corner right now. I overnighted a new Raptor via FedEx. If they're able to get it to us tomorrow... I'll go to the colo myself and swap it out. If not one of the other FA server techs will swap out the drive over this weekend.
> 
> It *WILL* be fixed no later than Monday.




ok thank you :3


----------



## Shark_the_raptor (Jun 19, 2008)

Thanks for the info, Dragoneer.  Much appreciated.  -goes back to browsing the forums-  X3


----------



## Dragoneer (Jun 19, 2008)

STrRedWolf said:


> Um... yeah.  I don't see the connection between server software and server hardware in that message (nor in Site Status).
> 
> Keenspot's Comic Genesis has a similar set up for their forums (being the admin there, I know this).  Unmounting the drive that has the forum site on it will start 404'ing the entire site (in the usual case).


Fur Affinity is not Comic Genesis, and our setup is different. Just because your site is setup one way does not reflect or represent our own.


----------



## rednec0 (Jun 19, 2008)

with it 404ing earlier i kinda figured it was either someone accidentally deleted the front page or a hardware failure, but since it is the latter issue this is more serious. i wouldn't be surprised if the original drive was a WD and if it IS then IMO it wouldn't be wise to use another WD to replace it (considering the reputation of immediate failure after install (sorry for the exaggeration)).


----------



## Dragoneer (Jun 19, 2008)

rednec0 said:


> i wouldn't be surprised if the original drive was a WD and if it IS then IMO it wouldn't be wise to use another WD to replace it (considering the reputation of immediate failure after install (sorry for the exaggeration)).


They are WD, but it utilizes 4X WD Raptors which are Enterprise-grade drives and not standard fair for WD. Keep in mind these drives have been spinning 24/7 non-stop since January 2007. Hard drive failures are common no matter what brand, and even for Enterprise-grade drives. It's unfortunate. We have punished our drives to no end, and they've held up well.

Out of 10 drives we're down 1 in 3.5 years. That's pretty good statistics, actually.

Gecko (Retired): 2X WD Caviar 250GB Drives
Bahamut (DB): 4X WD Raptor 74GB in RAID 10
Tiamat (Data): 4X Seagate Barracuda 500GB in RAID 5


----------



## STrRedWolf (Jun 19, 2008)

Dragoneer said:


> Fur Affinity is not Comic Genesis, and our setup is different. Just because your site is setup one way does not reflect or represent our own.



Obviously (our regular server runs on Scary-it-ain't-funny RAID).  As mentioned before, I don't know your setup and the lack of information only promotes stupid wild-ass guessing on everybody's part.

As I said before, more info means less confusing people asking why something's down.  And I'm one confuzzled fur (I just wanted to post my pics!). 

So, I ask once again:  Did the dead drive bearing array get removed from service, causing the 404's?  I'd love to know so I can stop bugging you and yak and just get on with queque'ing artwork to be send up to FA when it comes back up on most likely Monday (if I remember Newegg correctly, the cut-off is 12 noon PST, otherwise it's likely to ship the next day).


----------



## Wolf_Fox_Guy (Jun 19, 2008)

You said there was some kind of problem with the hard drive. Not panking but just checking, all of the information on the site such as art that has been submitted will not be affected or lost right?


----------



## Arcturus (Jun 19, 2008)

And you didn't have a hard drive in reserve ready to be swapped out.. why?


----------



## Erro (Jun 19, 2008)

Arcturus said:


> And you didn't have a hard drive in reserve ready to be swapped out.. why?


$$$$$$$$$$$$$$$$$$$$
it doesn't grow on trees


----------



## rednec0 (Jun 19, 2008)

Arcturus said:


> And you didn't have a hard drive in reserve ready to be swapped out.. why?



nothing but an afterthought. no offense but i'm sure they'll be sure to have a backup later on in case this does happen again.



Erro said:


> $$$$$$$$$$$$$$$$$$$$
> it doesn't grow on trees



that definitely calls for a motivational poster to be made later


----------



## Arcturus (Jun 19, 2008)

Well, FA has apparently had enough money to buy one -now-, so.. why not before? It's not like 500GB drives are expensive.


----------



## quentinwolf (Jun 19, 2008)

Wolf_Fox_Guy said:


> You said there was some kind of problem with the hard drive. Not panking but just checking, all of the information on the site such as art that has been submitted will not be affected or lost right?



No, All the data will be fine.  Raid10  Is basically Raid 0 (Drives are accessed simultaneously to increase the speed/transfer rate) and Raid 1 (Mirroring) Together.  So with 4 drives, thats's  2 x 2.  Drive A + B are in one set, and C + D are in another set.  (A+B is an exact copy of C+D.)  Does that make a little more sense? 

Just because 1 drive died, doesn't mean any data is lost, it just means that one half of the set is now down, and its usually best to get a replacement drive ASAP just incase another drive may go down (In which case data could be lost)  But they noted they are performing a immediate backup of the database, so even then, it will still be fine.

Hope that clears things up a little better.


----------



## Wolf_Fox_Guy (Jun 19, 2008)

quentinwolf said:


> No, All the data will be fine.  Raid10  Is basically Raid 0 (Drives are accessed simultaneously to increase the speed/transfer rate) and Raid 1 (Mirroring) Together.  So with 4 drives, thats's  2 x 2.  Drive A + B are in one set, and C + D are in another set.  (A+B is an exact copy of C+D.)  Does that make a little more sense?
> 
> Just because 1 drive died, doesn't mean any data is lost, it just means that one half of the set is now down, and its usually best to get a replacement drive ASAP just incase another drive may go down (In which case data could be lost)  But they noted they are performing a immediate backup of the database, so even then, it will still be fine.
> 
> Hope that clears things up a little better.



Yes it does, thank you. Not having FA till monday is admitedly a bit of a bother but I have other things to do in the mean time. the important thing to me is that the art is allright and that I know the staff is aware of the issue.


----------



## Dragoneer (Jun 19, 2008)

STrRedWolf said:


> So, I ask once again:  Did the dead drive bearing array get removed from service, causing the 404's?  I'd love to know so I can stop bugging you and yak and just get on with queque'ing artwork to be send up to FA when it comes back up on most likely Monday (if I remember Newegg correctly, the cut-off is 12 noon PST, otherwise it's likely to ship the next day).


No, the drive is still in the box, and will remain there for the moment.

And since I've also ordered over $30,000 worth of crap from Newegg I seem to tend to get better service.  I have literally ordered from them and gotten my stuff the same day of order.


----------



## Dragoneer (Jun 19, 2008)

Wolf_Fox_Guy said:


> You said there was some kind of problem with the hard drive. Not panking but just checking, all of the information on the site such as art that has been submitted will not be affected or lost right?


The data is more than fine.  We can move our backup data to the data server for further redundancy.


----------



## Dragoneer (Jun 19, 2008)

Arcturus said:


> Well, FA has apparently had enough money to buy one -now-, so.. why not before? It's not like 500GB drives are expensive.


The data server has a spare drive in it as a failsafe. There was never a spare drive for the DB server.

Given where I'm located, I'm sure I can find a 74GB WD Raptor... if the drive doesn't come in via FedEx in time I'll see if I can run out to a store after work tomorrow and get a new drive, and install it anyway. Then keep the other drive as a failsafe, just in case.

This'll eat up all the funds to retrofit the Gecko server, but I think this is a higher priority issue.


----------



## quentinwolf (Jun 19, 2008)

Wolf_Fox_Guy said:


> Yes it does, thank you. Not having FA till monday is admitedly a bit of a bother but I have other things to do in the mean time. the important thing to me is that the art is allright and that I know the staff is aware of the issue.



The art is more than alright,  Its on a seperate set of drives too.  The drive thats failing is one of the ones that holds all the posts, text, submission information, user database, etc I believe... The art is on a Rad 5 array, which is slower than Raid10, but even if a drive fails in that, it will also be okay, because it can calculate the missing data (As long as 2 drives dont fail.)   (I think I have this right.)  But again, no worries.

More information about the various levels of RAID here, http://en.wikipedia.org/wiki/Redundant_array_of_independent_disks

I agree though, kinda a bummer about this happening before a weekend...  Well happening any day for that matter..  Always throws my schedule off, I now have to fill in an hour with something else to do.. *chuckles*


----------



## Dragoneer (Jun 19, 2008)

quentinwolf said:


> The art is on a Rad 5 array, which is slower than Raid10, but even if a drive fails in that, it will also be okay, because it can calculate the missing data (As long as 2 drives dont fail.)


The art is actually on a RAID 5+1, so if one drive fails the system will auto-rebuild. We can potentially lose another drive on top of that (though I'd rather never have to chance that).


----------



## Arcturus (Jun 19, 2008)

Dragoneer said:


> The data server has a spare drive in it as a failsafe. There was never a spare drive for the DB server.
> 
> Given where I'm located, I'm sure I can find a 74GB WD Raptor... if the drive doesn't come in via FedEx in time I'll see if I can run out to a store after work tomorrow and get a new drive, and install it anyway. Then keep the other drive as a failsafe, just in case.
> 
> This'll eat up all the funds to retrofit the Gecko server, but I think this is a higher priority issue.



And why.. didn't you have a spare for the DB server?


----------



## Dragoneer (Jun 19, 2008)

Arcturus said:


> And why.. didn't you have a spare for the DB server?


Given how much I moved around, was in Asia... it wasn't practical at the time. I don't have spare CPUs or RAM on hand either.


----------



## Nanakisan (Jun 19, 2008)

To those who keep littering Dragoneer with this Non-sense please stop.
He has enough stress as it is.

The servers hardware has failed and because of that servers connected to the main server are being disrupted.
I'm not an admin or mod or whatever but when i see this blatent lack of respect for the wishes of someone it just infuriates me.

Please forgive me if i'm being brash but as of now i think we should just calmly stop annoying the admins and let them fix things like they have so i nthe past.
thank you.


----------



## STrRedWolf (Jun 19, 2008)

Nanakisan said:


> To those who keep littering Dragoneer with this Non-sense please stop.
> He has enough stress as it is.
> 
> The servers hardware has failed and because of that servers connected to the main server are being disrupted.
> ...



Now you know how I felt before I started explaining to folks what happened, with enough detail that folks understand and don't bug me (or the co-admin) when things go wrong, on my servers. 

Let's let Dragoneer post a "Okay, this is what we think is wrong, and these are the symptoms you may have encountered" update to Site Status.


----------



## Arcturus (Jun 19, 2008)

Dragoneer said:


> Given how much I moved around, was in Asia... it wasn't practical at the time. I don't have spare CPUs or RAM on hand either.



Well, you should, really.


----------



## Dragoneer (Jun 19, 2008)

STrRedWolf said:


> Let's let Dragoneer post a "Okay, this is what we think is wrong, and these are the symptoms you may have encountered" update to Site Status.


Yak posted that like... 12 hours ago.


----------



## quentinwolf (Jun 19, 2008)

Dragoneer said:


> The art is actually on a RAID 5+1, so if one drive fails the system will auto-rebuild. We can potentially lose another drive on top of that (though I'd rather never have to chance that).



Better yet. 

By the way...  What type of Raid 5 compatible controller would you recommend for a home server?  I'd like to expand my multi-purpose web/storage/media server, and would love to go with Raid 5, but would like a fairly good controller that 1. Wont cost me over $1000, and 2. Isn't one of those half software - half hardware based (or rather, Hardware assisted Software based cards.)  Raid 5 seems the best for balanced between cost/performance/reliability as I can grab 3 or 4 500gig drives and be set for good long while.  Reliability is my primary concern, and would like something that meets me in the middle, rather than 'Double the drives, double the reliability sort of thing' (Right now I have a couple of 200 gig drives in kinda slow software Raid 1 mode... Thanks Windows Server 2k3.)

I'd prefer a full hardware raid card, so my processor can worry about other things...  But I'm in Canada, and my computer supplier ( www.ncix.com ) doesn't really have many raid cards, and newegg doesn't ship to canada, so I'll have to find other means, but need a good reliable card to start hunting for in the first place.


----------



## lilEmber (Jun 19, 2008)

Why a raptor? a Samsung Spinpoint F1 is not only 1 TB <..< its faster due to the ammount of heads.

a raptor is spinning at 10000 with one or two heads a samsung spinpoint it 7200 with 4

Just saying its faster, and larger <..<

and cheeper...only $200


----------



## Dragoneer (Jun 19, 2008)

NewfDraggie said:


> Why a raptor? a Samsung Spinpoint F1 is not only 1 TB <..< its faster due to the ammount of heads.
> 
> a raptor is spinning at 10000 with one or two heads a samsung spinpoint it 7200 with 4
> 
> ...


The 1TB is faster for raw transfers, but it's still not suitable for heavy database work. Ideally, if we had the budget we'd have gone for 4X 15K RPM SAS drives.


----------



## creaturecorp (Jun 20, 2008)

Arcturus said:


> Well, FA has apparently had enough money to buy one -now-, so.. why not before? It's not like 500GB drives are expensive.



I beg to differ. I bet enterprise drives cost more than the average bargain basement drive, not to mention the cost to ship it.


----------



## kewlhotrod (Jun 20, 2008)

Well, apparently FA is back up for now, so I guess that's a plus, haha.


----------



## Nanakisan (Jun 20, 2008)

ha see what happens when everyone calms down.
now loo kthe server is back and we can all enjoy our favorite artists again
great job FA Staff
wooooooooot


----------



## Dragoneer (Jun 20, 2008)

Nanakisan said:


> ha see what happens when everyone calms down.
> now loo kthe server is back and we can all enjoy our favorite artists again
> great job Dragoneer
> wooooooooot


Yak and Tsawolf deserve more thanks than I in this matter.


----------



## kitsunefoxfire (Jun 20, 2008)

Was wondering what was going on.But i suppose hardware failure is pretty common.cant expect it not to wear out eventually.


----------



## nrr (Jun 20, 2008)

quentinwolf said:


> No, All the data will be fine.  Raid10  Is basically Raid 0 (Drives are accessed simultaneously to increase the speed/transfer rate) and Raid 1 (Mirroring) Together.


RAID 0 is striping, which means that you have an even amount of data spread across an even amount of disks.  If you had an object that you were committing to disk that was, say, 4k blocks in length, you'd have an alternating pattern of blocks strewn across both disks.  That is, object = {a[0], a[1], a[2], a[3], a[4], a[5], ..., a[4195]} => disk0 contains {a[n] | n is even} and disk1 contains {a[n] | n is odd}.

They're not exactly read simultaneously, but the disks are allowed to cache the reads so it appears that they are.


----------



## nrr (Jun 20, 2008)

quentinwolf said:


> I'd prefer a full hardware raid card


Caveat emptor: If your RAID controller goes south on you, and you don't have adequate backups of the data on the attached disks, unless you can procure a replacement card down to almost the letter, you can consider your data as good as gone.  This doesn't happen with software _usually_.

No, RAID is not a substitute for backups.  Buy yourself another set of disks and mirror to those periodically.  Run an incremental job through the week and a full job on Fridays.


----------



## Karukatsu (Jun 20, 2008)

it hopefully will holdup till tommorow! If it doesn't a hundreds of people are gonna pour into the forums again..


----------



## lilEmber (Jun 20, 2008)

Dragoneer said:


> The 1TB is faster for raw transfers, but it's still not suitable for heavy database work. Ideally, if we had the budget we'd have gone for 4X 15K RPM SAS drives.



The 14k SAS are unstable actually I hear a LOT of errors, even though they are solved with warranty it usually takes a few months for them to be right, even then you run the risk of it sticking and ripping half the pc apart o..o I seen once on a 14k drive's all heads stuck at the same time.


----------



## Manguneshane (Jun 20, 2008)

shit, what could that mean?
will It effect any of the,stored art, the user profiles?
There is a large majority of my work that, I had to delete to make room for more.
Is it that they are gone?


----------



## Erro (Jun 20, 2008)

Manguneshane said:


> shit, what could that mean?
> will It effect any of the,stored art, the user profiles?
> There is a large majority of my work that, I had to delete to make room for more.
> Is it that they are gone?


back-reading is your friend


----------



## Killerdwagon (Jun 20, 2008)

FA shall always prevail! no failure shall stop it! :B


----------



## Danza (Jun 20, 2008)

Killerdwagon said:


> FA shall always prevail! no failure shall stop it! :B



The failure of money will though ;P


----------



## nrr (Jun 20, 2008)

NewfDraggie said:


> The 14k SAS are unstable actually I hear a LOT of errors, even though they are solved with warranty it usually takes a few months for them to be right, *even then you run the risk of it sticking and ripping half the pc apart* o..o I seen once on a 14k drive's all heads stuck at the same time.


Emphasis mine.  Yeah, and I've seen other disks do this too.  I've also seen other disks refuse to unpark their heads and various other things.  Moreover, what do you mean "run the risk of it sticking and ripping half the pc apart"?  That sounds absurd at best.

In production, small 15kRPM SAS disks (on the order of 36GB or so) tend to be the best that's available for database applications, and that's going off of empirical data.


----------



## Wovstah (Jun 20, 2008)

Thanks for keeping us posted, Dragoneer.


----------



## fastturtle (Jun 20, 2008)

Habitually Correct:

You're dang tooten that RAID is not a backup solution and that everyone should backup their data regularly. For those who don't know what RAID is designed for it's pretty simple really. It's designed for High Availability, meaning something that absolutely has to stay up.


----------



## Frostflame (Jun 20, 2008)

Well, at least we know the source of the error 404 and that it will be fixed by Monday. FA have a good team of technicians! (if that's the correct word to use )


----------



## RailRide (Jun 20, 2008)

Manguneshane said:


> shit, what could that mean?
> will It effect any of the,stored art, the user profiles?
> There is a large majority of my work that, I had to delete to make room for more.
> Is it that they are gone?



Regardless of how secure FA's submissions are, the above situation is a precarious one to be in. The only people who actually _do_ lose their artwork to website malfunctions are the folks in this category.

Invest in an external hard drive, or if that's too costly, a couple of USB thumb drives. You can usually get 8GB models for less than $50 (I paid a little less than $40 for the last 8GB flash drive I bought, and at the same retailer, a 4GB version is going for $13 after a five-dollar rebate)

I keep my art projects (270+ seperate uploads on FA alone), including the full-size, uncompressed versions, backed up to two external USB hard drives _and_ the flash drive that harbored them during their WIP phase. For that matter, I even backup copies of my sumbission _descriptions_ too since they tend to contain details of the piece's creation and methods used. In a worst-case scenario, I can restore everything I posted and whatever I wrote to describe them. 

---PCJ


----------



## Pi (Jun 20, 2008)

NewfDraggie said:


> you run the risk of it sticking and ripping half the pc apart



I call bullshit. The only documented case of anything resembling anything like this that I've ever heard about (Feel free to correct me) was the power failing during backups on a VAX-11/70. These machines used "washing-machine" disks, and one of the spindles was marginal, and when the power failed, the rotational momentum from the platters flung them quite literally out the window, or into the tape racks, slicing their second-most recent backup tape in half.


----------



## Erro (Jun 20, 2008)

Pi said:


> I call bullshit. The only documented case of anything resembling anything like this that I've ever heard about (Feel free to correct me) was the power failing during backups on a VAX-11/70. These machines used "washing-machine" disks, and one of the spindles was marginal, and when the power failed, the rotational momentum from the platters flung them quite literally out the window, or into the tape racks, slicing their second-most recent backup tape in half.


Seconded. It's been tested repeatedly, and rpm test studies have run spindles at much higher speeds than any drive currently on the market, without any such issues. Hard drive failures happen, and it has been known for platters to come apart before, but a case of the disks actually coming off the spindle and making it through the hard drive casing to cause real damage is far too rare to be considered a plausible worry.


----------



## nrr (Jun 20, 2008)

fastturtle said:


> Habitually Correct:


The name's actually nrr, but I appreciate the flattery.



			
				fastturtle said:
			
		

> You're dang tooten that RAID is not a backup solution and that everyone should backup their data regularly. For those who don't know what RAID is designed for it's pretty simple really. It's designed for High Availability, meaning something that absolutely has to stay up.


Yes.

Also of note is the Law of Universal Irony.  It states that the universe tends toward maximum irony when the opportunity is presented.  As it applies to this circumstance with a homebrew RAID rig, if you don't have adequate backups, and one of the disks in your array fails, the universe will bend itself such that another disk in your array will fail.  The formal fields of game theory and probability in mathematics tend to support this notion.


----------



## Arbiter (Jun 20, 2008)

well's hes probably replacing the hardware now, so we shoul;d hopefully be on tonight


----------



## Cold_Burn (Jun 20, 2008)

Any Idea when it should be back up?



NVM it's up


----------



## yak (Jun 20, 2008)

Cold_Burn said:


> Any Idea when it should be back up?
> 
> 
> 
> NVM it's up



Why do you post a comment if you already know FA is up? You did not edit your comment to add the "NVM" part.


----------



## Erro (Jun 20, 2008)

yak said:


> Why do you post a comment if you already know FA is up? You did not edit your comment to add the "NVM" part.


It doesn't seem to add the 'last edited...' line unless you edit after someone else has posted below you.

Edit: further testing, after reply posts.


----------



## Janglur (Jun 20, 2008)

Yeah.  Good choice.  $120 well spent.

Raptors are some wickedly fast drives for being non-enterprise drives.  Surprisingly reliable too!

Kudos.  =3


----------



## Aurali (Jun 20, 2008)

yak said:


> Why do you post a comment if you already know FA is up? You did not edit your comment to add the "NVM" part.



It don't always put the edit there. watch

see nothing


----------



## Dragoneer (Jun 21, 2008)

Janglur said:


> Yeah.  Good choice.  $120 well spent.
> 
> Raptors are some wickedly fast drives for being non-enterprise drives.  Surprisingly reliable too!
> 
> Kudos.  =3


Ended up costing about $360 in the end, actually. I over-nighted a new Raptor to replace it, but the order was put in too late and the drive would not have arrived until Monday. With a problematic hard drive I am not one to take a chance and let an issue scale into something worse, so rather than wait 'til Monday, I zerged to the nearest mega-computer store, procured a new HD and slapped that bastard into the tray. So that ended up costing more than I'd have liked, but when it comes down to system security I want to do things the RIGHT way.

So we're going to end up with TWO spare drives to hotswap, so if we ever have issues in the future we'll be covered. The new drive that's coming in Monday and the eventual replacement I get from Thinkmate, our hardware vendor. All our stuff is covered under a three year warranty.

Also...







Also, I had mentioned that I just happen to work within a mile from the colo. In fact, if there wasn't this big ass street between me and the colo, I'd be exactly .2 miles. I could walk there during lunch.



I swear to god I did not plan it this way, it's purely coincidence.


----------



## Erro (Jun 21, 2008)

*cheers*


----------



## Nanakisan (Jun 21, 2008)

Dragoneer said:


> Yak and Tsawolf deserve more thanks than I in this matter.



i fixed that

lol
Great job FA staff woooot


----------



## nrr (Jun 21, 2008)

Dragoneer said:


>


I noticed that Virginia is pretty big on those huge thoroughfares in urban areas.  They're all over the place in Alexandria, etc.


----------



## Erro (Jun 21, 2008)

nrr said:


> I noticed that Virginia is pretty big on those huge thoroughfares in urban areas.  They're all over the place in Alexandria, etc.


Hehe yeah. Sadly, the first thing I thought when I saw the arrow pointing to the road was "whoa, that looks like where my dad almost rammed his big ass pickup into a pretty Mercedes while we were on a work trip >.> *sadface*
But yeah, I have to say getting around there was surprisingly not nearly as difficult as I expected. If I ever return to the east coast, I will totally be looking at Virginia, awesome place.


----------



## Dragoneer (Jun 21, 2008)

nrr said:


> I noticed that Virginia is pretty big on those huge thoroughfares in urban areas.  They're all over the place in Alexandria, etc.


Yep. This is about 8 miles from Alexandria to boot.


----------



## nrr (Jun 21, 2008)

Dragoneer said:


> Yep. This is about 8 miles from Alexandria to boot.


Hey, the next time I'm in your neck of the woods, let's do lunch.


----------



## Aurali (Jun 21, 2008)

Now I know where to bomb 

I miss Virginia. Lush trees. Large waterways... why I ever decided to move to the desert is baffling at times.


----------

