# FA Maint. Downtime Explaned



## Wet Coyote (Mar 19, 2011)

Downtime on FA explained.

Its easy to assume that when like this Sunday, the FA forums go down for  "20 to 30 minutes" for maintenance, That there will be a 20 to 30  minute downtime.

Looking back historically, theres actually a different phenomena involved, the dreaded FurAffinity NTP - "Net Time Paradox"

In the cases of downtime stated in the 2 to 4 hour range, you can  calculate that each hour of called downtime equals 6 hours in the real  world,
So that 2 to 4 hour called time means 12 to 24 hours in real time.

But, what we have seen in the past, those events calling for less than  an hour, usually translate into each minute of called downtime means  five in real time.

So, we should be looking at a real time downage of 2 hours to possibly 3 this Sunday.

We cannot yet fully explain this paradox, but its always effected FA the  most. Being as its random however and unstable in predictions, its also  highly frustrating to those who cannot live without FA during these  down periods.

Now, we do not know the prediction ratios at all if the 30 minute time  called for will be reached, but there might be a paradox with that end  of it, so our predicted three could spontaneously  turn into 6. We have  detection machinery ready to to go to perhaps capture this event, so we  may better be able to predict the real times in the future.

There will be a particular anomaly to contend with in this event were  not sure about however. With the 'neer down at FWA this weekend, this  might have some impact and skew what data we gather this time in  relation to how it actually happens next time. We will have to hang on  and wait and see. IF your spiritual,  asking for help before this  process starts cannot hurt.

For instance, back in my early days of computing SCSI hard drives  chained together would sometimes not be seen and/or be mounted. We  discovered that the process of sacrificing a chicken could be proven to  assist in SCSI mounting issues.

Those that cannot manage this downtime might want to start planning now,  and be ready with stories or art you want to read tabbed and queued up  already in a separate window that you can figure will give you 2 to 3  hours of FA without the site having to be actually accessed.  This will  help in staving off frustration, loosing feeling in your dominant paw of  lack of fast gripping movements from it, and generally keep you sane.

Don't let yourself be a victim of the NTP - be prepared!

For the rest of us, a good time to go outside for a bit.

This has been a FPSA.  The FPSA in Voluntary Association with FAP and UNF, have  come together to keep you informed and bring you this announcement. If there is an actual  emergency and downtime exceeds 6 hours, please tune to the FA  Forums in the site status section for more information.

This is only a jest


----------



## Mewtwolover (Mar 20, 2011)

Wet Coyote said:


> Those that cannot manage this downtime might want to start planning now,  and be ready with stories or art you want to read tabbed and queued up  already in a separate window that you can figure will give you 2 to 3  hours of FA without the site having to be actually accessed.  This will  help in staving off frustration, loosing feeling in your dominant paw of  lack of fast gripping movements from it, and generally keep you sane.


My signature gives good advice what to do while FA is down.


----------



## BRN (Mar 20, 2011)

I'm reading through this and I'm seeing a bunch of words.

So many words, used so incorrectly. 

Your god exists, and he is crying.


----------



## roobait (Mar 20, 2011)

SIX said:


> Your god exists, and he is crying.


 
_totally_ going in my signature.


----------



## Ozriel (Mar 20, 2011)

I bet a dollar that someone will come in not read the OP and accuse trolls causing the downtime. :V


----------



## kuopiofi (Mar 20, 2011)

Welp, it's been 1 hour since I noticed FA was down. Anyone know how's it going?


----------



## Xenke (Mar 20, 2011)

Hey, but at least now we know _why_ FA was taken down for maintenance.

Also, OP, too many big words. I r dum.


----------



## Ainoko (Mar 20, 2011)

Seems that Wet Coyote has it pegged. Not to mention that common sense dictates that down time is rarely if ever what the admins tell us.


----------



## Carenath (Mar 20, 2011)

Mewtwolover said:


> My signature gives good advice what to do while FA is down.


 Or you could, you know, save your favourites to your computer.


----------



## zachhart12 (Mar 20, 2011)

Wet Coyote said:


> Downtime on FA explained.
> 
> Its easy to assume that when like this Sunday, the FA forums go down for  "20 to 30 minutes" for maintenance, That there will be a 20 to 30  minute downtime.
> 
> ...



You must also take in consideration a quote which I remember Dragoneer saying from somewhere...an apparently 99% uptime...strange with all these paradoxes, isn't it? :O


----------



## Charem (Mar 20, 2011)

Of course, "20 - 30 minutes" is merely an estimate, as any downtime cannot be perfectly predicted.  It's unfair to actually hold staff to an estimate.

But, on the flip side, it is now almost exactly 4 hours after this downtime as I write this.  That's the estimated downtime, times 8.  Proving that the estimate was critically off...

I would just like FA staff to make more-accurate guesses, I guess?  =/  That's hard to do, but it's WAY better to guess something will take LONGER than it does, than it is to guess it will take SHORTER than it does.

In any case, yes, I saw this coming.


----------



## kuopiofi (Mar 20, 2011)

Getting somewhere, Error 503...


----------



## Darkfoxx (Mar 20, 2011)

Wet Coyote said:


> We cannot yet fully explain this paradox


 
Valve time.


----------



## KariLion (Mar 20, 2011)

Carem, 4 hours? FA was working fine about 2 hours ago.


----------



## tonecameg (Mar 20, 2011)

Mewtwolover said:


> My signature gives good advice what to do while FA is down.


 
sofurry sucks, imo
like someone said before, just save it yourself
or try e621
or changing your porn medium
or getting laid


----------



## Dragoneer (Mar 20, 2011)

darkfoxx said:


> Valve time.


 Well, I'm not as fat as Gabe Newell, but... =P


----------



## BRN (Mar 20, 2011)

Dragoneer said:


> Well, I'm not as fat as Gabe Newell, but... =P


 
Nobody can be as fat as Gabe Newell.

Even if anybody attempted, he would grow larger.


----------



## Ainoko (Mar 20, 2011)

The site appears to be back up, but extremely slow for now


----------



## HB Bun' (Mar 20, 2011)

"_Explaned_"


----------



## maxgoof (Mar 20, 2011)

Wet Coyote said:


> Downtime on FA explained.
> 
> Its easy to assume that when like this Sunday, the FA forums go down for  "20 to 30 minutes" for maintenance, That there will be a 20 to 30  minute downtime.
> 
> ...


 
Well, this certainly explains why the site went down at about 12:30 EDT, and is still down two hours later, for a "20-30 minute downtime".  Thanks!


----------



## kuopiofi (Mar 20, 2011)

Hmmm... out again...


----------



## jouva (Mar 20, 2011)

zachhart12 said:


> You must also take in consideration a quote which I remember Dragoneer saying from somewhere...an apparently 99% uptime...strange with all these paradoxes, isn't it? :O


 
99% uptime allows for over 1 1/2 hours of downtime per week FYI. 99.9% only allows about 45 minutes of downtime per month. 99.99% is under an hour for the entire year.



Charem said:


> Of course, "20 - 30 minutes" is merely an estimate, as any downtime cannot be perfectly predicted.  It's unfair to actually hold staff to an estimate.


This is true sometimes. Things happen. The problem is that when the FA staff estimates their down time, it's more often INCORRECT than it is correct.

THAT is my beef and probably other people's beefs. Either the staff is doing plenty of things wrong with server administration that always cause them to enter this "worst case scenario" consistently, or they have bad time estimation skills.

I think that somehow the staff needs to be more truthful with time estimates and be more forward about them, or just give a wider window. I have no problem with knowing it will be down for 4 hours if it's said ahead of time it will be down for 4 hours instead of 1. And I have no problem with it being not accurate once in a while, as opposed to being correct maybe once in a blue moon and incorrect other times.


----------



## Ainoko (Mar 20, 2011)

kuopiofi said:


> Hmmm... out again...


 
yep


----------



## maxgoof (Mar 20, 2011)

Charem said:


> Of course, "20 - 30 minutes" is merely an estimate, as any downtime cannot be perfectly predicted.  It's unfair to actually hold staff to an estimate.



It's not at all unfair to expect them to meet or be less than the estimate 50% of the time, however. When they CONSTANTLY exceed it, it is a problem.


----------



## Ainoko (Mar 20, 2011)

maxgoof said:


> It's not at all unfair to expect them to meet or be less than the estimate 50% of the time, however. When they CONSTANTLY exceed it, it is a problem.


 
I agree with you there Max


----------



## Jameless (Mar 20, 2011)

jouva said:


> 99% uptime allows for over 1 1/2 hours of downtime per week FYI. 99.9% only allows about 45 minutes of downtime per month. 99.99% is under an hour for the entire year.
> 
> 
> This is true sometimes. Things happen. The problem is that when the FA staff estimates their down time, it's more often INCORRECT than it is correct.
> ...



thank you! exactly what I was about to say


----------



## kuopiofi (Mar 20, 2011)

Right, is FA even more wonky than usual right now? Tried to check the page just in case and it's been loading at least five minutes...

I mean, usually it should say something at least instead of just trying to contact the site.


----------



## Ainoko (Mar 20, 2011)

jouva said:


> 99% uptime allows for over 1 1/2 hours of downtime per week FYI. 99.9% only allows about 45 minutes of downtime per month. 99.99% is under an hour for the entire year.
> 
> 
> This is true sometimes. Things happen. The problem is that when the FA staff estimates their down time, it's more often INCORRECT than it is correct.
> ...



Good point there jouva. I think that the admins would get more respect if they say that the the site will be down for 3-6 hours in the event that there are some unforeseen issues  instead of the site will be down for approximately 1 hour for maintenance. Not to mention if the site comes back up earlier than estimated, they look even better. To quote Scotty from Stsar Trek:



			
				Montgomery Scott said:
			
		

> It is better to over estimate the job and to look like a genius when you complete it early than to underestimate the job and look incompetent when unable to complete the job in the time fram you said.


----------



## MitchZer0 (Mar 20, 2011)

SIX said:


> Nobody can be as fat as Gabe Newell.
> 
> Even if anybody attempted, he would grow larger.


 Episode 3 now further delayed


----------



## Xenke (Mar 20, 2011)

Oh hey, FA is up.

Stop the bitching.

EDIT: Oh noes, it died. Let's not start the bitching again.


----------



## maxgoof (Mar 20, 2011)

Xenke said:


> Oh hey, FA is up.
> 
> Stop the bitching.
> 
> EDIT: Oh noes, it died. Let's not start the bitching again.


 
No, it isn't!!


----------



## Accountability (Mar 20, 2011)

zachhart12 said:


> You must also take in consideration a quote which I remember Dragoneer saying from somewhere...an apparently 99% uptime...strange with all these paradoxes, isn't it? :O


 
Server uptime and the site uptime are two different things. Even if the server is serving up 503 errors or the site is showing the offline page, it's still serving pages and is therefore "up".

Only outages like the one today would affect the server uptime number, since they were unreachable for a period of time.


----------



## Xenke (Mar 20, 2011)

maxgoof said:


> No, it isn't!!


 
It was.

Quit yur bitchin.


----------



## zachhart12 (Mar 20, 2011)

Accountability said:


> Server uptime and the site uptime are two different things. Even if the server is serving up 503 errors or the site is showing the offline page, it's still serving pages and is therefore "up".
> 
> Only outages like the one today would affect the server uptime number, since they were unreachable for a period of time.


 
Hmm..i see.  Every 2-3 months FA is down for something anyways...


----------



## coyoteOdin (Mar 20, 2011)

FA today is very slow, and sometimes do not open :-(


----------



## CannonFodder (Mar 20, 2011)

roobait said:


> _totally_ going in my signature.


 Fuck, I wanted that signature.


----------



## thoron (Mar 20, 2011)

There should be a documentary on what happens when furs are denied thier porn.


----------



## maxgoof (Mar 20, 2011)

Accountability said:


> Server uptime and the site uptime are two different things. Even if the server is serving up 503 errors or the site is showing the offline page, it's still serving pages and is therefore "up".
> 
> Only outages like the one today would affect the server uptime number, since they were unreachable for a period of time.


 
This is like claiming that a store is open for business as long as it has a "closed" sign on the locked front door, instead of being vacant.


----------



## Trpdwarf (Mar 20, 2011)

thoron said:


> There should be a documentary on what happens when furs are denied thier porn.


 
Furries have a 13 minute enrage timer when it comes to being denied their porn.
I know this because the last time there was down time myself and another timed how long it would take before some one completely raged.


----------



## Xenke (Mar 20, 2011)

maxgoof said:


> This is like claiming that a store is open for business as long as it has a "closed" sign on the locked front door, instead of being vacant.


 
Welcome to the wonderful world of statistics.


----------



## maxgoof (Mar 20, 2011)

Xenke said:


> It was.
> 
> Quit yur bitchin.


 
So, we can't bitch when it's up, and we can't bitch when it's down, but you can troll at any time.

Glad we now know the rules.


----------



## Devious Bane (Mar 20, 2011)

Great work on downgrading the site guys. 
It runs so much slower now


----------



## Xenke (Mar 20, 2011)

maxgoof said:


> Glad we now know the rules.


 
Yea, those sound about right.

God, if only everyone was as smart as you.



Devious Bane said:


> Great work on downgrading the site guys.
> It runs so much slower now


 
Furries keep mashing the F5.

Stop that furries.


----------



## zachhart12 (Mar 20, 2011)

Devious Bane said:


> Great work on downgrading the site guys.
> It runs so much slower now


 
LOL


----------



## CannonFodder (Mar 20, 2011)

thoron said:


> There should be a documentary on what happens when furs are denied thier porn.


 I'd imagine it's something like this


----------



## Wet Coyote (Mar 20, 2011)

Some took this bit of humor a bit more seriously than I expected. I after all was puling stuff out my butt to just make a funny read, but I did put a wee bit of thought into it.
My intention was never to be critical of scheduled downtime calls and what really happens, because there is so much that can take more time than needed and nothing you can do about it. 
Since stuff had to be shut down correctly, unplugged, rehooked, and restarted - its just common sense that it should not take that long, but probably will. I for one applaud the amount of uptime FA manages with the loads it has on it, of course excepting the unforeseen like equipment failures and DDoS attacks that cannot be predicted nor held to FA's fault.


----------



## ArielMT (Mar 20, 2011)

Got official word, and I posted an update:





ArielMT said:


> We made some changes to our network infrastructure and are currently in the process of optimizing things.  We're aware that access to Fur Affinity will be sluggish or intermittent until this is completed.



Edit: OP's attempt at humor was not lost on me for one.


----------



## CannonFodder (Mar 20, 2011)

ArielMT said:


> Got official word, and I posted an update:


 I don't mind, busy anyhow.


----------



## maxgoof (Mar 20, 2011)

Wet Coyote said:


> Some took this bit of humor a bit more seriously than I expected. I after all was puling stuff out my butt to just make a funny read, but I did put a wee bit of thought into it.
> My intention was never to be critical of scheduled downtime calls and what really happens, because there is so much that can take more time than needed and nothing you can do about it.
> Since stuff had to be shut down correctly, unplugged, rehooked, and restarted - its just common sense that it should not take that long, but probably will. I for one applaud the amount of uptime FA manages with the loads it has on it, of course excepting the unforeseen like equipment failures and DDoS attacks that cannot be predicted nor held to FA's fault.



You just happened to be the Oracle, since you posted that a few hours before it went down, and you were eerily correct that it would be down much longer than estimated.

So, now we are stoning the Oracle.


----------



## Devious Bane (Mar 20, 2011)

maxgoof said:


> You just happened to be *using common sense*.


 Fix'd

And OP wasn't funny at all.


----------



## Ainoko (Mar 20, 2011)

And now the site keeps going up and down


----------



## Wet Coyote (Mar 20, 2011)

maxgoof said:


> So, now we are stoning the Oracle.



LOL


----------



## Ainoko (Mar 20, 2011)

Wet Coyote said:


> LOL


 
You know, you have been spot on with your original post here Wet. As of now, people are going to listen to you (possibly).


----------



## tonecameg (Mar 20, 2011)

Ainoko said:


> And now the site keeps going up and down


 
They did the mash 
They did the f5 mash 
The f5 mash 
It was a furry smash 
They did the mash


----------



## Wet Coyote (Mar 20, 2011)

Ainoko said:


> You know, you have been spot on with your original post here Wet. As of now, people are going to listen to you (possibly).


 

Thats actually a pretty scary thought, Ainoko. But thanks for the vote of confidence


----------



## ArielMT (Mar 20, 2011)

tonecameg said:


> They did the mash
> They did the f5 mash
> The f5 mash
> It was a furry smash
> They did the mash


 
Whatever happened to my Pencilvanian twist?


----------



## maxgoof (Mar 20, 2011)

Glad to see I'm not the only one who recognizes that song.


----------



## Accountability (Mar 20, 2011)

tonecameg said:


> They did the mash
> They did the f5 mash
> The f5 mash
> It was a furry smash
> They did the mash



I don't know if you're trying to be serious or funny (I assume funny), but before people start thinking the site is down because of "F5", it isn't. F5 wouldn't bring down the entire network. This is simply people who have very little clue what they're doing smacking around shiny, expensive toys that _sound_ cool. Why get a load balancer when there's nothing to balance to?


----------



## Willow (Mar 20, 2011)

Accountability said:


> I don't know if you're trying to be serious or funny (I assume funny), but before people start thinking the site is down because of "F5", it isn't. F5 wouldn't bring down the entire network.


 Uhhhhhhhh.....:|


----------



## tonecameg (Mar 20, 2011)

Accountability said:


> I don't know if you're trying to be serious or funny (I assume funny), but before people start thinking the site is down because of "F5", it isn't. F5 wouldn't bring down the entire network. This is simply people who have very little clue what they're doing smacking around shiny, expensive toys that _sound_ cool. Why get a load balancer when there's nothing to balance to?


f5ers gonna f5
-k no more f5jokes-


----------



## Accountability (Mar 20, 2011)

Willow said:


> Uhhhhhhhh.....:|


 
Sorry, I'm running in extra "Serious Business" mode today. I only said that because I know there usually are several people that blame any post-downtime problems on "F5". 

But yes, "F5" wouldn't give me a "connection refused" message. :|


----------



## Ainoko (Mar 20, 2011)

Back up, but for how long?


----------



## Kingman (Mar 20, 2011)

thoron said:


> There should be a documentary on what happens when furs are denied thier porn.


 
That would be great if we could get it narrated by one of the people that do the Nat Geo specials, or go all out and grab Morgan Freeman.


----------



## Willow (Mar 20, 2011)

Accountability said:


> Sorry, I'm running in extra "Serious Business" mode today. I only said that because I know there usually are several people that blame any post-downtime problems on "F5".
> 
> But yes, "F5" wouldn't give me a "connection refused" message. :|


 Oh, okay then. Makes sense now.


----------



## Carenath (Mar 20, 2011)

Accountability said:


> This is simply people who have very little clue what they're doing smacking around shiny, expensive toys that _sound_ cool. Why get a load balancer when there's nothing to balance to?


 Jumping to conclusions again, how cute.


----------



## Accountability (Mar 20, 2011)

Carenath said:


> Jumping to conclusions again, how cute.


 
Okay, then enlighten me on why FA needs a load balancer.


----------



## Carenath (Mar 20, 2011)

Accountability said:


> Sorry, I'm running in extra "Serious Business" mode today.


 How about, every day. You only show up to 'criticise' but do you ever bother to say nice things, when something is done right?


----------



## Volkodav (Mar 20, 2011)

ahahaha i love it
so true.. I was wondering the same thing


----------



## Accountability (Mar 20, 2011)

Carenath said:


> How about, every day. You only show up to 'criticise' but do you ever bother to say nice things, when something is done right?


 
If you do something right, I have no reason to be here? Unless... you _want_ me to be around more. <3

Still would like to know why FA needs a load balancer, so I do not jump to any more conclusions. Maybe I'll even say "good job" because it _is_ necessary after all!


----------



## CR-V (Mar 20, 2011)

Hmm, the site seems to be back up, but now it's impossible to log in....Browsing the site works though.


----------



## Ainoko (Mar 20, 2011)

CR-V said:


> Hmm, the site seems to be back up, but now it's impossible to log in....Browsing the site works though.


 
Not to mention search is down again


----------



## Diocletian (Mar 20, 2011)

I think it's more important to focus on WHAT is being said, instead of WHO is saying it. I think it's been said before that when people are treated and talked to decently, there need not been any sniping and back and forth, and everything is more pleasant and people more co-operative and goodwill is earned.

It was asked why does FA needs a load balancer in this case.


----------



## Trpdwarf (Mar 20, 2011)

Diocletian said:


> I think it's more important to focus on WHAT  is being said, instead of WHO is saying it. I think it's been said  before that when people are treated and talked to decently, there need  not been any sniping and back and forth, and everything is more pleasant  and people more co-operative and goodwill is earned.
> 
> It was asked why does FA needs a load balancer in this case.





> "This is simply people who have very little clue what they're doing smacking around shiny, expensive toys that _sound_ cool. Why get a load balancer when there's nothing to balance to?"


This isn't exactly a question. It's little more than a pot shot and rather unproductive.

A question would be "How would getting a load balancer improve the site"?  "How will this load balancer be used" not "snark snark, there is no use  for this is there." followed by "well tell me what it would be used  for".

I would think users might get sniped at less if some of them would  stop sniping at first chance and ask polite, civil questions and wait  patiently for timely answers.


----------



## Carenath (Mar 20, 2011)

Diocletian said:


> I think it's been said before that when people are treated and talked to decently, there need not been any sniping and back and forth, and everything is more pleasant and people more co-operative and goodwill is earned.


 It is unfortunate, that, this turned out not to be true.

This however, is not the thread in which to debate that, this thread needs to get back on the OP's topic of discussion or it will be closed. I believe there is another thread around for discussing hardware. To that end, this thread's being moved to an appropriate location for discussing our 'Valve Time' tendencies


----------



## inaki (Mar 20, 2011)

So how is this load balancer to be used on the site?


----------



## CannonFodder (Mar 20, 2011)

inaki said:


> So how is this load balancer to be used on the site?


 Prevent DDOS


----------



## Ainoko (Mar 20, 2011)

CannonFodder said:


> Prevent DDOS


 
And hopefully those awful white screens


----------



## Randy-Darkshade (Mar 20, 2011)

Carenath said:


> Or you could, you know, save your favourites to your computer.



This is what I do.


----------



## ab2525 (Mar 20, 2011)

Ainoko said:


> And hopefully those awful white screens


 Only if they're actually balancing _between_ something. 

Also, (insert warning about this being speculation), as far as I know, the site is run with nginx on the front and fastcgi behind it. If that's the case, then you don't even need a load balancer. Just tell nginx about your multiple fcgi servers. (and actually *have* fcgi running on different machines)


If it's to loadbalance file access, something can be easily whipped up in something like tornado.py to loadbalance between multiple static servers, with almost 0 overhead at all (using 302s).


----------



## Ainoko (Mar 20, 2011)

ab2525 said:


> Only if they're actually balancing _between_ something.
> 
> Also, (insert warning about this being speculation), as far as I know, the site is run with nginx on the front and fastcgi behind it. If that's the case, then you don't even need a load balancer. Just tell nginx about your multiple fcgi servers. (and actually *have* fcgi running on different machines)
> 
> ...


 
Interesting


----------



## nrr (Mar 20, 2011)

ab2525 said:


> Also, (insert warning about this being speculation), as far as I know, the site is run with nginx on the front and fastcgi behind it. If that's the case, then you don't even need a load balancer. Just tell nginx about your multiple fcgi servers. (and actually *have* fcgi running on different machines)



Even if that really were the case, I personally wouldn't do it this way.  Make nginx's sole job to expose your FastCGI workers over HTTP and then go from there.

Now, while nginx is indeed well-suited for simply balancing out requests, it does a horrible job of fencing out of the box.  Be wary of that.



ab2525 said:


> If it's to loadbalance file access, something can be easily whipped up in something like tornado.py to loadbalance between multiple static servers, with almost 0 overhead at all (using 302s).



HAProxy + RHCS for redundancy and failover, not to mention something that halfway resembles a commercially available load balancer like the f5 BIG-IP 8800 or Brocade ServerIron ADX 1000.


----------



## theinkfox (Mar 21, 2011)

ab2525 said:


> Only if they're actually balancing _between_ something.
> 
> Also, (insert warning about this being speculation), as far as I know, the site is run with nginx on the front and fastcgi behind it. If that's the case, then you don't even need a load balancer. Just tell nginx about your multiple fcgi servers. (and actually *have* fcgi running on different machines)
> 
> ...


 
correct me if i'm wrong, but that system configuration will only be posible on servers with static and cloned material like backup servers


----------



## Bobskunk (Mar 21, 2011)

EDIT: oh wait there's an actual thread for this now


----------

