# extreme slowdown? [FA Slowdown/Outage log, fyi]



## Sym_Gryphon (Jan 3, 2006)

I know there may be slowdowns when the number of users and server load are high, but a few minutes ago I encountered a strange speed-related issue...

I was trying to load a user page and noticed it was taking an unusual amount of time. When it finally loaded, I scrolled down to the page stats and saw this:
"This page was generated in 171.30537 seconds. Queries used: 16. Server load: 0.74%"

The number of users online was less than 600 and is steadily dropping.

Is this normal?


----------



## uncia2000 (Jan 3, 2006)

Yep, suddenly started crawling 10-15 minutes ago (went from 0.whatever to 45-90 second response time almost instantly).
Discussing on IRC just now...  ( irc://irc.furnet.org/furaffinity )

Many thanks for the note, Sym.


----------



## Sym_Gryphon (Jan 3, 2006)

The slowdowns seemed to have stopped for now, and mostly affected only user pages and not galleries. Very strange.


----------



## uncia2000 (Jan 3, 2006)

*nods* Response time looking a lot better now, but server load seems a bit higher.

*[ed.]* Pretty much back to "normal" now, I think: trusting it will stay that way


----------



## uncia2000 (Jan 3, 2006)

*noted again, today, just the last few minutes (typically 40 second response time)*

*[ed.]* Only persisted for a few minutes at this level. Back to only being "slow" (5-8 seconds, with 600+ users)

Will keep an eye open to see whether this slowdown persists.
Thank you.


----------



## Myr (Jan 4, 2006)

I'm showing 680 users and getting extreme slowdown too. This also happened last night. Some pages are taking up to a minute or more to load.

EDIT: 692 users, possibly over 700 now
 This page was generated in 152.0749 seconds. Queries used: 16. Server load: 0.75%

EDIT 2: 11:47pm EST
System Locked
Warning!
The system is currently locked up while changes are made to the system and our database.
Please wait and refresh the page once in a while as we should be back online soon.


----------



## uncia2000 (Jan 4, 2006)

*nods*. Thanks, Myr.

Yeah, was back to 5-8 seconds for about 20 minutes, then got worse again.

Now getting MySQL errors, too


----------



## uncia2000 (Jan 4, 2006)

(notes, aside) Server has been locked/rebooted (message "currently locked up while changes are made to the system and our database") and is running normally again, now (2 seconds w/600+ users).

Please note/watch here if there any further long-term response time issues impacting the system's usability.

Thanks & Best wishes to y'all


----------



## uncia2000 (Jan 4, 2006)

*logs note for today*. Slow again the last 5-10 minutes (15-30-60 seconds response) despite being down at around 600-625 users rather than 725+ earlier in the evening.

(Typical response on own page; 
This page was generated in 52.61182 seconds. Queries used: 15. Server load: 1.52% )

Per last two days, the slowdown started relatively abruptly c/w a period of smooth running immediately before.


----------



## uncia2000 (Jan 4, 2006)

_fyi_

(now; 

MySql ERROR! Please check the logs for information on the error(s)

Index Page , as yesterday, albeit earlier in the day this time)


----------



## Myr (Jan 4, 2006)

uncia2000 said:
			
		

> _fyi_
> 
> (now;
> 
> ...


I just got the same thing. It seems to happen randomly when I try to access the site.


----------



## uncia2000 (Jan 4, 2006)

AuroranDragon said:
			
		

> I just got the same thing. It seems to happen randomly when I try to access the site.


Thank you for the confirmation, Myr.

Logging for user information & tech reference, anyhow.


----------



## uncia2000 (Jan 4, 2006)

And just as suddenly, back to normal(?) again (in the last few minutes).

Still 625+ users, but with ~1 second response time on own home page.



*[ed.]* _(If there are any other serious, prolonged response time issues tonight, please log those here.
I've got to be offline early (4.30am), alas. Best wishes and a good night to y'all )._


----------



## Almafeta (Jan 5, 2006)

I got a 30-second load time about 30 minutes ago.  I don't know if that's 'extreme' or not, but it was definately unusual.


----------



## uncia2000 (Jan 5, 2006)

Early evening note, today... (For reference/update).

Only ~525 users at present, but response times on own home page for the past 10-15 minutes or so fluctuating wildly between 1-2 seconds and 45-120 seconds with occasional "MySql ERROR!"s.


----------



## uncia2000 (Jan 5, 2006)

(And back to "normal" yet again. ~550 users with sub-1 second response times... although followed by board timeouts, instead).


----------



## uncia2000 (Jan 6, 2006)

Currently no (= zero) response on FA here in UK, or in Poland, for the past 20 minutes or so.
Seems to be working OK for at least one person in the US.

Leaving this note, anyhow, for reference/information; gotta go sleep, sry ^^

Please update, if required.


----------



## Gabe (Jan 6, 2006)

uncia2000 said:
			
		

> Currently no (= zero) response on FA here in UK, or in Poland, for the past 20 minutes or so.
> Seems to be working OK for at least one person in the US.
> 
> Leaving this note, anyhow, for reference/information; gotta go sleep, sry ^^
> ...



I think this subject belongs to this thread...and I hope its not just me, but these extreme slowdowns are even slower! Now nothing except the front page will open, when you click something, it'll load for around a minute before the loading times just cuts off and you remain on the front page!
Strange indeedie.


----------



## uncia2000 (Jan 6, 2006)

(fyi) Scheduled outage noted on FA front screen five minutes ago, prior to current "MySQL errors".

Should hopefully be back soon.

Regards.


----------



## uncia2000 (Jan 6, 2006)

And 'tis back... relatively painless that! 

Apologies for the "blip".


===
*[ed.]* Reponse = unable to send/"read only mode" when I try to reply to a note, but is not stated as such on the front page.
And can't see any new submissions posted since that brief outage.
This is affecting all users.

Fyi, only, for now; stated to have been applying a cookie fix, during that outage.

*{latest ping/update: 7:58PM PST/10:58PM EST/3:58AM GMT}*

=

IRC ( irc://irc.furnet.org/furaffinity ) for any "live" news/discussion.


----------



## uncia2000 (Jan 6, 2006)

FA is now properly back (/was back 5-10 minutes ago, but had to wait to see other submissions/notes going through AOK.

With any luck we might be having a few less cookie issues, too. _*points also to front page*_

Best wishes & Have a good weekend


----------



## uncia2000 (Jan 8, 2006)

No logged slowdown yesterday 

=

Logging for today (fyi):

Very slow; 30-60 second response time on my own user page for the last 10 minutes or so.


----------



## uncia2000 (Jan 8, 2006)

Seems to be back to normal again: response time after previous post varied 8-10 seconds (border-line usable) through 30+ seconds, again.
(Total duration 15-20 minutes?)

No MySQL errors observed.


----------



## uncia2000 (Jan 8, 2006)

Hanging for the last few minutes, back to "MySql ERROR!" briefly, followed by 8-10 second response for a minute or so, then to normal again.

_(Relatively minor impact, this time, but would still be interested to know if we're doing scheduled or unscheduled maint. during near-"peak time")_


----------



## uncia2000 (Jan 9, 2006)

Occasional slowdowns to 45+ second response time for >5 minutes, followed by more MySQL errors.


----------



## Dragoneer (Jan 9, 2006)

uncia2000 said:
			
		

> Occasional slowdowns to 45+ second response time for >5 minutes, followed by more MySQL errors.


According to Jheryn, the slowdown and SQL errors which affect the main page are due to too many people logging on at the same time. It's a server setting we should be able to fix.


----------



## uncia2000 (Jan 9, 2006)

Dragoneer said:
			
		

> uncia2000 said:
> 
> 
> 
> ...


Good, good!
I obviously couldn't tell during those shorter periods of slowdown how many users were actually "active" concurrently or otherwise attempting to fight over shared resources. In terms of overall (= recent) user count, we were still only running at ~600 both times, tonight.

Those longer (15-20-30 minute) periods of slowdown still _feel_ like a different root-cause to me, but one step at a time, I guess.
(Unless there's anything obvious in the system logs for the durations of those longer slowdowns noted previously).

Thanks for passing that on and my regards to Jheryn.


----------



## uncia2000 (Jan 9, 2006)

(retrospective note: 10-20 minutes ago; slowdown for 5-10 minutes w/response time 40-120 seconds; per. "short" slowdowns previously. OK again, now)


----------



## uncia2000 (Jan 9, 2006)

(and again: 5-10 minutes ago; very brief slowdown w/response time 45-90 seconds, followed by longer than usual "MySQL error" outage (several minutes long), then almost immediately back to normal; otherwise per. "short" slowdowns previously. OK again, now. ~600 users both before and afterwards)

(also brief outages on the Forum somewhat before, and a few minutes after the above)


----------



## uncia2000 (Jan 10, 2006)

(And just caught the tail-end of another, relatively (very?) short slow-down + MySQL error outage; went into it with 700-720 "active users" and emerged at ~600).

(Each of the above "relatively" low impact- although still noted by users- but _perhaps_ a bit more frequent than before. No longer period (20-40 minute) slowdowns noted tonight).


----------



## uncia2000 (Jan 10, 2006)

System lockdown at present...
"Please wait and refresh the page once in a while as we should be back online soon".

*[ed.]* (Relatively brief outage; back now. Please report any observed issues in the usual manner. Thank you).


----------



## uncia2000 (Jan 10, 2006)

Running generally slow (up to ~30 second response on my home page), from time-to-time over past 10-15 minutes. Currently only at ~650 users vs. ~725, earlier.

Anyone else observing same? (And for how long?)


*[ed.]* Better again, now. Not sure how long that lasted, all told (15 minutes?) and wasn't consistently slow, either.
Fyi/other observations, anyhow.


----------



## Myr (Jan 10, 2006)

I get database errors right now. I cannot access the site. Just a white screen error.

The site has been slow or unresponsive all day long for me.


----------



## uncia2000 (Jan 10, 2006)

Thanks, Myr! 


			
				AuroranDragon said:
			
		

> I get database errors right now. I cannot access the site. Just a white screen error.
> 
> The site has been slow or unresponsive all day long for me.


All day? Certainly hasn't been _that_ bad here, although it's going through another slow period just now 

Dragoneer confirms that the problem is due to hitting max connections and is awaiting a fix from Gushi.

(I'm still not 100% certain that accounts for _all_ slowdown issues, but will get that fix first...).


----------



## Myr (Jan 10, 2006)

uncia2000 said:
			
		

> All day? Certainly hasn't been _that_ bad here, although it's going through another slow period just now
> 
> Dragoneer confirms that the problem is due to hitting max connections and is awaiting a fix from Gushi.
> 
> (I'm still not 100% certain that accounts for _all_ slowdown issues, but will get that fix first...).


Yep, every time I've gone to access the site today it's been very sluggish to respond. I've only used it off and on though.

But the max connections is also a good thing since we know the site is popular.


----------



## uncia2000 (Jan 10, 2006)

AuroranDragon said:
			
		

> Yep, every time I've gone to access the site today it's been very sluggish to respond. I've only used it off and on though.


Hmm... unlucky? :?

Most of the time I'd been getting 2-5 seconds responses on my own user page, but there have been a larger number of more extended slow downs (really doesn't seem to tally with the number of users shown as being "online").



			
				AuroranDragon said:
			
		

> But the max connections is also a good thing since we know the site is popular.


I'll still be a lot happier when that's fixed: especially if cures the vast majority of the apparent slowdown/outage issues.

Problems like that tend to start small and grow gradually until they suddenly hit very hard. (Like that one we had when the backups ate our hard drive!). Still requires monitoring, even if "gentle prodding" is probably no longer required! 

Will look out for updates on that (soon?), anyhow.


----------



## uncia2000 (Jan 10, 2006)

Very slow (60-90 seconds on home page) again, last 30 minutes+, but haven't seen any MySQL errors.

=

question/observation aside:

Am still not sure about the significance of the number of "active users"  (counted how?) displayed at the top of the screen vs. the actual number of connections being held.

The way that there still appears to be an extended slowdown now, when there were only 550-650 users vs. the 725 and running "relatively" smoothly earlier might appear to indicate that connections are failing to be released properly? :?
Compare with the situation where there's a MySQL error (in parallel with a total refresh of connections?) and the system runs fast immediatlely afterwards.

_(Just a thought, anyhow...)._

=

_(and if so, someone please consider hitting the reset button, even if that's only a "short term solution")_


----------



## uncia2000 (Jan 11, 2006)

Extended and severe slowdown at present; please await further update or check with IRC ( irc://irc.furnet.org/furaffinity ) for any information as available/ compare with other users.

Apologies for this.


_note @ Admins/techs : feel free to edit this as required, since I'll be asleep..._


----------



## uncia2000 (Jan 11, 2006)

Total duration of severe slowdown/outage last night (per above) was ~4 hours.

I'm presuming that our techs are currently looking to ensure this doesn't recur again around peak time tonight.
More information when available.

Thank you, and appreciate your further patience should this recur (e.g. please don't hit "enter" every few seconds in the hope that it's back working again).
Please also consider bookmarking the Forums or IRC channel as neither of these are affected were affected by these problems, yesterday.


----------



## uncia2000 (Jan 12, 2006)

Repeated and prolonged slowdowns again, tonight.
Have not been logging any specific times, owing to their general nature, but your patience is again very much appreciated.

Will be sure to forward any update re. resolution when that's available. Thank you.


----------



## uncia2000 (Jan 13, 2006)

* Adds tentative note.


System is possibly running better tonight under high user count (~750+) and concurrent connection load around peak time.

Was going through some longer slow-downs earlier, but has more recently been relatively good at ~780-800 users. (At present perhaps varying up to ~15 seconds on my user home page, but down to 4-9 on the following refresh. Not constantly "stuck" at 45-90+...).

Still not "ideal", perhaps, but presume this is being monitored.

Thanks & will post any updates as/when available.


----------



## uncia2000 (Jan 13, 2006)

[update from IRC via Dragoneer, fyi:]

The root cause for the recent slowdowns around peak time is apparently owing to maxxing out RAM and use of swapfiles on the server's HDD.
At current load levels, the issue should resolve itself when FA is moved from the backup server to the primary server.
*
(Unsure as to temporary measures, at present, although the system has been performing better tonight, as above).


----------

