# Still more site lag and slowness



## Erethzium (Feb 12, 2013)

So yeah, even after the major slowness was fixed, the original problem is still persisting: From time to time, the site will just slow to a crawl for around ~40 seconds, loading pages incredibly slow (page load times of 20 seconds or more) for the duration, but then go right back to being fast again.

This happens again and again all day, no matter the time. Whether it be 7am, noon, 4-6pm, or 1am, it does this all the time.

As far as I can remember, it's been happening since last October or November.

Any clue what's causing it?


----------



## CrazyLee (Feb 18, 2013)

I'm curious to why the forums are laggy at times. I know about the main site lag, but I figured the forums were on a different server system.


----------



## Erethzium (Feb 19, 2013)

...Any admin insight to this? It's still happening.


----------



## Catilda Lily (Feb 19, 2013)

I'm not having any problems.


----------



## PheagleAdler (Feb 21, 2013)

I've gotten 503 errors on the main site occasionally when I try to open multiple pages at a time.


----------



## BlueStreak98 (Feb 21, 2013)

It's just poking along for me right now. I can go get a drink between page loads.


----------



## MRGamer01 (Feb 21, 2013)

So far, it's not as bad as it was a few weeks ago.  Page generations at .453 seconds, which is a lie compared to the actual 7 or so seconds it takes now.  Only at peek hours now it seems.


----------



## kayfox (Feb 23, 2013)

ShadowFur said:


> Yes, But what I am pointing out this is before it got real bad. So Something is going on. Just because Its lower than a few weeks ago doesn't mean its fine and dandy or something is not happening. If we are getting errors so often something is wrong. Errors don't just pop out of no where. Yes you may get errors on your home pc, but servers are designed for I/O ang heavy work.  ...



Im not sure how what you wrote is supposed to convey your premise.  But I can assure you that things can go terribly wrong without any actual hardware or even software failure.

It is obvious that the database server is having issues, this can be entirely caused by table sizes in respect to RAM.  Say they have a table that is too big for RAM, such as the notifications table.  This table will have to have all its writes flushed when they happen.  This means anything in the path those writes takes can slow it down to a crawl.  A slow RAID controller, single threaded filesystem, to little buffering, too much buffering, each of these can have an effect on the writes.  The problem here is that until it writes all of that data, it cant read any of it back, so everything waits because every time you hit any page while logged in it hits the notifications table.

Also, Apache is not badly written, and does not matter here, if you hadn't noticed nginx in those error messages.


----------



## Erethzium (Feb 24, 2013)

MRGamer01 said:


> So far, it's not as bad as it was a few weeks ago.  Page generations at .453 seconds, which is a lie compared to the actual 7 or so seconds it takes now.  Only at peek hours now it seems.



I guess 1am EST (10pm PDT) is a "peak hour" for FurAffinity, then, because even that late at night, I still get these huge lag spikes.


----------



## Erethzium (Feb 24, 2013)

Just now, pages taking 10-15 seconds to load, as in...sitting there "loading" for 10-15 seconds, and then everything appears instantly.












Any insight yet, Admins?


----------



## kayfox (Feb 24, 2013)

I bet the issues the site is having right now are not generating any real useful log messages.


----------



## Calemeyr (Feb 24, 2013)

It's an old site with old hardware, so it's going to be slow. It's not like there's a crack team of well-paid coders and computer engineers working 24/7 to fix all the issues. FA support is made up of volunteers who have lives outside of FA (hopefully). Still, the site is pretty slow at times. And yes, the forums are starting to be slow as well. It's all been getting slower compared to when I joined 2/3 years ago.


----------



## Erethzium (Mar 6, 2013)

Soooo...any word on this from the admins? It's still happening constantly all day, every day.


----------



## Erethzium (Mar 11, 2013)

Seems to be getting worse and worse lately. Like it's laggy and slow more often than normal speed. And now the forums are laggy as well.

What's going on with the servers?


----------



## CrazyLee (Mar 11, 2013)

Hmm. The forums do seem laggy today, as in when I click on a link nothing happens (but firefox says it's getting the page), and then all of a sudden after a 20-30 second delay, the entire page just pops into view.


----------



## Armaetus (Mar 11, 2013)

Even new hardware it takes a while to load but when it does it is very quick. Not even the forums are that laggy like they were last week.

Adding new hardware to a really old site is equal to giving an 85 year old some organs from a 25 year old.


----------



## Erethzium (Mar 23, 2013)

So...any word from the admins on when the severe slowness is getting fixed? Still happening multiple times per day.


----------



## MRGamer01 (Mar 26, 2013)

Marcus Stormchaser said:


> It's an old site with old hardware, so it's going to be slow. It's not like there's a crack team of well-paid coders and computer engineers working 24/7 to fix all the issues. FA support is made up of volunteers who have lives outside of FA (hopefully). Still, the site is pretty slow at times. And yes, the forums are starting to be slow as well. It's all been getting slower compared to when I joined 2/3 years ago.



In a way, "crack team of coders" sounds like a set up for the A-Team, just with computers and willingness to work with furries so they can have their art and porn.


----------



## Erethzium (Apr 7, 2013)

Still waiting for something to be done about this.


----------



## idejtauren (Apr 10, 2013)

It's currently just after 1pm EST...and FA does not load at all.
Sending a note gave me 504 Gateway Time-Out after about 5 minutes of trying to load.


----------



## Erethzium (Apr 24, 2013)

Soo...anything being done about this? Still happening all day every day.


----------



## Erethzium (May 5, 2013)

Still happening...admins still doing nothing about it.


----------



## MRGamer01 (May 6, 2013)

Erethzium said:


> Still happening...admins still doing nothing about it.



At this point I'd also like to get a sort of status update on the new server hardware.  Wasn't the donation drive about two months or so ago?


----------



## kayfox (May 11, 2013)

Yak said:


> Donwtime caused by the database server running out of swap space and having its main process killed by the OOM manager.
> This is unexpected. The server was configured to be well within the safe zone and never use up more RAM than it has available.
> Guess is, due to the growing amount of data the search server that shares the same hardware was starting to use up more and more memory over time.



I just have to say this... if this was unexpected, your doing it wrong, let me enumerate why:

1. Given a long enough timeline, you always run out of RAM and Swap.
2. You should have monitoring setup, both generating emails on the unit, and off the unit.  Let me enumerate:
 - SNMP should be monitored, if certain things reach certain thresholds--which would depend on how fast you respond to them--it should send emails. Example: Swap more than 80% utilized.
 - The OOM killer on the box should generate a trap and syslog messages that should both generate emails.
 - Syslog should be configured to log to a off-box syslog *in addition to* logging to /var/log
 - It would probably be a good idea to have a perl script or something that runs every so often and logs critical system variables, such as mem used, buffers+cache, swap used, load average, etc.  This should be configured with thresholds to change severity when crossed.  Above a certain threshold emails should be generated.
 - Pro-actively check in on the machines on a regular basis.
3. You should be troubleshooting the slowdowns, this would have meant that the memory issue would have gotten caught sooner and before OOM fired.
 - Slow query log.


----------



## Armaetus (May 11, 2013)

Hmmm, there is no status update in the appropriate forum for today..

I hope this doesn't mean there will be another donation drive :V


----------



## PheagleAdler (May 11, 2013)

yeah, a warning would have been nice.


----------



## marmelmm (May 11, 2013)

http://s289.photobucket.com/user/marmelmm/media/Drawings/Affinitytruth.jpg.html


----------



## Armaetus (May 12, 2013)

Even after the issue was resolved, there is still no update in that forum. Meh.

Why is the date May 5th when the issues happened May 11th/12th?


----------

