# Heavily degrading mainsite response. (server processing?)



## 2ndVenus (Jan 10, 2012)

Helloooo

I dont often come to the forums but i feel i have to at this point to express my concern, and information is always good input for those who want to keep things up tight with their servers and performance.

To sum it all up:
The response times between requesting a page - to - recieving the page is growing and growing. Download speeds on FA are absolutely fine, but within the last few months, the sites Time-To-Respond has grown and grown.


Download speeds are fine.
Time before recieving a page > very large now. Normally between 15 seconds to 40 seconds, when previously waits were no longer than 3 seconds.

After much checking of my network, ports, firewall, EVERYTHING, i can confirm the problem is not on my end. I have performed all i can.

I feel i am better telling you about it, than not, so, there ya go!


----------



## MRGamer01 (Jan 10, 2012)

This is nothing new.  I've seen the site get better one day for about ten minutes, maybe less, then it went right back to being slow.  Theres a thread somewhere in the support section, but not sure it's still alive.

Edit: Ok though today it seems generally okay, hopefully it'll stay that way.


----------



## Devious Bane (Jan 10, 2012)

It's because of people constantly attempting to DDoS the site, we're actually late for our monthly server crash.


----------



## Erethzium (Jan 10, 2012)

Yeah, I've noticed this too, FA is on-and-off extremely slow. It's not my internet, every other website loads just fine, but a lot of the time, FA will take upwards of 15 seconds to load each page.


----------



## MitchZer0 (Jan 10, 2012)

Maybe it's another bee in the server room, except this time he brought company.

And this time, they're camping in.


----------



## MRGamer01 (Jan 11, 2012)

MitchZer0 said:


> Maybe it's another bee in the server room, except this time he brought company.
> 
> And this time, they're camping in.



I think I just found my new signature.


----------



## Trpdwarf (Jan 11, 2012)

Hello there, I do apologize for the issues with site degradation. Currently the site is having issues with abusive scrapers that are causing site disruption. We are monitoring the situation.

EDIT: Four abusive scrapers just got kicked from the system and it should restore site functionality a bit.


----------



## thoron (Jan 11, 2012)

Trpdwarf said:


> Hello there, I do apologize for the issues with site degradation. Currently the site is having issues with abusive scrapers that are causing site disruption. We are monitoring the situation.
> 
> EDIT: Four abusive scrapers just got kicked from the system and it should restore site functionality a bit.



Abusive scrapers?


----------



## Pinkuh (Jan 11, 2012)

thoron said:


> Abusive scrapers?



Folks that are using crawlers to go through and save as much shit on FA as possible. Specifically images. It really bogs everything down.


----------



## Accountability (Jan 12, 2012)

And that's been going on for _days_? Given FA's track record of obsessively monitoring that stuff, I'm quite surprised that four scrapers were allowed to run for days unchecked.


----------



## CerbrusNL (Jan 12, 2012)

Accountability said:


> And that's been going on for _days_? Given FA's track record of obsessively monitoring that stuff, I'm quite surprised that four scrapers were allowed to run for days unchecked.



Just a tip:
You -could- also have worded that something like this:
"While that took a while, at least the scrapers are gone, right?"
Share the love, Accountability. That passive-aggressive posting ain't really motivating. We're working on this site in our spare time, a little appreciation would be appreciated.
Besides, track record? Care to elaborate?


----------



## sandfox (Jan 12, 2012)

im amazed that admins have the spare time to tell users how to word their posts.


----------



## CerbrusNL (Jan 12, 2012)

sandfox said:


> im amazed that admins have the spare time to tell users how to word their posts.


Because we're not allowed to do anything besides administrate FA.
Yea, if you're going to to criticise us, at least try to do a good job. [/personal opinion]


----------



## Trpdwarf (Jan 12, 2012)

thoron said:


> Abusive scrapers?



Assuming you know what scrapers are, these are cases where people are using it in such a way that it caused really bad site degradation.



Accountability said:


> And that's been going on for _days_?  Given FA's track record of obsessively monitoring that stuff, I'm quite  surprised that four scrapers were allowed to run for days unchecked.



To clarify these "Four scrapers" were not allowed to run for days. They were caught recently and removed but in the course of the week they are not the only ones that were caught. It is a bit of an issue right so we are monitoring the site and as these things we pop up we try to remove them ASAP because it's a great inconvenience to everyone.


----------



## thoron (Jan 12, 2012)

Trpdwarf said:


> Assuming you know what scrapers are, these are cases where people are using it in such a way that it caused really bad site degradation.



I initially thought an abusive scrapper was someone who was uploading a huge number of images and files to scraps.


----------



## CerbrusNL (Jan 12, 2012)

thoron said:


> I initially thought an abusive scrap*p*er was someone who was uploading a huge number of images and files to scraps.



[J. Hyneman / A. Savage] Well there's your problem! [/J. Hyneman / A. Savage]


----------



## SkieFire (Jan 14, 2012)

Actually it was sentient brillo pads.

And it wasn't 4 days before anything was done (you may have noticed the site wasn't slow for an entire 4 days, just odd periods). Each instance of slowness was someone scraping for all they were worth until they got nuked. Not all of the mods/admins have the access required to deal with this sort of thing, but once we notice the site being funny we are pretty quick to prod those with the power.


----------



## MRGamer01 (Jan 15, 2012)

I assume there was just another scraper or more a few moments ago?  Site seemed pretty slow to me.


----------



## 2ndVenus (Jan 16, 2012)

Download speeds are fine, its just it takes a long time before i get a response and recieve a page. DDoS attempts?

Its still bad right now*

* London time 8:30pm


----------



## 2ndVenus (Jan 17, 2012)

New post: 



			
				Yak said:
			
		

> Code:
> 
> Dec 19 03:25:40 novastorm kernel: swap_pager_getswapspace(16): failed
> Dec 19 03:25:52 novastorm kernel: pid 90279 (mysqld), uid 88, was killed: out of swap space



Is this linked to this problem? The site has been helluva sluggish the last week or so. I feel this is the worsened state of the same problem.


----------



## Devious Bane (Jan 17, 2012)

FA's poor coding does have a lot to do with it.


----------



## CerbrusNL (Jan 18, 2012)

Devious Bane said:


> FA's poor coding does have a lot to do with it.



So it was the codebase after all! [/Scooby-doo]
We didn't see that one coming!

Seriously, though, there's work being done on that. The problem with a site's coding is, though, that you can't see it when it's changed.


----------



## ShadowWalker (Jan 25, 2012)

CerbrusNL said:


> [J. Hyneman / A. Savage] Well there's your problem! [/J. Hyneman / A. Savage]



Are the admins so bored and out of anything to do that they are having to correct peoples spelling while spouting out gibberish?


----------



## Corto (Jan 25, 2012)

Jesus titty Christ, out of all criticism this site has received, good or bad, constructive or harmful, well or ill intended, deserved or undeserved, I think "omg admins are posting in the forums! How dare they waste those 2 and a half seconds" is the stupidest one I've seen yet.


----------



## ShadowWalker (Jan 25, 2012)

Corto said:


> Jesus titty Christ, out of all criticism this site has received, good or bad, constructive or harmful, well or ill intended, deserved or undeserved, I think "omg admins are posting in the forums! How dare they waste those 2 and a half seconds" is the stupidest one I've seen yet.



You idiot. So there's nothing wrong with an admin wasting time posting nonsense is stupid? I suggest you stop kissing ass and start thinking for yourself.


----------



## Accountability (Jan 25, 2012)

ShadowWalker said:


> You idiot. So there's nothing wrong with an admin wasting time posting nonsense is stupid? I suggest you stop kissing ass and start thinking for yourself.



There's a difference between admins like Cerbrus and the admins who work on the technical side of things that could actually fix the problem being discussed. 

If Yak was in here correcting spelling, then _that_ would be wasting time.


----------



## Janglur (Jan 25, 2012)

Yak wouldn't correct spelling, he'd just throw people into a woodchipper.


----------



## Arshes Nei (Jan 25, 2012)

ShadowWalker said:


> You idiot. So there's nothing wrong with an admin wasting time posting nonsense is stupid? I suggest you stop kissing ass and start thinking for yourself.



Just calm down and evaluate. As it's been said not every admin here works on the technical side. I'd expect your kind of response from a teenager, not a person who is nearing his thirties. :/


----------



## Corto (Jan 26, 2012)

ShadowWalker said:


> You idiot. So there's nothing wrong with an admin wasting time posting nonsense is stupid? I suggest you stop kissing ass and start thinking for yourself.


No way, kissing ass is my highest priority. I dream of becoming a mod one day.


----------



## ShadowWalker (Jan 26, 2012)

Arshes Nei said:


> Just calm down and evaluate. As it's been said not every admin here works on the technical side. I'd expect your kind of response from a teenager, not a person who is nearing his thirties. :/



Technical? I didn't say any thing about admins having to do anything with the technical side of things. My point was that the post (http://forums.furaffinity.net/threa...processing-)?p=2814281&viewfull=1#post2814281) had no point to it at all. If you don't agree then tell me what the point of the post was. 

Oh? So at 29 I'm not allowed to have an opinion or express it how I wish?


----------



## Devious Bane (Jan 26, 2012)

_*Evaluate.*_
Cerbrus is a support admin, not a tech admin. His time is better spent (asskissing) moderating typos then trying to work on something he knows nothing about.


----------



## timoran (Jan 27, 2012)

The admins delete/infract regular users' "pointless" posts, and then make their own. It reeks of hypocrisy.


----------



## 2ndVenus (Feb 1, 2012)

Howdy folks! Back on topic,  hows things looking?

Iv noticed a lot of improvement over the last week, until tonight, its gone back to where it is difficult to even Fav someones work. Keepin ya up to date  ciao!


----------



## theguywiththecamera (Feb 2, 2012)

The performance issue has been acknowledged main site for a bit now and I was wondering if any of the tech-people found anything. It seems to come and go and I remember seeing a mod in the forums mention some people "abusive scraping", is that still the case? It just doesn't seem to have any reason behind it, click a submission and it loads fine, then click fav or leave a comment a few seconds later and it takes half a minute to regenerate the page. 

Personally, I find the fickleness more annoying than just slow performance, at least you can anticipate and accommodate for the slowness. More curious than anything really.


----------



## Devious Bane (Feb 2, 2012)

timoran said:


> The admins delete/infract regular users' "pointless" posts, and then make their own. It reeks of hypocrisy.



*(USER WAS BANNED FOR THIS POST)*


----------



## MRGamer01 (Feb 3, 2012)

Question here.  Has it already been passed the week they said there would be unplanned maintenance?  Or was this a typo for unplanned maintenance in the coming week"s"?


----------



## SkieFire (Feb 3, 2012)

There should be an announcement about maintenance shortly, there will be downtime. I dont want to step on Yaks toes (and I dont know 100% for sure exactly what will be done or what it will fix) so I'll leave it at that.


----------



## Devious Bane (Feb 3, 2012)

MRGamer01 said:


> Question here.  Has it already been passed the week they said there would be unplanned maintenance?  Or was this a typo for unplanned maintenance in the coming week"s"?


Don't expect them to get it right, let alone get it right the first time.


----------



## MRGamer01 (Feb 3, 2012)

Devious Bane said:


> Don't expect them to get it right, let alone get it right the first time.



Done and done.


----------



## Manacat (Feb 5, 2012)

I just want to say yeah I've noticed it (especially right now) but I wish whoever working on it luck in fixing it.  It always amazes me the extreme rudeness I see when things like this happen to a site that is free to use. I can only imagine how rude some people get about things they actually pay for.

Thanks for working on this site and helping to keep it running!  I apologize if I'm just adding to spam here but I just wanted to add something more positive after reading so much negative feedback.


----------



## Devious Bane (Feb 6, 2012)

It's never a matter of luck.
Just a matter of finding the right people who can get the job done actually cares about the problems with the site.


----------



## Accountability (Feb 6, 2012)

Devious Bane said:


> It's never a matter of luck.
> Just a matter of finding the right people who can get the job done actually cares about the problems with the site.



Pretty much this. We've been waiting for "unscheduled maintenance in the upcoming week" for like two weeks now. It's obviously not a priority for the technical staff or it would have been scheduled and fixed already.

It's also a matter of finding the right people that won't treat a site with 5 million plus submissions and enough members to fill a small country like it's a hobby project that doesn't require constant attention.


----------



## MRGamer01 (Feb 6, 2012)

Manacat said:


> I just want to say yeah I've noticed it (especially right now) but I wish whoever working on it luck in fixing it.  It always amazes me the extreme rudeness I see when things like this happen to a site that is free to use. I can only imagine how rude some people get about things they actually pay for.
> 
> Thanks for working on this site and helping to keep it running!  I apologize if I'm just adding to spam here but I just wanted to add something more positive after reading so much negative feedback.




You'll see soon enough...


----------



## Soline (Feb 6, 2012)

Right, no idea if it's supposed to go here or make a new thread, but hey, I got nothing better to do while repeatedly refreshing.

I'm getting Error 503 messages -constantly-. And have been for the past month or so, it's worse at night GMT-0 time, but appears to actually be getting worse (opened 10 pictures today, ALL 503'ed, last week only half would have)


Oh, and Cerberus, you realise that if the coding is so bad is causes notable issues with the site, then fixing that coding _will_ be visible? namely in seeing the site behave as it should.


P.S I though Abusive Scrapers were some kind of bug living in the hardware


----------



## SkieFire (Feb 6, 2012)

Feb 11-12th for ze outage. (this weekend)


----------



## Accountability (Feb 6, 2012)

SkieFire said:


> Feb 11-12th for ze outage. (this weekend)



Is it a two day outage or have two weeks of planning only narrowed it down to "sometime over the weekend"?


----------



## Devious Bane (Feb 7, 2012)

The longer the better.
(Inb4RAIDdies)


----------



## yak (Feb 7, 2012)

Soline said:


> I'm getting Error 503 messages -constantly-.


Are you sharing an internet connection with somebody else who might be using FA at the same time? Or using a proxy? Who is your Internet Service Provider?


----------



## quoting_mungo (Feb 7, 2012)

Yeah, Yak, when submissions in the inbox come in 6-image rows, maybe limiting simultaneous connections to half that wasn't the best idea ever? I can understand wanting to cut down on the connections as a quickfix for people abusing the site, but I really don't think wanting to be able to open a full row of images at once with snaplinks is too much to ask.

Will this be fixed with/after the maintenance?


----------



## lostcat461 (Feb 7, 2012)

Or you could just make your browser skinnier so that your submissions line up in three row columns instead. (and no errors on this end, except minor slowdowns at what I assume is peak times. Kentucky must naturally just be amazing.)


----------



## yak (Feb 7, 2012)

quoting_mungo said:


> Yeah, Yak, when submissions in the inbox come in 6-image rows, maybe limiting simultaneous connections to half that wasn't the best idea ever? I can understand wanting to cut down on the connections as a quickfix for people abusing the site, but I really don't think wanting to be able to open a full row of images at once with snaplinks is too much to ask.
> 
> Will this be fixed with/after the maintenance?


The limit was set for requests to dynamically generated pages alone. No such limit is set for the static file server.


----------



## Soline (Feb 7, 2012)

I'm on a JANET access point, it's a 100mbps academic network used by several thousand people.

However, I've been in the same location, with the same wires, the same network and the same number of people have been accessing FA (I assume) for the past 5 months at least, this issue has only arisen over the past 2, tops. I see what you're suggesting, but the problem is nothing has changed at my end sofaras I can tell, I access FA exactly the same way I always do (In general, opening about a dozen files in the 'new submission' box, viewing them, closing them and repeating) And, unless a hundred others have suddenly discovered FA I can't see the limit for the JANET netwrok being an issue (Hell, I -think- I actually have my own IP, so I'm not sure if others make any difference at all)


----------



## yak (Feb 8, 2012)

I see.
Problem is that request limiting is being done on a per-IP basis, so when you have several people behind one public facing IP the limit is shared between them. Previously there has been such limit in place, and for a long time; at some point it was disabled altogether in good faith that no, or at least little amount of people would be abusing our resources to an extent where it starts impairing normal site operation. 
Lately there has been an influx of people who are mass-downloading content from FA; loading gallery, browse and scraps pages and scraping them for image links, then downloading the images. This is not a problem for the file server and does not appear to be a problem for our bandwidth - but the application, or more specifically the database is struggling to serve additional thousands of requests per minute for each of these downloaders along side what it already serves normally. 
Due to the nature of the requests generated by those scrapers - going way back in history to the oldest posts on farthest gallery pages - and a frustrating, half-a-decade year old MySQLs query optimizer bug related to `LIMIT $large_number_here, 60` the performance of the database server becomes severely impacted and it shows on how slow FA is performing.

So the rate limiting has been restored. Initially it was set to a max. of 3 requests from the same IP in parallel, now increased to 10 after this thread. 
We will be replacing our database software during the maintenance allotted for these weekends; which will allow us additional performance optimization options; as well as trying alternative approaches to large offset pagination.
I expect the limit will then be either removed, or raised to even higher values and left as a failsafe.


----------



## LizardKing (Feb 8, 2012)

yak said:


> We will be replacing our database software during the maintenance allotted for these weekends



Replacing what with what, exactly?


----------



## Devious Bane (Feb 8, 2012)

LizardKing said:


> Replacing what with what, exactly?



Everything with anything.


----------



## yak (Feb 9, 2012)

LizardKing said:


> Replacing what with what, exactly?


With Percona.


----------



## kayfox (Feb 9, 2012)

> <yak[away]> Pi: scrapers load gallery pages with large numbers, very fast and sometimes in parallel; with pagination in mind this generates sql queries with `LIMIT $big_number_here, 60`. half a decade old mysql query optimizer bug interprets that as "read $big_number_here rows from disk into the buffer pool, discard them, then read 60 more and return".



I dont see how Percona is going to fix this.  I hate to say it but this is expected behavior with the LIMIT keyword.


----------



## yak (Feb 9, 2012)

kayfox said:


> I dont see how Percona is going to fix this.  I hate to say it but this is expected behavior with the LIMIT keyword.


The upgrade to Percona is something I've been planning on for a while now; it is not directly related to fixing the current situation, although being a performance oriented fork of MySQL with fewer limitations and more tunable parameters - it will provide a passive benefit for hopefully long enough to have the slowdowns problem fixed directly. 
The reason for the switch to Percona is that it allows for easy online binary incremental backups as well as sane replication, load balancing and failover scenarios when used in conjunction with MMM. We are looking into scaling out with a secondary master.


----------



## Cloudchaser (Feb 11, 2012)

The biggest problem for me is repeatedly getting 502 when I try to load http://www.furaffinity.net/controls/journal/ as well as when I try to edit a specific journal entry.  Even though at the same time, I have no problem reading or posting responses to other people's journals.  I've submitted trouble tickets about the problem, but I'm not getting a response, nor is has the problem been fixed.


----------



## Jessica U. Ingmann (Feb 13, 2012)

Hey, guess what?  

The site is experiencing more trouble than it was _before_ the recent updates.  Now I realize that since it's back up and all, there's bound to be a mass influx of people uploading images, journals, and everything like that all at the same time, but it's pretty darn irritating when this is happening while there's a message box that says the site has "significantly improved response times".  I'm experiencing load times of up to 40 seconds per page and even "bad response from server" errors all across the board!  Very disappointing.

But!  Hopefully this is only temporary.  Once people stop bushwhacking the server a bit, it should taper off.  Hopefully!


----------



## wolfbeast (Feb 13, 2012)

OK, I think this would be the right thread for this - just to let you know that currently, trying to browse FA, even just calling up my own user page, takes close to a whole minute to give me the page. It was bad before the downtime, but it just got a hell-of-a-lot worse.
Is this temporary, i.e.: startup sluggishness, or is something really wrong here?


----------



## triage (Feb 13, 2012)

dis shit slow mang


----------



## Shad (Feb 13, 2012)

Give it some time to re-cache everything. It will be slow for a while until it does. I'm guessing it will take anywhere from 12 to 24 hours to fully re-cache, depending on how big the database is.


----------



## Jessica U. Ingmann (Feb 13, 2012)

Seems okay now!  That didn't take long at all!    Now comes the true test: How long is the server going to run fast for?  ;3


----------



## CannonFodder (Feb 13, 2012)

Jessica U. Ingmann said:


> Seems okay now!  That didn't take long at all!    Now comes the true test: How long is the server going to run fast for?  ;3


Until the member base realizes it's back up and begins spamming the shit out of it.


----------



## Kayla (Feb 13, 2012)

CannonFodder said:


> Until the member base realizes it's back up and begins spamming the shit out of it.



Tried uploading not too long ago :T Constant 502 errors.


----------



## RTDragon (Feb 13, 2012)

Cloudchaser said:


> The biggest problem for me is repeatedly getting 502 when I try to load http://www.furaffinity.net/controls/journal/ as well as when I try to edit a specific journal entry.  Even though at the same time, I have no problem reading or posting responses to other people's journals.  I've submitted trouble tickets about the problem, but I'm not getting a response, nor is has the problem been fixed.



Your not the only one it's happening to me as well now. I can't go to my control panel without getting a 502 error.


----------



## Drbigt (Feb 14, 2012)

I have never had the site be as slow as it is today. Images do not load up when I click on them and when I click download, I am never brought to seperate download. I hit download on a mp3 song for instance and literally 40 mins later, I still didn't get the option to save the song.


----------



## yak (Feb 14, 2012)

Jessica U. Ingmann said:


> ...... while there's a message box that says the site has "significantly improved response times" ....


Reading comprehension.
The notification said that updates were performed that would allow us to significantly improve page response time as well as introduce new features down the line. There is a difference.


----------



## Shad (Feb 14, 2012)

yak said:


> Reading comprehension.
> The notification said that updates were performed that would allow us to significantly improve page response time as well as introduce new features down the line. There is a difference.



Here's a suggestion for you, Mr. Snarky: A good portion of the people in this fandumb are either a) idiots, or b) going to understand things only how they WANT to understand them. So why don't you guys make announcements that can't be misinterpreted or, at the *very least*, put more thought into the announcements you post? Like adding in a little blurb like "When the site comes back up it will be slow for a while as the data re-caches, please do not mash your F5 keys as it will not help make the site faster. There may also be some performance issues as we perform some tweaks to the new system." or something of that ilk. This happens every single time there is a "planned" downtime, you would think that you would learn from past experience.


----------



## Pinkuh (Feb 14, 2012)

So I am having issues as well. 502s' and REALLY long load times. 

attempted to open my journal page and this is what I got.

Page generated in 15.993 seconds [ 2.2% PHP, 97.8% SQL ] (229 queries)

I didn't have this problem before the update :/


----------



## Pinkuh (Feb 14, 2012)

YAY The journal thing is fixed!


----------



## LizardKing (Feb 14, 2012)

Pinkuh said:


> Page generated in 15.993 seconds [ 2.2% PHP, 97.8% SQL ] (*229 queries*)



lolwut? What page is this you're opening, exactly?

Edit: Oh. I guess I took a long time replying there.


----------



## Devious Bane (Feb 14, 2012)

All of them.
Though speeds are noticeably getting better as stuff get Cache'd.


----------

