# Packet loss at new site?



## keiko (Dec 9, 2009)

Hi,

I've tested from 2 completely different locations (one from my cable in California, and 1 from my DSL connection in Switzerland) and am seeing about a 20% packet loss at FA's last hop. Anyone else seeing this issue?

MTR below:


```
Packets               Pings
 Host                                                          Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. midas-array.orcas.net                                       0.0%   346    0.2   0.1   0.1   0.9   0.1
 2. ???
 3. te-7-2-ur01.hayward.ca.sfba.comcast.net                     0.0%   346   11.3   9.1   6.6  61.3   4.0
 4. te-0-8-0-6-ar01.sfsutro.ca.sfba.comcast.net                 0.0%   346   11.5  11.5   9.5  32.0   2.5
 5. pos-1-7-0-0-cr01.sanjose.ca.ibone.comcast.net               0.0%   346   13.4  13.3  11.1  36.6   2.5
 6. xe-10-2-0.edge1.SanJose1.Level3.net                         0.0%   346   20.2  16.3  10.6  83.9  11.9
 7. vlan99.csw4.SanJose1.Level3.net                             0.0%   345   15.5  17.5  11.0  37.9   4.9
 8. ae-94-94.ebr4.SanJose1.Level3.net                           0.0%   345   19.5  18.0  11.2  48.2   5.0
 9. ae-2.ebr4.NewYork1.Level3.net                               2.9%   345   99.0 105.9  98.4 205.7   7.4
10. ae-74-74.csw2.NewYork1.Level3.net                           0.0%   345   99.4 107.7  98.7 170.4   6.5
11. ae-71-71.ebr1.NewYork1.Level3.net                           0.0%   345   99.6 101.2  98.3 201.7   5.9
12. ae-3-3.ebr4.Washington1.Level3.net                          0.3%   345  100.9  95.3  87.9 165.1   6.4
13. ae-64-64.csw1.Washington1.Level3.net                        0.0%   345   88.5  93.7  87.6 130.4   5.1
14. ae-13-69.car3.Washington1.Level3.net                        0.0%   345  223.3  99.8  87.9 295.3  31.9
15. level3.cr2.iad1.inforelay.net                               0.0%   345  297.7 105.5  88.1 486.5  48.2
16. cr1.iad2.inforelay.net                                      0.0%   345   90.5  97.8  88.6 366.5  28.6
17. 70.33.186.196                                              18.8%   345  145.0 144.9 140.3 179.6   2.9
```


----------



## CannonFodder (Dec 9, 2009)

I think it's because everyone tried to get on and submit everything all at once.


----------



## keiko (Dec 9, 2009)

CannonFodder said:


> I think it's because everyone tried to get on and submit everything all at once.


This should cause the web server to respond slower - Not packet loss.

BTW - I've seen this issue at lots of colo facilities that use Cisco switches. They set the ports to 100MB/Full duplex (instead of auto negotiate) and the servers are still set to Auto Negotiate. When you connect this together, the Cisco switch will assume 100 MB/Full (as it's configured) and the server will assume 100 MBit/Half duplex, causing the server to see collisions when there are none. This will make it look like the packet loss is getting worse, the more the server is loaded (since the chance of a collision increases).


----------



## DragonTrew (Dec 9, 2009)

Well, another possibility is the fact that the bandwidth is limited, it is a 100Mbps link versus everyone else trying to read/write at the same time... In other words, the link is overloaded, so the border router will drop packets of the queue...


----------



## Vibration (Dec 9, 2009)

Just-ping seems to be showing similar results, from a large number of sources.


----------



## keiko (Dec 9, 2009)

DragonTrew said:


> Well, another possibility is the fact that the bandwidth is limited, it is a 100Mbps link versus everyone else trying to read/write at the same time... In other words, the link is overloaded, so the border router will drop packets of the queue...


FA never had 100 MBit before, and never had this problem before. It should be better now than it was at the previous site, not worse...


----------



## DragonTrew (Dec 9, 2009)

keiko said:


> FA never had 100 MBit before, and never had this problem before. It should be better now than it was at the previous site, not worse...



True, worrying true...

Jut did some tracerout and ping tests: the ip 66.231.176.25 is the last hop before hitting FA address, and pinging it gives me 100% success. now 70.33.186.196 which is FA is having some loss... I did this test from Telefonica's Speedy network here at SÃ£o Paulo - Brazil just in case you're wondering...

Another thing may be on the server side, the webserver could be too busy processing http requests...

I think it is highly improbable that the problem is in the Cisco switch, unless something is configured wrong or something... I often see problems when dealing with VLANs and packet filtering...


----------



## Dragoneer (Dec 9, 2009)

We're monitoring things right now, and we'll be following up with it. We'll give it past the first 48 hours before we follow through and see what's going on.


----------



## yak (Dec 9, 2009)

To clarify.
There is an issue, it's confirmed and the reason for it is known. We are looking into ways of dealing with it in several ways at once.

If you're interested in a reason, just take a guess. It's fairly obvious.


----------



## DragonTrew (Dec 9, 2009)

My guessing list:

Server overload;
Switch config;
Router config;
Cabling;
DoS attack;
Hardware-related thing...

Am I forgetting something? I think so... But I'm too tired to think harder...

Hope you guys can solve the issue. If there is anything we -the community as a hole- can do to help, I'm sure we'll be happy to help (or at least some of us  ).

ps.: sorry for the bad English


----------



## Dragoneer (Dec 9, 2009)

DragonTrew said:


> My guessing list:
> 
> Server overload;
> Switch config;
> ...


Nope. We were using /too much/ bandwidth, and the extra packets were being discarded. We have resolved the issue for now, so... woot!


----------



## Adelio Altomar (Dec 9, 2009)

What the heck's a packet and just what does this all mean?


----------



## WolfoxOkamichan (Dec 9, 2009)

Why is it still slow in my side


----------



## Dragoneer (Dec 9, 2009)

WolfoxOkamichan said:


> Why is it still slow in my side


Because the site is still under significant load.


----------



## john8792 (Dec 9, 2009)

Dragoneer said:


> Nope. We were using /too much/ bandwidth, and the extra packets were being discarded. We have resolved the issue for now, so... woot!


too much bandwidth?!?!

i thought you just got an upgrade to a 100Mbps connection (~12-13MBps)

what you should consider in the future is getting a second connection and a load balancing router, to better cope with tons of furries on the site, and in general speed it up


----------



## WolfoxOkamichan (Dec 9, 2009)

Ah, right. It's midnight over there.


----------



## quentinwolf (Dec 9, 2009)

john8792 said:


> too much bandwidth?!?!
> 
> i thought you just got an upgrade to a 100Mbps connection (~12-13MBps)
> 
> what you should consider in the future is getting a second connection and a load balancing router, to better cope with tons of furries on the site, and in general speed it up



Because a second connection would cost an arm and/or a leg more. 

http://inforelay.com/offers.php

This Month's Special!
100 Mbps Unmetered 	  	$995/month

 If it weren't the special it seems to get 100Mbps it would be roughly $2000. (at $20 per Mbps.)


----------



## john8792 (Dec 10, 2009)

quentinwolf said:


> Because a second connection would cost an arm and/or a leg more.
> 
> http://inforelay.com/offers.php
> 
> ...




$1000/mo for 100Mbps, jeez, that's expensive

if i were to get several fios lines and a router, it still wouldn't be that much

5 20/20 fios lines= $324.75/mo and is prettymuch un-metered


----------



## DragonTrew (Dec 10, 2009)

Adelio Altomar said:


> What the heck's a packet and just what does this all mean?



A packet is like a container, that carries data to and from end systems [like your machine and FA server]. If a packet is lost, it means that data is also lost, happily, web traffic (HTTP) has ways of dealing with such loss (because it runs over TCP, go wiki that to learn more). so nobody [besides network admins] actually sees what is going on UNLESS someone (like our friend keiko) diagnose it.

In other thoughts, I'm happy you guys were able to solve the issue.


----------



## Ainoko (Dec 11, 2009)

Please let us know what the cause of the packet loss was and what you did to solve the problem


----------



## Dragoneer (Dec 11, 2009)

Ainoko said:


> Please let us know what the cause of the packet loss was and what you did to solve the problem


The cause of the packet loss was by the ISP/Provider. Any packets over our limits were being discarded, causing slowdowns. We tweaked our Cisco router to limit all outgoing data from FA to 95% of its full allotment, which alleviated a lot of the pain.

Yak is continuing to tweak the server during primetime, which can still be slow, but that should ease back a bit in the next few days.


----------



## IRsystems (Dec 11, 2009)

Greetings Fur community,

I am a network engineer with InfoRelay, and I came across this thread through my Google alerts : ) . I just wanted to chime in, and if anyone has questions or comments, I would be happy to further clarify a few items. 

Our routers are rate limited for the full potential of the vlan, which ensures our clients received the connection speed they paid for... If you think of a vlan in terms of a sink drain, it helps make better sense of how internet transport works. If you turn both the hot and the cold supplies on full blast at the same time, your sink is not going to drain fast enough and you would have runoff. This is the same idea, except you would experience packet loss (a bottleneck if you will). . . The good news is that we have seen a sustained rate of data transfer at it's maximum potential, but there must be a lot of downloads to catch up on.


Internet transport in a data center is not necessarily the same as Verizon Fios, whereas they use dynamic IP addresses and are not designed for hosting. It also would not be a full duplex internet connection (in and out). The internet bandwidth you would get would also be shared amongst your  neighbors as opposed to a dedicated internet pipe. Verizon residential services would claim 20Mbps, but if you sustained that amount of usage they would absolutely limit your connection further (try running a bit torrent for an extended period of time, you might get a call from your ISP : ) ! 

InfoRelay operates one of the top networks in the country, so once the webserver has a chance to catch up on uploads, you should be experiencing great results here on furaffinity!

Thanks,
Tom


----------



## DragonTrew (Dec 11, 2009)

IRsystems said:


> Greetings Fur community,
> 
> I am a network engineer with InfoRelay, and I came across this thread through my Google alerts : ) . I just wanted to chime in, and if anyone has questions or comments, I would be happy to further clarify a few items.
> 
> ...



Ok, assuming you really are who you claim to be, that is a first here at FA. I've never heard of any member of any previous colo posting explanations on the forum.

Yes, that is true, the internet infrastructure tend to behave that way, it is pretty common. Packets are dropped ever since the internet were born, happily the majority of the applications runs over TCP which has some pretty nice congestion control [the famous congestion avoidance algorithm: duplicate ACKs, timeouts, congestion window, slow start, etc etc]. The thing is: those mechanisms are becoming a bottleneck, the TCP/IP architecture has it's limitations, and we came to it recently when connections started to exceed the gigabit barrier [one of the motives telco companies run SONET, ATM, GMPSL, etc etc.. on their cores]. On other words, we need to re-think the internet infra... That was one of the topics discussed on IEEE Infocom 2009 [which I was able to attend]. The problem is it is expensive to make changes, and companies tend to "dislike" changes.

I don't know about how stuff works on US regarding ISPs, but I've never received any call form my ISP reguarding bandwidth usage...

Once more sorry for the bad English. Take care


----------



## DragonTalon (Dec 11, 2009)

DragonTrew said:


> I don't know about how stuff works on US regarding ISPs, but I've never received any call form my ISP reguarding bandwidth usage...



The big ISPs that have a near monopoly in much of the US don't call you if you run bittorrent too long.

They simply throttle you down to nothing, block your packets or in the past, forge disconnect packets to kill your streams.  Back when the whole mess started they didn't even SAY there was a limit.  If you were throttled or shut off you might be told a limit exists if you asked, but they wouldn't say what it was or how close you were to it.

It's a little better now, but they still are doing the same basic thing.   Selling you a high speed connection and then threatening you if you actually use it.

Go figure.

As for the post from the ISP here, thats pretty good customer service there.


----------



## Bruce (Dec 12, 2009)

I suppose that's why Yak and Dragoneer are trying to throttle the outgoing load at the FA rack router - that would be a soft cushion rather than running into a hard limiter that InfoRelay would have to throw up.

And for everyone with "High Speed Unlimited" service from Verizon FIOS or at&t U-Verse or Time-Warner "Roadrunner" or Comcast...  There ARE Limits - what the big print gives, the fine print takes away.  TANSTAAFL.

  You have to read that fine print, you are supposed to use the line for "residential uses" like web-surfing.  They give you a "generous" limit like 5 GB throughput a month, and then you start setting off alarms at the office.

  The Telco or CableCo doesn't want you running a BitTorrent host streaming free movies, they want to sell you their movies on demand.  They don't want you streaming multiple Skype or Ooma or MagicJack connections, they want to sell you their VOIP phone with the $30 a month national calling package.  They don't want you running a server at home, they want to sell you hosting services.

  It's all about "Average Revenue Per User" - ARPU.   Needs to be kept as high as possible, and by any means necessary.

  You can get throttled for going over the Unlimited Limit, or get a 'Nastygram' that you need to upgrade to "Business Class" service for a lot more money, or just plain shut down for "abuse", no refund, no recourse.  And if they are the only game in town, even if you are telecommuting and you'll get fired without a connection  - "Too bad, So sad.  Take your laptop and go work at Starbucks, they have WiFi."

This is why I went with DSLExtreme for DSL - Unlike the Telco or CableCo, when they say "Unlimited" they mean it.  ;-)  Only limit I have is the almost 20K FT of cable back to my CO, so I'm barely able to hold base-rate 384K.  But that can be overcome.

  I might have to buy a pair of Bashlins and get back in practice on my hooks (Trained Construction Splicer & COE Installer) and go hang a few repeaters back to the CO, been a few years but nothing I can't handle...  Go ahead, Dare me.  :evil:

--<< Bruce >>--

PS:  Ahh, there are hidden smileys.  Let's try another one that fits...  :lol:


----------



## Carenath (Dec 12, 2009)

Bruce said:


> I suppose that's why Yak and Dragoneer are trying to throttle the outgoing load at the FA rack router - that would be a soft cushion rather than running into a hard limiter that InfoRelay would have to throw up.
> 
> And for everyone with "High Speed Unlimited" service from Verizon FIOS or at&t U-Verse or Time-Warner "Roadrunner" or Comcast...  There ARE Limits - what the big print gives, the fine print takes away.  TANSTAAFL.
> 
> ...


True this, and it is more or less how ISPs around the world operate.
Regarding DSL.. this is exactly why I like the Cable infrastructure.. no distance limitations.. but at the price of shared bandwidth, variable and sometimes high contention ratios. I'd love a 1Gbps fibre connection and a /28 but that's not gonna happen anytime soon until I end up with a 7-figure salary 



john8792 said:


> $1000/mo for 100Mbps, jeez, that's expensive
> 
> if i were to get several fios lines and a router, it still wouldn't be that much
> 
> 5 20/20 fios lines= $324.75/mo and is prettymuch un-metered


The difference my friend, is between dedicated bandwidth and shared bandwidth.
A dedicated unmetered 100Mbps connection at a datacenter is the equivilent of an ISP giving you a 100Mbit line, connected to their core router, and ~20TB of monthly transfer. You could host a website off that, and not have much bother, but it is not cheap.

In practise.. what your ISP gives you is the rough equivilent of a shared hosting plan.. or low-end dedicated server plan. With both, you are given a certain amount of throughput.. in GB, and you will either be cut off if you exceed it, or you'll be charged for the excess use, and the bandwidth you use, is shared by everyone else on the same server, or network segment.
I believe the Inforelay tech did a decent job explaining this.


----------



## quentinwolf (Dec 12, 2009)

Bruce said:


> I suppose that's why Yak and Dragoneer are trying to throttle the outgoing load at the FA rack router - that would be a soft cushion rather than running into a hard limiter that InfoRelay would have to throw up.
> 
> And for everyone with "High Speed Unlimited" service from Verizon FIOS or at&t U-Verse or Time-Warner "Roadrunner" or Comcast...  There ARE Limits - what the big print gives, the fine print takes away.  TANSTAAFL.
> 
> ...



 I definitely know that my connection has limits... I have an 80GB a month limit, although I can push it up to about 150GB before my ISP cares.  Not that I ever have, I've pushed 100GB once, but I know a ton of other people that do push past that, and unless you approach 150GB++ a month, They just don't care.

Plus, I am actually one of the few that does pay 3x more than the Residential plans for my Business Class ADSL, I do get Tech support faster than others on the Residential...  I'll admit mine isn't near T1 Class support, but I do get Static IP's,  extremely stable connection, and my contract allows me to host any type of server I want.  (No throttling either.  They don't throttle on the Residential plans, but they do block outgoing port 80 and possibly others, so no web-hosting unless you use a non-standard port, but getting caught can have penalties.)

Residential: $25-$35 a month
My current Server Plan: $90-$95 a month

 Worth every penny.  Thanks Telus! 3mbit down/up to 1mbit up (typically syncs around 768/896kbps for upload though.  I do get extremely good download, Consistant 352KB/sec is my maximum.)

Wish they'd offer faster stuff for us poor people out in the middle of nowhere (aka, not in a major city, they offer 10-15mbps down/1mbps up connections there, but only currently in major cities, and that is residential, no static ip's, and no server abilities... If it does become available, I'll probably get a second connection for my downloading/speed stuff, and keep the server one for my actual server+websites.  Who knows.


----------



## Ainoko (Dec 13, 2009)

Is it me or is the site having loading issues again?


----------

