Author Topic: Server Slow-Downs, and Excessive Traffic  (Read 2206 times)

0 Members and 1 Guest are viewing this topic.

Offline Meadow Whisper (Natasha)

  • ᎣᏏᏲ! I'm a horse now! ^.^
  • FT Founder
  • *
  • Posts: 4409
  • Cookies: 397
  • Avatar by Fawn! :D
    • DeviantArt - lupiniastudios
    • FurAffinity - softpaw
    • Twitter - lupinia
    • Weasyl - softpaw
    • Lupinia Studios
  • Fursona Species: Chincoteague Pony
Server Slow-Downs, and Excessive Traffic
« on: March 09, 2010, 10:29:12 pm »
For the past few weeks, the Lupinia server (which hosts FT, for free, at my expense) has experience intermittant slowdowns which have impacted the performance of all services and websites hosted by Lupinia, including email, web, and database services.  This has been most noticeable on FT, since it has the largest userbase.  I've been investigating the source of this problem for awhile, but with its intermittant nature, I had a hard time pinpointing an exact source.  Plus, parsing log files that can be measured in megabytes is painfully tedious, especially for a volunteer.

The reason this has gone on for so long is because I neglected to look at one possible source for the problems:  FT itself.  I checked into a few log files and traffic figures, and found some very interesting data.  Aside from the fact that FT's average daily hits nearly doubled starting February 6-7, experimenting with blocking a few forum member IP addresses in the server's firewall resulted in completely relieving the server of a rather crippling load.  What this means is that a small number of members of this forum have been responsible for impairing service not only to the rest of FT, but to several dozen other websites, as well as Lupinia's email services and my ability to manage the server.

I've already identified one culprit, and I'll be parsing the log files over the next few days to find any others.  I've also implemented the following server/forum policies in response to this:

1.  Excessive PM user blocked from sending PMs
Our PM system is not an instant messenger conversation, it's an email conversation, and it's not designed for rapid-fire messages.  I've identified heavy PM traffic as a big contributor to these heavy loads, so the biggest offenders have been blocked from using the system.

2.  Per-IP TCP limit reduced from 150 to 100
This is hard to explain without getting really geeky, but the short version is that if your web browser hits this limit, the server will deny it any other connections until old ones time out.  If you frequently hit this limit, you've either adjusted your browser to use more connections beyond normal specs (via manual adjustment or certain "speed" plugins), or you're browsing/refreshing too rapidly.

3.  Anyone suspected of using an auto-refresh script/plugin will be banned from the entire server
We're not 4chan or Gaia, things just don't move that fast here, and while we have a much better server than we used to be on, we're just not equipped to handle such a constant load.  Thus, there's no reason to run any sort of auto-refresh system.  Anyone suspected of running such a script will be blocked in the server's firewall.

In addition to these, I'll be investigating some caching options to help FT continue to grow.  But, identifying and addressing these abusive behaviours has already improved the site's performance just in the last half-hour.
  • Award - Beta Tester
  • Site Donor
  • Pride Flag - transgender
  • Pride Flag - gay
  • Pronouns: She/Her/Hers
Here is a horse I have made for you:
        |,\_/,\_
 ,-__/,   ,       )kk
.                     \,kkk
 \    ,___....--     \kkkk
           \               \kkkkk
             \               \kkkkk

Note:  I'm no longer actively involved in running this site, so while I welcome any personal messages, please do not contact me about account issues.  Please direct all such inquiries to Panda.  Thanks for all the years of awesome memories and friendship!  <3

Offline Alex Galloway

  • The Defender
  • Blue Level
  • *
  • Posts: 1764
  • Cookies: 62
  • :D
    • FurAffinity - HaddockWolf
  • Fursona Species: Dracocatwolf, Illogical Tribrid from another dimension
Re: Server Slow-Downs, and Excessive Traffic
« Reply #1 on: March 10, 2010, 08:33:12 pm »
This Explains a lot! Thanks for the info, I was having issues with connecting to FT. How Do I check my TCP limit?
"In the end we will not remember the words of our enemies, but the silence of our friends." -Dr. MLK Jr.
Every person is born a white canvas, and each person we touch, each person we talk to, they impart a shade of their color upon you. and these mixes of colors blend into a unique shade that is you - Alexander (me)

Offline copb.phoenix

  • So long and goodnight
  • Purple Level
  • *
  • Posts: 2229
  • Cookies: 115
    • FurAffinity - copb_phoenix
Re: Server Slow-Downs, and Excessive Traffic
« Reply #2 on: March 10, 2010, 09:04:22 pm »
I had wondered.

:shuts down update scanner and kills all but his sitewide .rss feed:

It's good to know I'm not crazy.

EDIT: Turns out I didn't have US turned on anything here, anyway. Thank yourself and the SMF dev team for the "updated topics" feature.
« Last Edit: March 10, 2010, 09:06:00 pm by copb.phoenix »
  • Flag - turing
  • Award - Beta Tester
Isn't this where...?

Offline Schism

  • Blue Level
  • *
  • Posts: 1495
  • Cookies: 82
Re: Server Slow-Downs, and Excessive Traffic
« Reply #3 on: March 10, 2010, 09:05:47 pm »
I'm glad you did something about the rapid-fire PMing, I've noticed a lot of people do that.

Offline Alex Galloway

  • The Defender
  • Blue Level
  • *
  • Posts: 1764
  • Cookies: 62
  • :D
    • FurAffinity - HaddockWolf
  • Fursona Species: Dracocatwolf, Illogical Tribrid from another dimension
Re: Server Slow-Downs, and Excessive Traffic
« Reply #4 on: March 10, 2010, 09:15:07 pm »
Does emptying my pm box help? I still have ever one I've ever received, as I have never had a full box...
"In the end we will not remember the words of our enemies, but the silence of our friends." -Dr. MLK Jr.
Every person is born a white canvas, and each person we touch, each person we talk to, they impart a shade of their color upon you. and these mixes of colors blend into a unique shade that is you - Alexander (me)

Offline Meadow Whisper (Natasha)

  • ᎣᏏᏲ! I'm a horse now! ^.^
  • FT Founder
  • *
  • Posts: 4409
  • Cookies: 397
  • Avatar by Fawn! :D
    • DeviantArt - lupiniastudios
    • FurAffinity - softpaw
    • Twitter - lupinia
    • Weasyl - softpaw
    • Lupinia Studios
  • Fursona Species: Chincoteague Pony
Re: Server Slow-Downs, and Excessive Traffic
« Reply #5 on: March 10, 2010, 09:22:06 pm »
Not really, the issue is active traffic, not stored data.
  • Award - Beta Tester
  • Site Donor
  • Pride Flag - transgender
  • Pride Flag - gay
  • Pronouns: She/Her/Hers
Here is a horse I have made for you:
        |,\_/,\_
 ,-__/,   ,       )kk
.                     \,kkk
 \    ,___....--     \kkkk
           \               \kkkkk
             \               \kkkkk

Note:  I'm no longer actively involved in running this site, so while I welcome any personal messages, please do not contact me about account issues.  Please direct all such inquiries to Panda.  Thanks for all the years of awesome memories and friendship!  <3

Offline Alex Galloway

  • The Defender
  • Blue Level
  • *
  • Posts: 1764
  • Cookies: 62
  • :D
    • FurAffinity - HaddockWolf
  • Fursona Species: Dracocatwolf, Illogical Tribrid from another dimension
Re: Server Slow-Downs, and Excessive Traffic
« Reply #6 on: March 10, 2010, 09:26:11 pm »
This Explains a lot! Thanks for the info, I was having issues with connecting to FT. How Do I check my TCP limit?
Just a bump.
"In the end we will not remember the words of our enemies, but the silence of our friends." -Dr. MLK Jr.
Every person is born a white canvas, and each person we touch, each person we talk to, they impart a shade of their color upon you. and these mixes of colors blend into a unique shade that is you - Alexander (me)

Offline Meadow Whisper (Natasha)

  • ᎣᏏᏲ! I'm a horse now! ^.^
  • FT Founder
  • *
  • Posts: 4409
  • Cookies: 397
  • Avatar by Fawn! :D
    • DeviantArt - lupiniastudios
    • FurAffinity - softpaw
    • Twitter - lupinia
    • Weasyl - softpaw
    • Lupinia Studios
  • Fursona Species: Chincoteague Pony
Re: Server Slow-Downs, and Excessive Traffic
« Reply #7 on: March 10, 2010, 09:37:04 pm »
Depends on your browser, and any plugins you're using with it.
  • Award - Beta Tester
  • Site Donor
  • Pride Flag - transgender
  • Pride Flag - gay
  • Pronouns: She/Her/Hers
Here is a horse I have made for you:
        |,\_/,\_
 ,-__/,   ,       )kk
.                     \,kkk
 \    ,___....--     \kkkk
           \               \kkkkk
             \               \kkkkk

Note:  I'm no longer actively involved in running this site, so while I welcome any personal messages, please do not contact me about account issues.  Please direct all such inquiries to Panda.  Thanks for all the years of awesome memories and friendship!  <3

Offline copb.phoenix

  • So long and goodnight
  • Purple Level
  • *
  • Posts: 2229
  • Cookies: 115
    • FurAffinity - copb_phoenix
Re: Server Slow-Downs, and Excessive Traffic
« Reply #8 on: March 10, 2010, 09:38:40 pm »
Depends on your browser, and any plugins you're using with it.

Stark curiosity, on my part: Firefox, Chrome, and IE 8? If you know any, I'll be happy to check mine, not to mention those are all fairly common.

EDIT: Assume no plug-ins. I know how to check the only three that could be a potential issue.
« Last Edit: March 10, 2010, 09:40:31 pm by copb.phoenix »
  • Flag - turing
  • Award - Beta Tester
Isn't this where...?

Offline Meadow Whisper (Natasha)

  • ᎣᏏᏲ! I'm a horse now! ^.^
  • FT Founder
  • *
  • Posts: 4409
  • Cookies: 397
  • Avatar by Fawn! :D
    • DeviantArt - lupiniastudios
    • FurAffinity - softpaw
    • Twitter - lupinia
    • Weasyl - softpaw
    • Lupinia Studios
  • Fursona Species: Chincoteague Pony
Re: Server Slow-Downs, and Excessive Traffic
« Reply #9 on: March 11, 2010, 05:33:30 am »
I'm about to go to bed, I'll post the browser information sometime tomorrow.  I did want to update this and mention that things are still running sporadically slowly, and I'm still investigating.  I'm also experimenting with ways to optimize the site to handle more traffic, some of which are more effective than others, so if things run slowly (or don't load at all), please bear with me, and just re-try whatever you were doing in a few minutes.  A few things I've already done:

-Shifted all static content (themes, images, smileys, etc) to a different virtual host, and set the content expiration to keep that data in your browser cache for 30 days or until the cache is cleared.  Also set content on this virtual host to reject hotlinking, just in case we have some leechers out there.
-Dedicated even more server resources to FT, giving it priority over every other website hosted on Lupinia, and giving it a dedicated "worker process" (how IIS handles load-balancing).
-Disabled offsite-linked avatars.  This doesn't really affect the performance of FT itself, but it does affect page loading times and create the illusion of poor performance.
-Experimented with server-side compression, though this hasn't worked as smoothly as I anticipated.  I'm looking into other methods of doing it.
-Disabled RSS feeds (experimental, I'm curious to see how it affects traffic tomorrow).

Honestly, I'm a little worried that FT is already outgrowing its current home, because we don't really have anywhere else to go except to a different, higher-grade co-located server, and I don't have thousands of dollars to invest in another one.  But, we'll cross that bridge when we get there.  In the meantime, if anyone would like to donate to the "Upgraded Server for FT" fund, contact me privately.
« Last Edit: March 11, 2010, 06:03:49 am by Natasha Softpaw »
  • Award - Beta Tester
  • Site Donor
  • Pride Flag - transgender
  • Pride Flag - gay
  • Pronouns: She/Her/Hers
Here is a horse I have made for you:
        |,\_/,\_
 ,-__/,   ,       )kk
.                     \,kkk
 \    ,___....--     \kkkk
           \               \kkkkk
             \               \kkkkk

Note:  I'm no longer actively involved in running this site, so while I welcome any personal messages, please do not contact me about account issues.  Please direct all such inquiries to Panda.  Thanks for all the years of awesome memories and friendship!  <3

Offline Daniel A. Skirtandzy

  • Purple Level
  • *
  • Posts: 3409
  • Cookies: 162
    • FurAffinity - Skirtandzy1
    • LiveJournal - Skirtandzy
    • Twitter - Skirtandzy
  • Fursona Species: Domestic Cat
Re: Server Slow-Downs, and Excessive Traffic
« Reply #10 on: March 11, 2010, 02:41:07 pm »
Honestly, I'm a little worried that FT is already outgrowing its current home, because we don't really have anywhere else to go except to a different, higher-grade co-located server, and I don't have thousands of dollars to invest in another one.  But, we'll cross that bridge when we get there.  In the meantime, if anyone would like to donate to the "Upgraded Server for FT" fund, contact me privately.

Before I even got a quarter of the way through, this came to mind.
I have a little bit of work to finish up request wise, but when I fininish that, I may just post a journal telling people they can get a cheap comission for a small donation.
  • Flag - lawyer
  • Award - Beta Tester
  • Site Donor

Offline Varlen

  • So Amaze, Such
  • Blue Level
  • *
  • Posts: 1275
  • Cookies: 103
  • FTS RESODENT WIERDO SMILE DOOER.
  • Fursona Species: El Skunko!
Re: Server Slow-Downs, and Excessive Traffic
« Reply #11 on: March 11, 2010, 05:25:52 pm »
A little off topic;

Is anyone else outside the US/Canada having insane connection times? It's taking me around 20 seconds just to load the reply box, several minutes if I dare look at a new or updated thread. It's more effort than it's worth >.<
  • Flag - banana
  • Award - Beta Tester
Var's Vid of the Week:
Week 13, AK Underwater. If you have any suggestions for the Vid of the Week, pm me! ^.^


Offline Kambo

  • Personally knew Abe Lincoln
  • Blue Level
  • *
  • Posts: 1801
  • Cookies: 85
  • True story.
    • DeviantArt - kamboblack
    • FurAffinity - harlan
    • Twitter - kamboblack
    • Weasyl - harlan
    • Unfinished Portfolio Site :(
  • Fursona Species: Grey fox
Re: Server Slow-Downs, and Excessive Traffic
« Reply #12 on: March 11, 2010, 08:33:21 pm »
Me! I'm having horrible connection issues. About a third of the time, the page can't load at all and I have to try again. Other times, it just takes forever.
  • Award - Beta Tester
  • Pronouns: He/Him/His
Feel free to PM me, I don't bite! I reserve that treatment for sushi and pretzels.

Fursona References (Kambo and Camille)

Offline Meadow Whisper (Natasha)

  • ᎣᏏᏲ! I'm a horse now! ^.^
  • FT Founder
  • *
  • Posts: 4409
  • Cookies: 397
  • Avatar by Fawn! :D
    • DeviantArt - lupiniastudios
    • FurAffinity - softpaw
    • Twitter - lupinia
    • Weasyl - softpaw
    • Lupinia Studios
  • Fursona Species: Chincoteague Pony
Re: Server Slow-Downs, and Excessive Traffic
« Reply #13 on: March 12, 2010, 08:18:34 am »
Thanks to a conversation with Green Reaper, one of the most awesome furries evar, we came up with some things to try.  Our attempts to install PHP code caching were unsuccessful due to compatibility issues, but I was able to configure the server's firewall for content compression, a technique that will save a ton of bandwidth.  And, since it's handled by the firewall (which processes web traffic anyway), I could disable it in PHP and the web server application, something that should cut down on processor load.

I also dramatically trimmed down the number of triggers that create firewall log entries, removing any redundant data from the firewall logs, a move that should help at least a little bit.

Hopefully, these changes in addition to the new forum software will help ease the load at peak times.
  • Award - Beta Tester
  • Site Donor
  • Pride Flag - transgender
  • Pride Flag - gay
  • Pronouns: She/Her/Hers
Here is a horse I have made for you:
        |,\_/,\_
 ,-__/,   ,       )kk
.                     \,kkk
 \    ,___....--     \kkkk
           \               \kkkkk
             \               \kkkkk

Note:  I'm no longer actively involved in running this site, so while I welcome any personal messages, please do not contact me about account issues.  Please direct all such inquiries to Panda.  Thanks for all the years of awesome memories and friendship!  <3

Offline copb.phoenix

  • So long and goodnight
  • Purple Level
  • *
  • Posts: 2229
  • Cookies: 115
    • FurAffinity - copb_phoenix
Re: Server Slow-Downs, and Excessive Traffic
« Reply #14 on: March 12, 2010, 08:22:50 am »
<3 GR, one of the most persistent people I've ever seen on the Wikipedia Furry Project (and for many obvious and good reasons).

Still needing me? I've been awake for 12 hours; I guess what I'm asking is --> Is there any reason for me to hang tight now that the sun is shining and it's done raining?
  • Flag - turing
  • Award - Beta Tester
Isn't this where...?

Offline Varlen

  • So Amaze, Such
  • Blue Level
  • *
  • Posts: 1275
  • Cookies: 103
  • FTS RESODENT WIERDO SMILE DOOER.
  • Fursona Species: El Skunko!
Re: Server Slow-Downs, and Excessive Traffic
« Reply #15 on: March 12, 2010, 04:49:03 pm »
This version of the forums seems to have slowed things down for me, drastically. >.<.

I also tried accessing the site via my sisters PC (as opposed to this Mac) and it's equally slow - could just be our ISP, but I'm putting it out there anyway; SOMETHING HAS BROKEND THE INTERMERNETS BETWEEN US AND YOU.

D'= fix it, cyberfairy!
  • Flag - banana
  • Award - Beta Tester
Var's Vid of the Week:
Week 13, AK Underwater. If you have any suggestions for the Vid of the Week, pm me! ^.^


Offline Meadow Whisper (Natasha)

  • ᎣᏏᏲ! I'm a horse now! ^.^
  • FT Founder
  • *
  • Posts: 4409
  • Cookies: 397
  • Avatar by Fawn! :D
    • DeviantArt - lupiniastudios
    • FurAffinity - softpaw
    • Twitter - lupinia
    • Weasyl - softpaw
    • Lupinia Studios
  • Fursona Species: Chincoteague Pony
Re: Server Slow-Downs, and Excessive Traffic
« Reply #16 on: March 12, 2010, 05:19:51 pm »
This version of the forums seems to have slowed things down for me, drastically. >.<.

I also tried accessing the site via my sisters PC (as opposed to this Mac) and it's equally slow - could just be our ISP, but I'm putting it out there anyway; SOMETHING HAS BROKEND THE INTERMERNETS BETWEEN US AND YOU.

D'= fix it, cyberfairy!

Do you know how to do a tracert, ping, and nslookup?  Also, note that at the moment, we're hitting peak traffic, and the server is yet again getting rather pwned :-(
  • Award - Beta Tester
  • Site Donor
  • Pride Flag - transgender
  • Pride Flag - gay
  • Pronouns: She/Her/Hers
Here is a horse I have made for you:
        |,\_/,\_
 ,-__/,   ,       )kk
.                     \,kkk
 \    ,___....--     \kkkk
           \               \kkkkk
             \               \kkkkk

Note:  I'm no longer actively involved in running this site, so while I welcome any personal messages, please do not contact me about account issues.  Please direct all such inquiries to Panda.  Thanks for all the years of awesome memories and friendship!  <3

Offline GreenReaper

  • Rambling Norn
  • New Member
  • *
  • Posts: 6
  • Cookies: 3
  • Edits: More than you!
    • DeviantArt - greenreaper
    • FurAffinity - greenreaper
    • LiveJournal - greenreaper
    • Twitter - FlayrahNews
    • GreenReaper Studios
Re: Server Slow-Downs, and Excessive Traffic
« Reply #17 on: March 12, 2010, 06:41:31 pm »
With the changes made, it's delivering each portion of the page really fast once it's generated. Unfortunately, without APC, the forum software has to be interpreted into native code every time a page is accessed. It's like having to rebuild your car every time you want to go somewhere.

APC caches this code, so you only have to build it once. It also lets you store arbitrary bits of data, which can speed things up even more. But the PHP extension project that handles APC hasn't got its stuff together with Windows builds of its extensions - as a result, it's hard to find builds of them, and they often don't work properly when you do find them. :/
Laurence "GreenReaper" Parry
GreenReaper Studios - WikiFur - Flayrah

Offline Meadow Whisper (Natasha)

  • ᎣᏏᏲ! I'm a horse now! ^.^
  • FT Founder
  • *
  • Posts: 4409
  • Cookies: 397
  • Avatar by Fawn! :D
    • DeviantArt - lupiniastudios
    • FurAffinity - softpaw
    • Twitter - lupinia
    • Weasyl - softpaw
    • Lupinia Studios
  • Fursona Species: Chincoteague Pony
Re: Server Slow-Downs, and Excessive Traffic
« Reply #18 on: March 12, 2010, 06:56:49 pm »
Actually, I think the problem now is the firewall/NIC, there isn't really anything left. One of Lupinia's co-admins is a certified ISA expert, I'm consulting him this evening.
  • Award - Beta Tester
  • Site Donor
  • Pride Flag - transgender
  • Pride Flag - gay
  • Pronouns: She/Her/Hers
Here is a horse I have made for you:
        |,\_/,\_
 ,-__/,   ,       )kk
.                     \,kkk
 \    ,___....--     \kkkk
           \               \kkkkk
             \               \kkkkk

Note:  I'm no longer actively involved in running this site, so while I welcome any personal messages, please do not contact me about account issues.  Please direct all such inquiries to Panda.  Thanks for all the years of awesome memories and friendship!  <3

Offline GreenReaper

  • Rambling Norn
  • New Member
  • *
  • Posts: 6
  • Cookies: 3
  • Edits: More than you!
    • DeviantArt - greenreaper
    • FurAffinity - greenreaper
    • LiveJournal - greenreaper
    • Twitter - FlayrahNews
    • GreenReaper Studios
Re: Server Slow-Downs, and Excessive Traffic
« Reply #19 on: March 12, 2010, 07:30:05 pm »
Whatever you just did, if anything, it got a lot faster . . .

Maybe it's a bot? I notice robots.txt is kinda bare. Bots probably don't need to access more than the base pages, if that - you could add Deny: /forum/index.php?action=
Laurence "GreenReaper" Parry
GreenReaper Studios - WikiFur - Flayrah