Server issues, again

April 5, 2005 | 15 comments
By

We just had a technical mini-crisis. I think it’s past.

The crisis went like this — server load shot up to 80, memory used was at 80%, and cppop, imap, and ftpd all failed. (This is _exactly_ what happened to us at our old new server, and got that account suspended.)

I temporarily disabled a few plugins. It didn’t seem to help.

The thought occurred to me that blacklist might be causing the problem. I took blacklist and flood control off for a few minutes. It lookied like things were getting better.

Then we got hit with a huge resource spike that bumped server use to 160 (our green-light amount is 4!), and knocked Apache into “failed.” At that point, I disabled the comment script altogether. (If that killed your comment-in-progress, I apologize).

That brought things back to earth very quickly. Apache eventually went green again (whew!).

I believe that the problem is that we’re getting hit with a flood of spam. This is triggering repeated use of moderation and blacklist, which is straining server resources. (Each time blacklist runs, it checks each comment against a list of over 3000 banned sites).

I could be wrong. Anyway, we’re continuing to see if we can figure out what’s going on.

15 Responses to Server issues, again

  1. Kaimi on April 5, 2005 at 3:35 pm

    Test comment

  2. Kaimi on April 5, 2005 at 3:38 pm

    Woo-hoo! Green lights all across, except for server load which is at a mere 6.36 and dropping.

  3. Kaimi on April 5, 2005 at 3:41 pm

    Damn it! I thought the crisis was over. We even had all green, briefly. Now server load is going up again. Not good.

  4. Kaimi on April 5, 2005 at 3:58 pm

    TEST.

  5. Kaimi on April 5, 2005 at 4:16 pm

    TEST AGAIN.

  6. Eric James Stone on April 5, 2005 at 4:27 pm

    As far as I know, the “die” option ends the script immediately, which helps cut the server load.

  7. Dave on April 5, 2005 at 4:52 pm

    Yes, last night and this morning I noticed spam in the comment queue. If you start charging for comments, the problem should go away. ;-) Make it a per line charge, and the comment bloat problem will go away too.

  8. Kaimi on April 5, 2005 at 4:52 pm

    Thanks for the information, Eric.

  9. Kaimi on April 5, 2005 at 5:34 pm

    TEST TEST

  10. a random John on April 6, 2005 at 10:50 am

    does the blacklist search through each site in the list one by one? A hashtable would be a more efficient structure. Still, it seems that even 3,000 sites isn’t that many.

  11. Kaimi on April 7, 2005 at 11:18 am

    TEST COMMENT.

    TEST TEST TEST.

  12. danithew on April 7, 2005 at 12:41 pm

    When trying to arrive at T&S I find that about 50% of the time I’m getting an error page of some sort. Is anyone/everyone else having this problem?

  13. danithew on April 7, 2005 at 1:06 pm

    This is the url I’m hitting almost constantly when trying to access T&S:

    http://baby-insight.com/suspended.page/

  14. Kaimi on April 7, 2005 at 1:12 pm

    Ack. DNS errors, again.

  15. danithew on April 7, 2005 at 7:59 pm

    By the way … nice favicon.

WELCOME

Times and Seasons is a place to gather and discuss ideas of interest to faithful Latter-day Saints.