8 8
8 8

Help! Bing/msn bot using huge load of my server

by
kurosaki4d
kurosaki4d
Profile picture of kurosaki4d
kurosaki4d
Posts: Threads: Thanks: Account age: less than a second
7 replies
I am facing a big problem with my server. I have a website that keeps getting massive page requests coming from "Bing/Msn" bot every second or two and the ip changes now and then. Which is putting a heavy load on my server.

My CPU is constantly over 90%

I tried to block the bot from htaccess and robots.txt but they don't seem to have any effect.

If anyone has an idea how to defeat this it would be much appreciated.

Thank you in advance

#bing or msn #bot #huge #load #server
Avatar of Unregistered

Need 10 more characters to post

Loading...
  • Profile picture of the author dave_hermansen
    dave_hermansen
    dave_hermansen
    Profile picture of dave_hermansen
    dave_hermansen
    Posts: Threads: Thanks: Account age: less than a second
    Blocking your website from search engines is a pretty drastic move unless you don't care if it is ever found.

    Sure seems like those are not normal bot errors unless you have a ton of URLs that no longer exist. If so, you need to 301 them to the correct pages. If there are no relevant pages to 301 them to, remove them from Bing.

    The following link shows you how to do that - https://www.bing.com/webmaster/help/...index-37c07477
    Signature
    BizSellers.com - The #1 place to buy & sell websites!
    We help sellers get the MAXIMUM amount for their websites and all buyers know that these sites are 100% vetted.
    • Profile picture of the author kurosaki4d
      kurosaki4d
      kurosaki4d
      Profile picture of kurosaki4d
      kurosaki4d
      Posts: Threads: Thanks: Account age: less than a second
      Thank you for your help, that's good to know, i'm sure it might come up handy someday !

      For now what i did is i reduced the crawl rate of Bing and Google to the absolute minimum from "Search console" and i added the crawl rate also in the robots.txt file, plus i added wordfence to block any malicious bots that pass on as google and to block it if it exceeds a certain amount of requests within 1 minute.

      That gave me a great result and relieved the extra pressure from the server.

      Thank you !
  • Profile picture of the author savidge4
    savidge4
    savidge4
    Profile picture of savidge4
    savidge4
    Posts: Threads: Thanks: Account age: less than a second
    The IP set 44.77.167.0 to 44.77.167.255 would be the University of California ( San Diego )

    Knowing that I can for certain say that it is NOT a Bing / MSN bot. To further add to not just me saying this is so ( https://en.wikipedia.org/wiki/Msnbot )

    I would block the entire set for a period of time and call it a day.
    Signature
    Success is an ACT not an idea
  • Profile picture of the author sendizo
    sendizo
    sendizo
    Profile picture of sendizo
    sendizo
    Posts: Threads: Thanks: Account age: less than a second
    Do you happen to have large files in your server?
    It's possible that your robots.txt file's directives are not being honored because of a formatting problem, Maybe!?

    Last, You could adjust Crwaler Speed from Bing Webmaster Tool Panel,

    https://www.bing.com/webmaster/help/...ntrol-55a30302
    • Profile picture of the author kurosaki4d
      kurosaki4d
      kurosaki4d
      Profile picture of kurosaki4d
      kurosaki4d
      Posts: Threads: Thanks: Account age: less than a second
      Thank you man for your suggestion !

      Hmm for this particular site no, the robots.txt file actualy wasn't there, i had to create it myself and add in it the code to block the bots !

      As for the adjusting the crawl speed, i already verified my site with the webmasters tools, i already did that from Bing and Google and somehow it doesn't seem to work !

      Any other ideas ?
Avatar of Unregistered

Need 10 more characters to post

Loading...

Trending Topics