12 {{ upvoteCount | shortNum }}
12 {{ upvoteCount | shortNum }}

Keeping bots out during development

by rgwhitaker
10 replies
I'm in the process of getting my website up and running. I'm having an individual finish with the final development aspect.

During this time, while he's doing the backend development, I need to put the files on the server where the site is going to be hosted. But, during development, I don't want the search bots finding the site.

Do I simply place a Disallow * in the robots.txt file, so that the entire site is passed over by the bots? Is that the best way to go?

Thanks for any and all help.
#bots #development #keeping
  • Profile picture of the author dripable
    The best thing we always do is password protect the site so no one can access it. I also find it much easier to develop on mysite.com/special_folder_for_development (make up your own name for it) this way we can simply just migrate new changes when needed with out interrupting our clients or ability to use our site.
    Signature
    {{ DiscussionBoard.errors[4415911].message }}
    • Profile picture of the author Dee Odus
      Originally Posted by dripable View Post

      The best thing we always do is password protect the site so no one can access it. I also find it much easier to develop on mysite.com/special_folder_for_development (make up your own name for it) this way we can simply just migrate new changes when needed with out interrupting our clients or ability to use our site.
      He's asking how to keep Google bots away, not users :-)
      {{ DiscussionBoard.errors[4417949].message }}
  • Profile picture of the author sbucciarel
    Banned
    this is what you use in robots.txt

    User-agent: *
    Disallow: /
    {{ DiscussionBoard.errors[4415997].message }}
    • Profile picture of the author AnniePot
      Originally Posted by sbucciarel View Post

      this is what you use in robots.txt

      User-agent: *
      Disallow: /
      Exactly: this is all you need to do. Or, to remove your site from just Google's crawls, amend it to:

      User-agent: Googlebot
      Disallow: /
      {{ DiscussionBoard.errors[4418196].message }}
      • Profile picture of the author rgwhitaker
        I will be using WordPress, but that's only going to be a portion of the website--the blog portion. The entire site is not WordPress enabled.

        So, during development I'll just use the suggestions above to disallow all bots from crawling my site.

        Another question, what is the lines I include in the robots.txt file when I'm ready to go live?
        {{ DiscussionBoard.errors[4420172].message }}
        • Profile picture of the author Istvan Horvath
          Originally Posted by rgwhitaker View Post

          Another question, what is the lines I include in the robots.txt file when I'm ready to go live?
          It depends what you want the bots to do...

          Let them crawl and index your site - except, maybe, the portions not intended for visitors.

          Everything you need is there: The Web Robots Pages
          Signature

          {{ DiscussionBoard.errors[4420231].message }}
  • Profile picture of the author andreasnrb
    You can also use a maintenance mode plugin so visitors don't just wander in while your developing or updating.
    {{ DiscussionBoard.errors[4418165].message }}
    • Profile picture of the author Dee Odus
      Originally Posted by andreasnrb View Post

      You can also use a maintenance mode plugin so visitors don't just wander in while your developing or updating.
      I was going to suggest this as well but it doesnt look like he's using wordpress. I also realised recently that wordpress automatically defaults to "private" after installation which means until you actually enable "public" in the settings area you can continue to work on it without having to worry about Google indexing your website
      {{ DiscussionBoard.errors[4418179].message }}
  • Profile picture of the author SebastianJ
    As one warrior already posted, modify your robots.txt:

    User-agent: *
    Disallow: /
    {{ DiscussionBoard.errors[4432826].message }}

Trending Topics