Have forgotten what a good robots.txt should read like
- SEO |
I've been using CMS's so long I've forgotten many basic html syntax basics
this request is so 2003... what *should* a good robots.txt read like?
I googled this:
Writing a good robots.txt file | SEO Tips
the author (and this post is a bit dated) has a long line of sites to block
http://urbanoalvarez.es/robots.txt
is there any good reason for mimicing this guys robots.txt?
NOTE: we have nothing to block from google, no membership site, we do not sell an ebook (yet) that we want the spiders to not tell everyone where they can get the ebook we're selling for free
is there a sound arguement for having such an extensive robots.txt?
this request is so 2003... what *should* a good robots.txt read like?
I googled this:
Writing a good robots.txt file | SEO Tips
the author (and this post is a bit dated) has a long line of sites to block
http://urbanoalvarez.es/robots.txt
is there any good reason for mimicing this guys robots.txt?
NOTE: we have nothing to block from google, no membership site, we do not sell an ebook (yet) that we want the spiders to not tell everyone where they can get the ebook we're selling for free
is there a sound arguement for having such an extensive robots.txt?