Google webmaster question on robots.txt

The googlebot not being able to crawl your site does
not mean others can't view it.

Most of the time, if the bot can't crawl it, then
nobody can. But visitors can view the cache. and maybe
at certain points they are let in.

robots.txt has nothing to do with the googlebot not crawling
your site.

Google looks for the robots.txt file first, so if not found,
that's the first thing they list. But that does not mean if
they do not find a robots.txt then they don't crawl it.
I have no robots.txt on the majority of my sites.

If google gives that error, it means they cannot access the
entire site, period. It's a server or hosting issue, not
a robots.txt issue.

Paul

If nothing has changed in your robots.txt then fine. But what if somehow a disallow: googlebot has been added to your site's robots.txt?
Just look at it and look for any mention of googlebot. You can look at it from google webmaster tools if you want. You could also use the tool "fetch as google" from the same place, and aks to look at yoursite.com/robots.txt, JUST IN CASE.
I don't know anything about your site, but I can think of things to do with .htaccess, server script and robots.txt to screw someone else rankings, while keeping the robots.txt file intact to any outside eyes but googlebot, if I have access to his server's ftp.

Thanks Paul & James.

Hi,

I have found out the hard way, so I will try and make this as simple as I can... You should submit your web site to google web master tools, sign in with your google account and submit your site - they have all sorts of metrics were you can test your site, index it and it comes straight from google, they also have a health section were you can see how well your site is optimized and if it is performing well in google search. I highly recommend it for newbies!

Mileagedriver,

Robots.txt is an optional file. Its purpose is to instruct search engines (and other bots) on which pages you want them to crawl. Note that a program can be made to ignore this file. Google and other major search engines abide by these files. If there is no reason to block them from some pages of your site, I would suggest just deleting the file.

Also note that if Googlebot cannot crawl your site, regular visitors will still be able to. You could block Google from indexing your site completely and people would never know the difference.

Your Google Analytics code will still execute whether you have blocked Google or not. This is completely separate from robots.txt. The only reason Analytics would not execute is if a visitor had JavaScript disabled in their browser or if you had it installed on the bottom of a page and the visitor left the page before the script could execute.

Here is an article on the basics of robots.txt,

What is Robots.txt

Hope that helps,

Shawn

HI

Agree with @TheContentAuthority, robots.txt is optional, but still a "nice to have".

However, after experiencing a similar issue myself, I think you should be more concerned with an availability factor, because this may indicate downtime issues.

Keep in mind, 33% is pretty high and prolonged/frequent downtime issues can and will hurt your SEO (among other things, such as repeat visits, trust and etc)

I would recommend using Pingdom (or similar) to monitor you websites availability.

i have same problem here

use robots.txt configured properly on your website.
This will mostly solve.