Apa? [ Your site returned an error when requested for robots.txt ]

Ya, jadi hari ini saya mengecek webmaster tools secara tidak sengaja, dan menemukan pemberitahuan berupa "Severe health issues are found on your site." pada status salah satu blog saya. Saya klik dan ada peringatan yaitu "Is robots.txt blocking important pages?" diikuti dengan "Your site returned an error when requested for robots.txt.".

Nah lah... apaan nie. Saya search, dapet ini :

The Web Robots Pages

Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses.

On this site you can learn more about web robots.

• About /robots.txt explains what /robots.txt is, and how to use it.
• The FAQ answers many frequently asked questions, such as How do I stop robots visiting my site? and How can I get the best listing in search engines?"
• The Other Sites page links to external resources for robot writers and webmasters.
• The Robots Database has a list of robots.
• The /robots.txt checker can check your site's /robots.txt file and meta tags.
• The IP Lookup can help find out more about what robots are visiting you.

sumber : robotstxt

Jadi kalau ada pemberitahuan itu, berarti dia mencari file robots.txt namun tidak ketemu. Tapi itu tidak masalah kalaupun tidak ada, namun untuk menghilangkan pemberitahuan error itu, kita bisa membuat file robots.txt sendiri. Cara membuatnya bisa dilihat di sini :

http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html
http://tools.seobook.com/robots-txt/generator/