Showing posts with label blogger Robots.txt. Show all posts
Showing posts with label blogger Robots.txt. Show all posts

28 April, 2012

Custom Robots.txt

Robots.txt file helps you allow or block the search engine Robots to Crawl/index the pages. Now you can control what content you want to crawl or ignore by search engine. 
 
Not only this now you can use the robots.txt file to block search engines from crawling different parts of your blog like pages, archives, tags.  You can also add sitemap URL to BlogSpot’s robots.txt file for indexing. 
 
You can access the robots.txt file of any blog by visiting this url (http://yourblog.blogspot.com/robot.txt)
Warning!! If you don’t have much knowledge about this, never try to changing or Modifiying this file because it can effect the search engine visibility.