Generating robots.txt files with Mavention Robots.txt

Robots.txt files are a way to tell Web Robots which part of your site they should skip while scanning it. Although it’s not an official standard using robots.txt is a common approach used for Internet-facing websites to exclude pieces of your website from crawling. Find out how you can easily create and manage the robots.txt file with Mavention Robots.txt.

What is Mavention Robots.txt?

Mavention Robots.txt is a solution that allows you to easily manage the contents of the robots.txt file on your SharePoint web site.

After deploying the Solution a new Site Collection Feature is installed.

Upon activation the Robots.txt option is added to the Site Collection Administration section in Site Settings.

Additionally a list for storing robots.txt entries is created in the Root Web. This list allows you to manage the contents of the robots.txt file.

Creating a robots.txt file using Mavention Robots.txt

Creating a robots.txt file with Mavention Robots.txt is very easy. All you need to do is to add entries to the Robots.txt entries list. For example, if you want Web Robots not to crawl the archives section on your website, you could add the following entry:

To make it easier all your entries are grouped by User-agent so that it is easy for you to track which Web Robots have access to which parts of your website.

Once you have your entries in place the last thing to do is to generate the robots.txt file. For this go to Site Actions > Site Settings > Site Collection Administration > Robots.txt and click the Generate button.

Mavention Robots.txt will automatically generate a robots.txt file for you using entries that you defined in the Robots.txt entries list.

Summary

Mavention Robots.txt is a simple yet convenient solution for creating and managing robots.txt files on Internet-facing websites built on the SharePoint platform. Although it’s not that difficult to create a robots.txt file by hand, Mavention Robots.txt becomes extra useful in scenario’s when you don’t allow using SharePoint Designer or want to provide your webmasters with an easy interface.

Download: Mavention Robots.txt (12KB, WSP)

Technorati Tags: SharePoint 2010,Web Content Management