WCM tip #27: Use XML Sitemap and Robots.txt
On your public-facing website use the XML Sitemap and Robots.txt file to optimize your website for Internet search engines.
SharePoint 2013 allows you to automatically generate an XML Sitemap and the Robots.txt file for your website. Both files help Internet search engines discover and index the contents of your website. Although it is only a matter of time until Internet search engines will discover the newest content on your website, using an XML Sitemap and the Robots.txt file can help you notify Internet search engines of the content updates and have the indexed faster.
In order to use the standard XML Sitemap and Robots.txt files generated by SharePoint 2013 you have to enable the Search Engine Sitemap Site Collection Feature. Additionally, because the process of generating the XML Sitemap is based on SharePoint 2013 Search, you have to ensure that your environment has SharePoint 2013 Search configured and that the contents of your website are properly indexed. If everything is configured correctly, SharePoint 2013 will automatically create a new XML Sitemap for you every day including all latest changes to the contents of your website.