As many times as I tell people to not develop on a live site for numerious reasons, it is still commonplace. I admit, I do it sometimes as well. Let’s block Google with robots.txt
The problem is Google and search engines will be actively indexing your new site and invariably you will be stuck with crappy search results for a period of time after you finish your site. Read more to find out how to fix this embarrassing error.
How many times have you looked in Google for your site, only to find the default description of Joomla being a dynamic portal showing up in your results? I can say it’s happened to me more then once. Here, a quick peek shows literally pages and pages of this.
Aside from the obvious, and actually forgetting to change your meta information, some of these sites might be in development currently. If so, all you need to do is edit your robots.txt to look like this:
Do not forget to change this back when your site goes live, as this configuration will mean Google will never index your site.
Robots.txt is very powerful and very useful, and often overlooked. For more information, read about it here.
