NoosferoGov / errbit

Authored by Stephen Crosby 2014-10-19 14:40:09 -0700

Disable all crawlers by default

Showing 1 changed file with 2 additions and 4 deletions Show diff stats

public/robots.txt

1	1	# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
2		-#
3		-# To ban all spiders from the entire site uncomment the next two lines:
4		-# User-Agent: *
5		-# Disallow: /
	2	+User-Agent: *
	3	+Disallow: /
...	...