Commit f7792c2615a8948a40e836740bdc5e9658245aef
Exists in
master
and in
1 other branch
Merge pull request #743 from nfedyashev/patch-2
Disable all crawlers by default
Showing
1 changed file
with
2 additions
and
4 deletions
Show diff stats
public/robots.txt
| 1 | # See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file | 1 | # See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file |
| 2 | -# | ||
| 3 | -# To ban all spiders from the entire site uncomment the next two lines: | ||
| 4 | -# User-Agent: * | ||
| 5 | -# Disallow: / | 2 | +User-Agent: * |
| 3 | +Disallow: / |