Commit 735245597898b69d9784b8ee07e1293c5daba5bd
Exists in
staging
and in
42 other branches
Merge branch 'ai3108' into 'stable'
By default, block bots on searches http://noosfero.org/Development/ActionItem3108
Showing
1 changed file
with
7 additions
and
1 deletions
Show diff stats
public/robots.txt
| 1 | -# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file | ||
| 2 | \ No newline at end of file | 1 | \ No newline at end of file |
| 2 | +# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file | ||
| 3 | +# See also http://en.wikipedia.org/wiki/Robots_exclusion_standard for extensions | ||
| 4 | + | ||
| 5 | +User-agent: * | ||
| 6 | +# block searches' queries | ||
| 7 | +Disallow: /search*query=* | ||
| 8 | +Disallow: /cat/* |