Download as
Browse Code »

Commit 735245597898b69d9784b8ee07e1293c5daba5bd

Authored by Rodrigo Souto
2 parents 1d9bf2f0 4743c28c
Exists in master and in 28 other branches 1.2+SPB2, 2_split_by_role, add_fields_in_api, api_fixies, api_pagination, api_private_token, api_tasks, article_hit_count, blog_page_bug, checkbox_to_user_can_edit_page, communities_ratings, communities_ratings_moderation, community_ratings_rebase, fix_cloned_article_parent, fix_gallery_image_url, fix_javascript_categories, fix_optional_fields, fix_suggest_article, fix_user_email_validation, highlight_block_fix, notification_plugin, notification_plugin_rebase, script_to_update, split_by_role, stable, temp_ratings, tests, tests_fixies

Merge branch 'ai3108' into 'stable' 

By default, block bots on searches

http://noosfero.org/Development/ActionItem3108
Showing 1 changed file with 7 additions and 1 deletions   Show diff stats
public/robots.txt
  Diff comments   View file @7352455
1   -# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
2 1 \ No newline at end of file
  2 +# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
  3 +# See also http://en.wikipedia.org/wiki/Robots_exclusion_standard for extensions
  4 +
  5 +User-agent: *
  6 +# block searches' queries
  7 +Disallow: /search*query=*
  8 +Disallow: /cat/*
... ...