Web Crawler & User Agent Blocking Techniques

From securityboulevard.com

This is a simple script that allows hackers to block specific crawlers based upon website requests from specific user-agents. This is useful when you don’t want certain traffic from being able to load certain content – usually a phishing page or a malicious download.

Read more…