This Apache plugin tracks data downloaded from a server.
Mod_Spambot is an incredible Apache module that actively monitors the data streaming in from a server. It has a preset level of requests for a client. If a client exceeds that level, no more downloads are allowed for a preset period, and the client receives a tailored message informing them about what happened. This feature is incredibly useful for keeping a server safe from unwanted crawlers that bombard sites with requests.
One of the best features of mod_spambot is how customizable it is. Webmasters can tailor many of its functions to prevent false positives and customize the definition of a client to blacklist. Mod_spambot was born as a result of the code a developer wrote for an old website. The developer's ISP had blacklisted the site for being too popular. Research showed that unwanted crawlers were bombarding the site with requests. Since spiders are not actual users, the developer had to put in code to stop them. Mod_spambot was born out of that crucial need.
Mod_spambot also has an incredible side-effect that helps the system during a DoS attack. When a client that hasn't been whitelisted downloads over 100 pages within an hour, they are blacklisted unless there has been more than 10 minutes between their downloads. These values are configurable, and the algorithm is surprisingly effective. The figures are also customizable. Users can be tracked either by IP or by their mod_usertrack cookie, and after being blacklisted, they remain so until ten minutes elapse between requests.
Users that blast many requests in a short time can be throttled back before being blacklisted. HEAD requests are ignored, and users receive a customized page, the default 403 ErrorDocument, or a honeypot when blacklisted. In summary, mod_spambot is an essential tool for any webmaster looking for a reliable way of preventing unwanted crawlers from invading their website.
One of the best features of mod_spambot is how customizable it is. Webmasters can tailor many of its functions to prevent false positives and customize the definition of a client to blacklist. Mod_spambot was born as a result of the code a developer wrote for an old website. The developer's ISP had blacklisted the site for being too popular. Research showed that unwanted crawlers were bombarding the site with requests. Since spiders are not actual users, the developer had to put in code to stop them. Mod_spambot was born out of that crucial need.
Mod_spambot also has an incredible side-effect that helps the system during a DoS attack. When a client that hasn't been whitelisted downloads over 100 pages within an hour, they are blacklisted unless there has been more than 10 minutes between their downloads. These values are configurable, and the algorithm is surprisingly effective. The figures are also customizable. Users can be tracked either by IP or by their mod_usertrack cookie, and after being blacklisted, they remain so until ten minutes elapse between requests.
Users that blast many requests in a short time can be throttled back before being blacklisted. HEAD requests are ignored, and users receive a customized page, the default 403 ErrorDocument, or a honeypot when blacklisted. In summary, mod_spambot is an essential tool for any webmaster looking for a reliable way of preventing unwanted crawlers from invading their website.
What's New
Version 1.0: N/A