109,896
edits
Problem http User-Agents: Difference between revisions
no edit summary
(Created page with " <!-- seo --> {{#seo: |title=--title-- |titlemode=replace |description=--description-- }} <!-- /seo --> === Stop the Bandwidth Hogs=== An Apache rewrite rule to block the pesky httpd_User_Agents that slow up my server because of the bandwidth they use up. Add the following lines to your httpd config file: /etc/httpd/conf.d/domainname.conf ''multi-line version'' <pre> RewriteCond %{HTTP_USER_AGENT} \ Custom-AsyncHttpClient|EzoicBot|Turnitin|sqlmap|wikiteam3dumpgene...") |
No edit summary |
||
| Line 9: | Line 9: | ||
=== Stop the Bandwidth Hogs=== | === Stop the Bandwidth Hogs=== | ||
An Apache rewrite rule to block the pesky httpd_User_Agents that slow up my server because of the bandwidth they use up. | An Apache rewrite rule to block the pesky httpd_User_Agents that slow up my server because of the bandwidth they use up. | ||
I have built up this list of problem user-agents over months of log analysis. Most of these agents ignore robits.txt or at best, ignore the crawler-delay command in robots.txt: | |||
<pre> | |||
User-agent: * | |||
Crawl-delay: 4 | |||
</pre> | |||
Add the following lines to your httpd config file: /etc/httpd/conf.d/domainname.conf | Add the following lines to your httpd config file: /etc/httpd/conf.d/domainname.conf | ||