109,896
edits
Problem http User-Agents: Difference between revisions
no edit summary
No edit summary |
No edit summary |
||
| Line 1: | Line 1: | ||
<!-- seo --> | <!-- seo --> | ||
{{#seo: | {{#seo: | ||
|title=Problem http User-Agents | |title=Problem http User-Agents | ||
|titlemode=replace | |titlemode=replace | ||
|description=How I block pesky web bots from my | |keywords=#tools #problemhttpuseragents #badbots #apache #httpdconf | ||
|hashtagrev=12032020 | |||
|description=How I block pesky web bots from my webserver. Apache | |||
}} | }} | ||
<!-- /seo --> | <!-- /seo --> | ||
| Line 10: | Line 11: | ||
An Apache rewrite rule to block the pesky httpd_User_Agents that slow up my server because of the bandwidth they use up. | An Apache rewrite rule to block the pesky httpd_User_Agents that slow up my server because of the bandwidth they use up. | ||
I have built up this list of problem user-agents over months of log analysis. Most of these agents ignore | I have built up this list of problem user-agents over months of log analysis. Most of these agents ignore robots.txt or at best, ignore the crawler-delay command in robots.txt: | ||
<pre> | <pre> | ||
| Line 54: | Line 55: | ||
{{CategoryLine}} | {{CategoryLine}} | ||
[[Category:Tools]] | [[Category:Tools]] | ||
<!-- footer hashtags --><code 'hashtagrev:12032020'>#tools #problemhttpuseragents #badbots #apache #httpdconf</code><!-- /footer_hashtags --> | |||