Block badbot with fail2ban via user agents in access.log
0
How can I create a filter to block these with fail2ban?
476 Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
892 ltx71 - (http://ltx71.com/)
5367 Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
6449 Barkrowler/0.9 (+http://www.exensa.com/crawl)
This list come out from this:
sudo cat /var/log/apache2/access.log | awk -F" '{print $6}' | sort | uniq -c | sort -n
I've tried apache-badbot.conf, but it does not seem to work ...
fail2ban
asked 22 hours ago
alebalalebal
1115
add a comment |
0
How can I create a filter to block these with fail2ban?
476 Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
892 ltx71 - (http://ltx71.com/)
5367 Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
6449 Barkrowler/0.9 (+http://www.exensa.com/crawl)
This list come out from this:
sudo cat /var/log/apache2/access.log | awk -F" '{print $6}' | sort | uniq -c | sort -n
I've tried apache-badbot.conf, but it does not seem to work ...
fail2ban
asked 22 hours ago
alebalalebal
1115
Those are user agents, not referrers. In my experience DotBot and BLEXBot obey robots.txt, if aDisallowdirective exits for them. ltx71 ignores robots.txt, and I had to block all of their IP addresses (at the time: 54.174.24.200, 54.175.147.90, 54.175.196.96) via (in my case) iptables rules.
– Doug Smythies
19 hours ago
I looked at your previous questions, and realize that you have been having issues for months. It is not clear to me why you are having so many continuing issues with the typical ways of dealing with these type bots. By the way, my iptables rules for ltx71.com are because those 3 IP addresses did not identify themselves asltx71via user agent string. I also have rewrite rules that allow ltx71 to get my robots.txt file, but otherwise tell them they are banned because they don't obey the robots.txt file.
– Doug Smythies
11 hours ago
"Those are user agents" - You're right... sorry I got confused after all this googling. These are the ones that are in my robots.txt, but they ignore the directive. At the end after the old thread I created a dynamic robots.txt that updates simultaneously in all my sites, and contains about fifty working rules. Until some time ago, I added with a script all the IPs of unwanted bots to UFW, but at one point I arrived at thousands of IP, and I had to look for another way. Of ltx71 I have two new ones only today (35.192.84.7, 104.155.179.226).
– alebal
35 mins ago
add a comment |
0
0
0
How can I create a filter to block these with fail2ban?
476 Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
892 ltx71 - (http://ltx71.com/)
5367 Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
6449 Barkrowler/0.9 (+http://www.exensa.com/crawl)
This list come out from this:
sudo cat /var/log/apache2/access.log | awk -F" '{print $6}' | sort | uniq -c | sort -n
I've tried apache-badbot.conf, but it does not seem to work ...
fail2ban
asked 22 hours ago
alebalalebal
1115
How can I create a filter to block these with fail2ban?
476 Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
892 ltx71 - (http://ltx71.com/)
5367 Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
6449 Barkrowler/0.9 (+http://www.exensa.com/crawl)
This list come out from this:
sudo cat /var/log/apache2/access.log | awk -F" '{print $6}' | sort | uniq -c | sort -n
I've tried apache-badbot.conf, but it does not seem to work ...
fail2ban
fail2ban
asked 22 hours ago
alebalalebal
1115
asked 22 hours ago
alebalalebal
1115
edited 30 mins ago
alebal
asked 22 hours ago
alebalalebal
1115
asked 22 hours ago
alebalalebal
1115
asked 22 hours ago
alebalalebal
1115
1115
Those are user agents, not referrers. In my experience DotBot and BLEXBot obey robots.txt, if aDisallowdirective exits for them. ltx71 ignores robots.txt, and I had to block all of their IP addresses (at the time: 54.174.24.200, 54.175.147.90, 54.175.196.96) via (in my case) iptables rules.
– Doug Smythies
19 hours ago
I looked at your previous questions, and realize that you have been having issues for months. It is not clear to me why you are having so many continuing issues with the typical ways of dealing with these type bots. By the way, my iptables rules for ltx71.com are because those 3 IP addresses did not identify themselves asltx71via user agent string. I also have rewrite rules that allow ltx71 to get my robots.txt file, but otherwise tell them they are banned because they don't obey the robots.txt file.
– Doug Smythies
11 hours ago
"Those are user agents" - You're right... sorry I got confused after all this googling. These are the ones that are in my robots.txt, but they ignore the directive. At the end after the old thread I created a dynamic robots.txt that updates simultaneously in all my sites, and contains about fifty working rules. Until some time ago, I added with a script all the IPs of unwanted bots to UFW, but at one point I arrived at thousands of IP, and I had to look for another way. Of ltx71 I have two new ones only today (35.192.84.7, 104.155.179.226).
– alebal
35 mins ago
add a comment |
Those are user agents, not referrers. In my experience DotBot and BLEXBot obey robots.txt, if aDisallowdirective exits for them. ltx71 ignores robots.txt, and I had to block all of their IP addresses (at the time: 54.174.24.200, 54.175.147.90, 54.175.196.96) via (in my case) iptables rules.
– Doug Smythies
19 hours ago
I looked at your previous questions, and realize that you have been having issues for months. It is not clear to me why you are having so many continuing issues with the typical ways of dealing with these type bots. By the way, my iptables rules for ltx71.com are because those 3 IP addresses did not identify themselves asltx71via user agent string. I also have rewrite rules that allow ltx71 to get my robots.txt file, but otherwise tell them they are banned because they don't obey the robots.txt file.
– Doug Smythies
11 hours ago
"Those are user agents" - You're right... sorry I got confused after all this googling. These are the ones that are in my robots.txt, but they ignore the directive. At the end after the old thread I created a dynamic robots.txt that updates simultaneously in all my sites, and contains about fifty working rules. Until some time ago, I added with a script all the IPs of unwanted bots to UFW, but at one point I arrived at thousands of IP, and I had to look for another way. Of ltx71 I have two new ones only today (35.192.84.7, 104.155.179.226).
– alebal
35 mins ago
Those are user agents, not referrers. In my experience DotBot and BLEXBot obey robots.txt, if a
Disallow directive exits for them. ltx71 ignores robots.txt, and I had to block all of their IP addresses (at the time: 54.174.24.200, 54.175.147.90, 54.175.196.96) via (in my case) iptables rules.– Doug Smythies
19 hours ago
Those are user agents, not referrers. In my experience DotBot and BLEXBot obey robots.txt, if a
Disallow directive exits for them. ltx71 ignores robots.txt, and I had to block all of their IP addresses (at the time: 54.174.24.200, 54.175.147.90, 54.175.196.96) via (in my case) iptables rules.– Doug Smythies
19 hours ago
I looked at your previous questions, and realize that you have been having issues for months. It is not clear to me why you are having so many continuing issues with the typical ways of dealing with these type bots. By the way, my iptables rules for ltx71.com are because those 3 IP addresses did not identify themselves as
ltx71 via user agent string. I also have rewrite rules that allow ltx71 to get my robots.txt file, but otherwise tell them they are banned because they don't obey the robots.txt file.– Doug Smythies
11 hours ago
I looked at your previous questions, and realize that you have been having issues for months. It is not clear to me why you are having so many continuing issues with the typical ways of dealing with these type bots. By the way, my iptables rules for ltx71.com are because those 3 IP addresses did not identify themselves as
ltx71 via user agent string. I also have rewrite rules that allow ltx71 to get my robots.txt file, but otherwise tell them they are banned because they don't obey the robots.txt file.– Doug Smythies
11 hours ago
"Those are user agents" - You're right... sorry I got confused after all this googling. These are the ones that are in my robots.txt, but they ignore the directive. At the end after the old thread I created a dynamic robots.txt that updates simultaneously in all my sites, and contains about fifty working rules. Until some time ago, I added with a script all the IPs of unwanted bots to UFW, but at one point I arrived at thousands of IP, and I had to look for another way. Of ltx71 I have two new ones only today (35.192.84.7, 104.155.179.226).
– alebal
35 mins ago
"Those are user agents" - You're right... sorry I got confused after all this googling. These are the ones that are in my robots.txt, but they ignore the directive. At the end after the old thread I created a dynamic robots.txt that updates simultaneously in all my sites, and contains about fifty working rules. Until some time ago, I added with a script all the IPs of unwanted bots to UFW, but at one point I arrived at thousands of IP, and I had to look for another way. Of ltx71 I have two new ones only today (35.192.84.7, 104.155.179.226).
– alebal
35 mins ago
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "89"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2faskubuntu.com%2fquestions%2f1116001%2fblock-badbot-with-fail2ban-via-user-agents-in-access-log%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Ask Ubuntu!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2faskubuntu.com%2fquestions%2f1116001%2fblock-badbot-with-fail2ban-via-user-agents-in-access-log%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Those are user agents, not referrers. In my experience DotBot and BLEXBot obey robots.txt, if a
Disallowdirective exits for them. ltx71 ignores robots.txt, and I had to block all of their IP addresses (at the time: 54.174.24.200, 54.175.147.90, 54.175.196.96) via (in my case) iptables rules.– Doug Smythies
19 hours ago
I looked at your previous questions, and realize that you have been having issues for months. It is not clear to me why you are having so many continuing issues with the typical ways of dealing with these type bots. By the way, my iptables rules for ltx71.com are because those 3 IP addresses did not identify themselves as
ltx71via user agent string. I also have rewrite rules that allow ltx71 to get my robots.txt file, but otherwise tell them they are banned because they don't obey the robots.txt file.– Doug Smythies
11 hours ago
"Those are user agents" - You're right... sorry I got confused after all this googling. These are the ones that are in my robots.txt, but they ignore the directive. At the end after the old thread I created a dynamic robots.txt that updates simultaneously in all my sites, and contains about fifty working rules. Until some time ago, I added with a script all the IPs of unwanted bots to UFW, but at one point I arrived at thousands of IP, and I had to look for another way. Of ltx71 I have two new ones only today (35.192.84.7, 104.155.179.226).
– alebal
35 mins ago