1

My access.log are full of:

Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)
CCBot/2.0 (http://commoncrawl.org/faq/)
Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)
Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)

and of:

85.55.237.132 - - [16/Jul/2018:06:30:44 +0000] "GET /wp-login.php HTTP/1.1" 404 11462 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
51.253.65.140 - - [16/Jul/2018:06:30:47 +0000] "GET /wp-login.php HTTP/1.1" 404 7645 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
113.161.75.37 - - [16/Jul/2018:06:30:57 +0000] "GET /wp-login.php HTTP/1.1" 404 12145 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
114.94.208.2 - - [16/Jul/2018:06:31:04 +0000] "GET /wp-login.php HTTP/1.1" 404 11096 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
112.134.172.163 - - [16/Jul/2018:06:31:07 +0000] "GET /wp-login.php HTTP/1.1" 404 11177 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
144.48.75.34 - - [16/Jul/2018:06:31:13 +0000] "GET /wp-login.php HTTP/1.1" 404 7736 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"

And none of my sites is wordpress.

Currently I block access to these via UFW blocking the IPs, but they have become thousands and every day the number increases.

Is there any way to say to Ubuntu for example:

If the user-agent (or requested URL) contains "Mjbot" block it.

I've seen several tools like Fail2ban, modsecurity, etc, but they seem to complicate things rather than simplify them.

I'm looking for a really simple solution.

Amith KK
  • 13,547
alebal
  • 473

3 Answers3

2

This can be done easily by the use of mod-rewrite. Just modify the start of your .htaccess to say

<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteCond %{HTTP_USER_AGENT} *Mjbot* [NC]
  RewriteRule .* - [F,L]
</IfModule>

You can replace *Mjbot* with a regex of your choice

The RewriteCond directive sets up a condition for the url to be 'rewritten' or modified. In this case, it checks if the variable HTTP_USER_AGENT matches the regular expression *Mjbot* with no case sensitivity [NC]

The RewriteRule directive tells apache to respond to all requests under this .htaccess (.*) to be responded with a 403 Forbidden (the F flag) and to stop the rewriting process immediately after (the L flag)

Furthermore you can also prevent such requests from cluttering your access log by use of the already packaged mod_setenvif

To do that, to your .htaccess add this line:

SetEnvIf User-Agent "*Mjbot*" dontlog
Amith KK
  • 13,547
0

I think it's not working...

I have this rules in apache2.conf/httpd.conf

RewriteCond %{REQUEST_URI} wp-login.php$ [NC]
RewriteRule .* - [F,L]

If I text a URL with curl, I see this:

curl -Is https://www.alebalweb-blog.com/wp-login.php | head -n 1
HTTP/2 404

Shouldn't I see 403?

In fact if I add the same rule directly on the .htaccess of the site I see this:

curl -Is https://www.alebalweb-blog.com/wp-login.php | head -n 1
HTTP/2 403

So I did a little bit of research on the URL rewrite rules in apache2.conf/httpd.conf, and found lots of confusing people and a lot of confusion.

Among the many things some create a bit of clarity but also other confusion, such as this: http://tltech.com/info/rewriterule-in-htaccess-vs-httpd-conf/

Here it is said that the rules of url rewrite must be enclosed between tags: 

<Directory></Directory>

But wanting to use apache2.conf/httpd.conf I did not understand in which, there are already 3 in my apache2.conf: 

<Directory />
<Directory /usr/share>
<Directory /var/www/>

I tried to add a tag:

<Directory>

around my code at the beginning of the file, Apache didn't like it.

So I tried so:

<Directory />

apache does not complain, but it does not seem to work.

So I tried so:

<Directory /var/www/>

again apache does not complain, but it does not seem to change anything.

Then I tried, not to add a tag, but enter the URL rewrite rules directly in the existing tags, but apache did not like.

Then I searched again and found this: https://stackoverflow.com/questions/344588/how-can-i-implement-a-global-rewritecond-rewriterule-in-apache-that-applies-to

Here they talk about new options for url rewrite: InheritDown, InheritDownBefore, IgnoreInherit.

But again, wanting to use apache2.conf/httpd.conf, where should I put these new options and where my rules?

I'm sure all this can work, but I need to figure out where every single piece has to go to make everything work properly.

Can someone help me please?

alebal
  • 473
0

Maybe you should buy a book about fail2ban, if there is one.

You posted some more questions in an answer. Wouldn't it make more sense to work on your initial question?

If you want to know my answer/opinion: Change from Apache to Nginx and do something like that:

if ($http_user_agent ~* "AhrefsBot")
{
    rewrite .* /error.php?e=403 last;
}

In error.php you can get more data from the requester via $_SERVER and mail it to you or better put it in a log file.

It is even more fun to prohibit SQL in GET parameters:

if ($query_string ~* "(alter|delete|update|select)(?![a-z._-])")
{
    set $lalala "bad_main_word";
}

if ($query_string ~* "1=1|ALL|DISTINCT|ON|AS|FROM|WHERE|GROUP|BY|HAVING|WINDOW|UNION|INTERSECT|EXCEPT|ORDER|ASC|DESC|USING|NULLS|FIRST|LAST|LIMIT|OFFSET|ROW|ROWS|FETCH|NEXT|ONLY|FOR|NO|KEY|SHARE|OF|NOWAIT|LATERAL|NATURAL|TABLE|CURRENT|RETURNING")
{
    set $lalala "${lalala}_and_bad_sub_word";
}

if ($query_string ~ "Put some RegEx here to whitelist your stuff, check your mail/log!")
{
    set $lalala "whitelist";
}

if ($lalala = "bad_main_word_and_bad_sub_word")
{
    rewrite .* /error.php?e=403 last;
}

Of course, you still need the following.

location = /error.php {
    alias /var/www/error.php;
    fastcgi_index error.php;
    fastcgi_pass php;
    include fastcgi_params;
    internal;
}

error_page 403 /error.php?e=403;
uav
  • 413