Monat: Dezember 2021

Follow up – Docker and fail2ban – How I solved it (for me)

| tkaefer

Since my post about Docker and fail2ban quite a lot of time has passed (since August 2019), but the page gets still most of the attention on my blog.

I did quite some more work on that due to several reasons.

First of all, I had severe performance issues as soon as ther were too many ips blocked. The iptables are not very good when they need to handle quite some rules.

Adding a new rule for every ip being blocked, is a pretty bad idea.

Specially when all traffic is passing the rules sometime twice.

I do just hook into three different chains:

  • INPUT
  • FORWARD
  • DOCKER-USER

Normally FORWARD would be suffiecient, but docker also faciliates the FORWARD chain. For me it is not deterministic how this actually behaves.

Therefore I’m also hooking into the DOCKER-USER chain (see https://docs.docker.com/network/iptables/ for details).

I also use ipset and iptables to reduce the number individual iptables rules. And there is a single ipset for all ips to be blocked.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
[0] # cat /etc/fail2ban/action.d/iptables-mangle-allports-ipset.conf
# Fail2Ban configuration file
#
# Author: Cyril Jaquier
# Modified: Yaroslav O. Halchenko <debian@onerussian.com>
#           made active on all ports from original iptables.conf
#           Tobias Kaefer <tobias@tkaefer.de>
#
#
 
[INCLUDES]
 
before = iptables-common.conf
 
 
[Definition]
 
# Option:  actionstart
# Notes.:  command executed once at the start of Fail2Ban.
# Values:  CMD
#
actionstart = ipset create f2b-<name> hash:net forceadd
              <iptables> -t filter -I INPUT -p <protocol> -m set --match-set f2b-<name> src -j REJECT --reject-with icmp-host-unreachable
              <iptables> -t filter -I FORWARD -p <protocol> -m set --match-set f2b-<name> src -j REJECT --reject-with icmp-host-unreachable
              <iptables> -t filter -I DOCKER-USER -p <protocol> -m set --match-set f2b-<name> src -j REJECT --reject-with icmp-host-unreachable
 
# Option:  actionflush
# Notes.:  command executed once to flush IPS, by shutdown (resp. by stop of the jail or this action)
# Values:  CMD
#
actionflush = ipset flush f2b-<name>
 
# Option:  actionstop
# Notes.:  command executed at the stop of jail (or at the end of Fail2Ban)
# Values:  CMD
#
actionstop = <iptables> -t filter -D INPUT -p <protocol> -m set --match-set f2b-<name> src -j REJECT --reject-with icmp-host-unreachable
             <iptables> -t filter -D FORWARD -p <protocol> -m set --match-set f2b-<name> src -j REJECT --reject-with icmp-host-unreachable
             <iptables> -t filter -D DOCKER-USER -p <protocol> -m set --match-set f2b-<name> src -j REJECT --reject-with icmp-host-unreachable
             <actionflush>
             ipset destroy f2b-<name>
 
 
# Option:  actioncheck
# Notes.:  command executed once before each actionban command
# Values:  CMD
#
# actioncheck = <iptables> -t filter -n -L <chain> | grep -q 'f2b-<name>[ \t]'
 
# Option:  actionban
# Notes.:  command executed when banning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionban = /usr/local/bin/ipset-fail2ban.sh add f2b-<name> <ip>
 
# Option:  actionunban
# Notes.:  command executed when unbanning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionunban = /usr/local/bin/ipset-fail2ban.sh del f2b-<name> <ip>
 
[Init]

There were comments about „-j REJECT –reject-with icmp-host-unreachable“ not being available on certain systems and therefore „-j DROP“ was used. Which should be fine. They both pervent any more data being routed to the services – the meaning is different though.

I also use a generic shell script to ban or unban an IP for a given fail2ban jail (/usr/local/bin/ipset-fail2ban.sh):

1
2
3
4
5
6
7
8
9
10
11
12
[0] # cat /usr/local/bin/ipset-fail2ban.sh
#!/bin/bash
 
ipsetcommand="$1"
ipsetname="$2"
IP="$3"
 
if [[ "del" == ""${ipsetcommand}"" ]]; then
  /usr/sbin/ipset test "${ipsetname}" "${IP}" && /usr/sbin/ipset "${ipsetcommand}" "${ipsetname}" "${IP}"
else
  /usr/sbin/ipset test "${ipsetname}" "${IP}" || /usr/sbin/ipset "${ipsetcommand}" "${ipsetname}" "${IP}"
fi

It does several things:

  1. For delete
    1. Check whether the IP is in the ipset
    2. Delete if it is in the ipset
  2. For add
    1. Check whether the IP is in the ipset
    2. Add if it is not in the ipset

The jail mail.conf looks something like this:

1
2
3
4
5
6
7
8
9
10
[0] # cat /etc/fail2ban/jail.d/mailserver.conf
# 3 ban in 1 hour > Ban for 1 hour
[mailserver]
enabled = true
filter = mailserver
logpath = /var/log/syslog
maxretry = 2
findtime = 86400
bantime = 86400
banaction = iptables-mangle-allports-ipset[name="mailserver"]

And the filter looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[0] # cat /etc/fail2ban/filter.d/mailserver.conf
# Fail2Ban configuration file
[Definition]
 
# Option: failregex
# Filter "client login failed" in the Syslog
 
failregex = .* client login failed: .+ client:\ <HOST>
 
# Option: ignoreregex
# Notes.: regex to ignore. If this regex matches, the line is ignored.
# Values: TEXT
#
ignoreregex =

The docker compose logging hasn’t been changed since my last blog post about that topic.

I am also using blocklist ipsets to eliminate already known malicious IPs with a cron job running this script here:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
[0] # cat /usr/local/bin/blockSubnets.sh
#!/bin/bash
 
fail2banjail="mailserver"
 
IPS=""
 
WHITELIST="0.0.0.0/8 10.0.0.0/8 100.64.0.0/10 127.0.0.0/8 169.254.0.0/16 172.16.0.0/12 192.168.0.0/16 255.255.255.255/32"
 
SOURCE_URLS="http://lists.blocklist.de/lists/strongips.txt https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset"
 
# There a several other lists to be considered, like:
#  https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/dshield_7d.netset \
#  https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/greensnow.ipset \
#  https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset"
#   \
#  https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/darklist_de.netset \
#  https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_abusers_1d.netset"
 
 
for SOURCE_URL in ${SOURCE_URLS}; do
  CURRENT_IPS=$(curl -s ${SOURCE_URL} | grep -v '^#')
  IPS="${IPS} ${CURRENT_IPS}"
done
 
IPS="$(echo ${IPS} | sort -u)"
 
for IP in ${IPS}; do
  # echo "${IP}";
  if  [[ "${WHITELIST}" == *"${IP}"* ]]; then
    echo "not blocking ${IP}"
  else
    /usr/sbin/ipset --test "f2b-${fail2banjail}" "${IP}" || /usr/bin/fail2ban-client set "${fail2banjail}" banip "${IP}" &> /dev/null
  fi
done
 
## You might also want to add the IP from your cable/DSL/fiber connection at home to not block yourself out, like:
/usr/bin/fail2ban-client set mailu addignoreip $(/usr/bin/dig +short A <<<mydyndnsipv4name.dyndnsprovider.tld>>>)
/usr/bin/fail2ban-client set mailu addignoreip $(/usr/bin/dig +short AAAA <<<mydyndnsipv6name.dyndnsprovider.tld>>>)

Please replace „mydyndnsipv4name.dyndnsprovider.tld“ and „mydyndnsipv6name.dyndnsprovider.tld“ with an appropriate dns record for your cable/DSL/fiber connection.