Docker and fail2ban – How I solved it (for me)

Docker is great when running your own services in an isolated ephemeral setup.
I’ve been using this pattern for quite a while now (approx since Linux VServer had been introduced, afterwards with plain LXC and then with Docker).

But SMTP, IMAP and other services are very attractive to not so nice people. And one might want to add some security to the services, eg. using layer 7 level information to block network traffic on layer 3/4.
So as soon as there are like bruteforce login attacks, drop the eg. TCP/IP paket for the IP of the attacker.

Normally you’d use fail2ban out of the box. It provides pretty good detections for the most well known service implementations and integrates the counter measures (eg. iptable based actions) very good into your OS.

With docker this is a little bit different. Docker also uses the iptables network utils for passing the incomming traffic on public IP towards an private IP by DNATting (Destination Network Address Translation) into a docker container private network.

Let’s have a look at the iptables chain flows:

packet_flow10.png
iptables chaining flow, source: http://xkr47.outerspace.dyndns.org/netfilter/packet_flow/

Let’s check the acual config:

$ iptables -t nat -L


Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
...
Chain DOCKER (2 references)
target prot opt source destination
...
DNAT tcp -- anywhere myhost.tdl tcp dpt:https to:10.10.10.10:443

So at a very early stage the traffic is passed on via DNAT. Solutions you can find for Docker and fail2ban mostly deal with the FORWARD chain.

This did not really work oput for me. Therefore I did setup my own fail2ban action (eg. /etc/fail2ban/action.d/iptables-mangle-allports.conf):

# Fail2Ban configuration file
#
# Author: Cyril Jaquier
# Modified: Yaroslav O. Halchenko <debian@onerussian.com>
#                       made active on all ports from original iptables.conf
#
#

[INCLUDES]

before = iptables-common.conf


[Definition]

# Option:  actionstart
# Notes.:  command executed once at the start of Fail2Ban.
# Values:  CMD
#
actionstart = <iptables> -t mangle -N f2b-<name>
              <iptables> -t mangle -A f2b-<name> -j <returntype>
              <iptables> -t mangle -I <chain> -p <protocol> -j f2b-<name>

# Option:  actionstop
# Notes.:  command executed once at the end of Fail2Ban
# Values:  CMD
#
actionstop = <iptables> -t mangle -D <chain> -p <protocol> -j f2b-<name>
             <actionflush>
             <iptables> -t mangle -X f2b-<name>

# Option:  actioncheck
# Notes.:  command executed once before each actionban command
# Values:  CMD
#
actioncheck = <iptables> -t mangle -n -L <chain> | grep -q 'f2b-<name>[ \t]'

# Option:  actionban
# Notes.:  command executed when banning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionban = <iptables> -t mangle -I f2b-<name> 1 -s <ip> -j DROP

# Option:  actionunban
# Notes.:  command executed when unbanning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionunban = <iptables> -t mangle -D f2b-<name> -s <ip> -j DROP

[Init]

I am now able to DROP the packages at a very eraly stage. Which helped me a lot.
Some jail example for my docker based mail setup:

[emailserver]
enabled = true
filter = mailu
logpath = /var/log/syslog
maxretry = 2
findtime = 72000
bantime = 7200
chain = PREROUTING
banaction = iptables-mangle-allports[name="emailserver", chain="PREROUTING"]

New Alpine Linux based Docker image for WordPress 5.1.1

Photo by Markus Spiske on Unsplash

Since quite a while I do maintain my own version of Docker WordPress images.

At one point in time, there was a lack of stable WordPress updates for Docker images. At that time I did create my own github.com repo and establishes some automated builds at quay.io.

Ever since I am trying to keep up with the WordPress releases, but unfortunately I was not that perfect in the recent time.

But now there is an updated image including:

  • WordPress 5.1.1 (from 5.0.2)
  • Alpine 3.9 (from 3.7)
  • PHP 7.3 (from 7.2)

It would be nice to hear if anybody is using this image and if there are any flaws I did not recognize until now.

Series: Signing Messages for Message Broker – Using Bouncycastle library to read PGP key and sign plain text

Why would you need this?

I am currently investigating how an advanced level of security can be applied to a message based micro service architecture.

One could easily rely on the authentication and authorization of the message broker. But this requires extensive options in the message broker.

So why not add an additional layer, by signing the messages via OpenPGP, GnuPG, PGP or similar patterns.

Therefore you’d also be able to sign the keys and create a trusted group within your artifacts.

As a first step, I’ve analyzed the options in Java to use a GPG public/private key set, in an export file. This seems to be handier than a real GPG trust store. It makes the distribution of the keys easier and less dependend on the base operating system.

Here is some source code…

I’ve used the bouncycastle library to do the heavy lifting of cryptographics, but anyways it is still a little tricky to put all the different pieces together.

Therefore I decided to give a little bit of an idea what has to be done by providing a little code snippet:

        Security.addProvider(new BouncyCastleProvider());

        String input = "Sign Me";
        String passphrase = "test1234";

        long keyId = Long.decode("0x566F1E11219B208A");

        @Cleanup
        InputStream fileInputStream =
                new FileInputStream("/tmp/exported-keys.asc");

        InputStream in = PGPUtil.getDecoderStream(fileInputStream);
        PGPSecretKeyRingCollection pgpSec = new PGPSecretKeyRingCollection(in, new BcKeyFingerprintCalculator());

        PGPSecretKey secretKey = pgpSec.getSecretKey(keyId);

        if (secretKey == null) {
            throw new IllegalArgumentException("Can't find encryption key in key ring.");
        }


        PGPPrivateKey privateKey =
                secretKey.extractPrivateKey(
                        new JcePBESecretKeyDecryptorBuilder()
                                .setProvider("BC").build(passphrase.toCharArray()));
        PGPSignatureGenerator sigGenerator = new PGPSignatureGenerator(
                new JcaPGPContentSignerBuilder(secretKey.getPublicKey().getAlgorithm(), PGPUtil.SHA256)
                        .setProvider("BC"));

        sigGenerator.init(PGPSignature.BINARY_DOCUMENT, privateKey);

        ByteArrayOutputStream buffer = new ByteArrayOutputStream();

        try (ArmoredOutputStream aOut = new ArmoredOutputStream(buffer)) {
            BCPGOutputStream bOut = new BCPGOutputStream(aOut);
            sigGenerator.update(input.getBytes(StandardCharsets.UTF_8));
            sigGenerator.generate().encode(bOut);
        }

        System.out.println(new String(buffer.toByteArray(), StandardCharsets.UTF_8));

a hidden gem in docker 18.06 – define your base CIDR for networks

Where you ever annoyed by the CIDR ranges used when a docker network was created without any further ipam spec (eg. in docker-compose.yaml)?

There is something hidden in the PR https://github.com/moby/moby/pull/36396: You are able to set the subnet CIDR from where docker networks are supposed to be created, plus you are able to define the size of the subnet,

Since I am a big fan of the 100.64.0.0/10 carrier grade NAT segment (it’s huuuge  and a cool alternative to 10.0.0.0/8) and it’s a private network.

So what needs to be done is running dockerd like

dockerd --default-address-pool base=100.96.0.0/11,size=26

or you’ll add something like this to your daemon.json file

{
"fixed-cidr": "100.64.0.0/23",
"default-address-pools":[
{"base": "100.96.0.0/11", "size": 26}
]
}

Notice the plural in the json file – it took me quite a while to add the plural 😉

Unfortunately, this cannot be found in the official dockerd documentation up until now. I just found it as a PR comment (see here).

BI-directional Git to svn sync script

Something that I did create over some years – there is nothing similar that I have found as a open/OSS solution, so therefore I’d like to share it.

#!/bin/bash

if [ -f ~/tmp/git2svn.lock ]; then
 echo "git2svn is already processing..."
 exit 0
else
 touch ~/tmp/git2svn.lock
fi

echo "$(date --iso-8601=minutes) === git2svn.sh sync repos ==="

for repo in $HOME/repos/* ; do
 cd "${repo}"

 git checkout -b svn/git-svn
 git checkout svn/git-svn
 git svn fetch
 git svn rebase
 git checkout master
 git pull --rebase upstream master
 git checkout svn/git-svn
 MESSAGE=`git log --pretty=format:'%ai | %B [%an]' HEAD..master`
 git merge --no-ff --no-log -m "${MESSAGE}" master
 git svn dcommit
 git checkout master
 git merge svn/git-svn
 git push upstream master

done

rm ~/tmp/git2svn.lock

The trick is to use two different local only branches in which you do the merging vice versa. Just using one branch like master will cause serious issues.

„Let’s Encrypt“ here I come.

I just switched from „StartSSL“ certificates to „Let’s Encrypt“ certificates.

Although „StartSSL“ is providing an API to create certificates (I haven’t used it, so I can’t tell anything about it), I made a change this evening in switching to „Let’s Encrypt“ certificates generated in a own nginx reverse proxy setup. I use the image from eforce21/letsencrypt-nginx-proxy for this.

Besides some setup issues (IPv6 and running a Apache somewhere on the Host), it worked really smooth.

That’s what I like about Docker: Putting stuff together in a meaningful way.

Thumbs up.

IPv6 – Docker, serious?

I am still playing around with my Docker setup (so you might wonder why I this website might be down sometime, this is just because I restart some services ;))

The toughest part was IPv6 so far. But it is working – somehow.

You can’t specify something obvious in a docker-compose.yml like

...
  ports:
    - [2000::1]80:80

I won’t talk about the whole findings on Google that are complaining about the issue.

Let’s be constructive:
You need to add a IPv6 subnet to your docker0-bridge interface:

ip -6 r add 2a01:1313:1313:666:1313::/80 dev docker0

You need to change your docker daemon setup to use this subnet, since I am using systemd I’ve created a overriding config file for the docker daemon (eg. /etc/systemd/system/docker.service.d/docker.conf):

[Service]
ExecStart=/usr/bin/docker daemon -H fd:// -g /srv/docker-lib --ipv6 --fixed-cidr-v6="2a01:1313:1313:666:1313::/80"

After a service docker restart (plus some docker-compose up -d calls) you are able to use the IPv6 table assigned from the /80 subnet.

To ensure that you’ll always end up with the same IPv6 address you should probably set the mac_address property in the docker compose file.

I did actually some additional tweaking of the nginx proxy by adjusting some nginx templating.

Now I needed to set the AAAA records – and that’s it. 🙂

How MagicPrefs and a Mac OS X security update are messing up your keychain access usability

I was actually facing some strange issues: Using Mac OS X keychain access app and SSL/TLS client certificates (and other authentication items in keychain) in Chrome or Safari did not work. I saw the „allow“ or „always allow“ buttons; I could klick on them, but nothing happened. It was very strange. It was that strange that I did reset my login keychain – without any impact.

And also Google did not help – unless I stumbled today on a discussion at the Apple forums. There is a reference to a security update from Apple, which includes this change/fix:

SecurityAgent
Available for: OS X El Capitan 10.11
Impact: A malicious application can programmatically control keychain access prompts
Description: A method existed for applications to create synthetic clicks on keychain prompts. This was addressed by disabling synthetic clicks for keychain access windows.
CVE-ID
CVE-2015-5943

Which practically means that any tool that interferes with the input devices is not allowed to grant keychain access rights. So does MagicPrefs.

What a painful thingy – I was almost resetting my whole system from scratch (like the private and business MacBook Pro).

After all – it’s good to query such issues over and over again.