Excellent writeup, and I appreciate the transparency. I have some suggestions on how to mitigate something like this from happening in the future.

1. Use a separate DBMS (that is, a separate postgres/mariasql/etc container) for each service. Give each one service unique passwords, which you can define in the docker compose.

This is simpler than trying to control postgres permissions granularity. Even if one application that connects to a database gets owned, it doesn’t have access to other postgres databases, preventing data leaks/exfiltration.

2. Use a virtual machine or application container based runtime for your containers.

Kata containers is a container runtime, that is virtual machine.

There is also Gvisor and Syd Box, which are application kernels. Application kernels are reimplimentations of the parts of the Linux kernel needed to run apps, and in this case both Gvisor (Go) and Syd Box (Rust) are in memory safe langauges.

Kata containers are faster, but you will need nested virtualization in order to use them. Application kernels are slower, but you can install them anywhere, including hosts where virtualization is disabled (like a VPS that doesn’t let you enable nested virtualization.

Both take a tiny bit more resources intensive due to no longer being able to share the host kernel, but for most part, it is worth it. They don’t bring an entire kernel along, just what is needed to run apps.

Both offer similar levels of isolation, and preventing applications running inside them from touching the host kernel directly. They effectively manage to prevent issues like copy fail, dirty frag, and so on, from owning your host.

They are fairly easy to install, docker has some docs here: https://docs.docker.com/engine/daemon/alternative-runtimes/ . But if you are using podman or kubernetes, you can also install them there.

3. Enable automatic security updates (and reboots) on stable distros.

A large part of the draw of stable Linux like Debian or Red Hat, is that they only do security updates. They don’t do feature updates, or even bug fixes (except for critical ones). In doing so, there is essentially a guarantee of reliability, where it is impossible for updates to break anything.

This makes it possible to enable automatic security updates, and you can even configure it to automatically reboot in order to load a new kernel that includes mitigations against issues like dirty frag. Make sure your docker containers are configured to automatically restart and everything will be smooth.

“Just patch” is a good but it is never enough, and I am frustrated hearing it so frequently. The way I view it is, any time I have to patch, what I really need to do is to improve my security architecture so I never have to “patch” this specific issues again. Patches are the exact kind of security toil that I complain about in this comment.