Blahaj.zone experienced a security breach and is handling it to properly reduce the risk of harm to their users. the current eta for their reture is in about 7 hours.
Really appreciate the transparency. Time to change my blahaj password!
Excellent writeup, and I appreciate the transparency. I have some suggestions on how to mitigate something like this from happening in the future.
- Use a separate DBMS (that is, a separate postgres/mariasql/etc container) for each service. Give each one service unique passwords, which you can define in the docker compose.
This is simpler than trying to control postgres permissions granularity. Even if one application that connects to a database gets owned, it doesn’t have access to other postgres databases, preventing data leaks/exfiltration.
- Use a virtual machine or application container based runtime for your containers.
Kata containers is a container runtime, that is virtual machine.
There is also Gvisor and Syd Box, which are application kernels. Application kernels are reimplimentations of the parts of the Linux kernel needed to run apps, and in this case both Gvisor (Go) and Syd Box (Rust) are in memory safe langauges.
Kata containers are faster, but you will need nested virtualization in order to use them. Application kernels are slower, but you can install them anywhere, including hosts where virtualization is disabled (like a VPS that doesn’t let you enable nested virtualization.
Both take a tiny bit more resources intensive due to no longer being able to share the host kernel, but for most part, it is worth it. They don’t bring an entire kernel along, just what is needed to run apps.
Both offer similar levels of isolation, and preventing applications running inside them from touching the host kernel directly. They effectively manage to prevent issues like copy fail, dirty frag, and so on, from owning your host.
They are fairly easy to install, docker has some docs here: https://docs.docker.com/engine/daemon/alternative-runtimes/ . But if you are using podman or kubernetes, you can also install them there.
- Enable automatic security updates (and reboots) on stable distros.
A large part of the draw of stable Linux like Debian or Red Hat, is that they only do security updates. They don’t do feature updates, or even bug fixes (except for critical ones). In doing so, there is essentially a guarantee of reliability, where it is impossible for updates to break anything.
This makes it possible to enable automatic security updates, and you can even configure it to automatically reboot in order to load a new kernel that includes mitigations against issues like dirty frag. Make sure your docker containers are configured to automatically restart and everything will be smooth.
“Just patch” is a good but it is never enough, and I am frustrated hearing it so frequently. The way I view it is, any time I have to patch, what I really need to do is to improve my security architecture so I never have to “patch” this specific issues again. Patches are the exact kind of security toil that I complain about in this comment.
Use a separate DBMS (that is, a separate postgres/mariasql/etc container) for each service. Give each one service unique passwords, which you can define in the docker compose.
unique passwords is good practice, but separate db server for each of the services is extreme. it brings much more resource consumption. the solution here is being subscribed to security releases and updating soon. those application kernels also sound like a good idea. and as I understand, postgres permissions were not at fault, the permission system had a bug.
Even if one application that connects to a database gets owned, it doesn’t have access to other postgres databases, preventing data leaks/exfiltration.
except that because of the bug, anyone with query permission could have become postgres superuser.
Then they transfered a file to /tmp/exp which was linux kernel CVE-2026-43500, nicknamed ‘Dirty Frag’, an RxRPC local privilege escalation. I had not patched these internal servers that nobody should have access to against this.
Lessons Learned #1:
Install your patches.
“But I have a firewall!”
That is not a sufficient control.
Install.
Your.
Fucking.
Patches!“Just patch” is advice for a windows administrator, where updates break everything so you have to sit and baby them and apply them manually.
On Linux, there are ways to enable automatic security updates, including automatic reboots, so you can safely receive the mitigations your distro provides. That way, you don’t have to worry about forgetting to patch (until the distro release becomes unmaintained, at least).
Now, dirty frag was a zero day, meaning that it was released and probably in the wild before a mitigation was pushed out to handle it. So you did need to apply an actual configuration patch… unless you had some form of kernel based isolation, which I mention as #2 of my other comment in this thread: https://programming.dev/post/52129409/24414213
“Should” is a four-letter word in fields like safety and security.
Oof, hope it works out for them 😣
Thanks for sharing the details 👍
Oof, that sucks :(
Fookin’ hell. Nightmare scenario.
Damn, i hope the femcelmemes crew is fine
well, there goes 75% of shitposts on lemmy.
Also a vital support network for LGBT+ people, I guess?
Also a vital support network for LGBT+ people, I guess?
Mhm and the instance where the only women’s community is hosted :(
i know this isn’t my place as a cis white dude, but can anarchist.nexus be a resource for a women’s space? i’ve been looking for gaps in the threadiverse we could nurture and foster. my #1 concern lately with the makeup of the hegemony of the threadiverse lately has been:
- eurocentricism
- authoritarianism by default
- black and white thinking (see above)
- unchecked and unchallenged mysoginy
my commitment to you as a cis white dude admin would be to stay the fuck out of the way of moderators looking to host a women’s issues community here, within reason. obviously it wouldn’t be okay to break instance rules against bigotry and such, but i can offer my commitment to shutting my mouth and knowing my role
edit: also, i bet @poVoq@slrpnk.net and @ProdigalFrog@slrpnk.net from my second original home instance would be happy to have a women’s issues community hosted on slrpnk.net, i’m still figuring out what my new relationship and responsibility to the threadiverse is now, but i think just like how there’s more than one world news community, there could be more than one women’s issues community so that people have a preferred and fallback space on days like today. the one thing i don’t want to diminish in this comment is the incredible work people like @ada@lemmy.blahaj.zone have done in creating a welcoming space on the threadiverse for marginalized people. i just don’t want that to be work that is only done alone. we should be collaborative towards a better, freer, world
There used to be a women’s space on slrpnk.net (although a bit unfortunately named after a controversial Reddit community), but we could never really find women willing to moderate it longer term and the amount of dudes showing up with trollish comments became a bit too much for us admins to handle.
the amount of dudes showing up with trollish comments became a bit too much for us admins to handle.
The mods in womensstuff work really, really hard.
Maybe! LadyButterfly is the mod of the womensstuff community and she does a great job, maybe we can get this info to her.
I really appreciate your acknowledgement of some of the issues with the diversity in the threadiverse. And appreciate your support <3
@LadyButterfly@piefed.blahaj.zone , when your instance is online again let’s have a discussion about what resources i can make available to you that will center your community’s needs without someone like me colonizing the discussion.
(unless i have the wrong ladybutterfly, then my apologies)
That’s her :)
I’ve been wondering why it’s down, that’s my home instance. That’s unfortunate :/
Bummer! I was wondering what had happened. Thanks for the update!
This is horrible. And I say this as someone who’s had horrible experiences with users over there, particularly when it comes to bullying and harassment. They didn’t deserve to get shut down like this. Hope they’re able to fix whatever the problem is and get back up and running again.
