I’m thinking of setting up multi user nix on a compute cluster. The advantage would be to have a shared storage where common packages are reused, this is a great advantage compared to conda where every environment duplicates storage and inodes.
However, the packages are installed as root. As such I’m a bit wary of whether a user installing something could have the system run malware as root by installing a package.
What are the safeguards in place and how do I know I can trust them?
However, the packages are installed as root. As such I’m a bit wary of whether a user installing something could have the system run malware as root by installing a package.
I’m pretty sure the packages are built as unprivileged users in a sandbox, and you can also restrict what users can use certain Nix features.
The main security flaw with a multi-user Nix system is that everything you build ends up in a world-readable store.
I’m pretty sure the packages are built as unprivileged users in a sandbox, and you can also restrict what users can use certain Nix features.
Are Nix packages sandboxed by default?
At runtime no, you’d need to use Nixpak or a similar tool
A user only installs packages into their user environment (and into the nix store). It won’t be accidentally run by root.
Assuming you’re talking about Nix the package manager,
Nothing is actually “installed as root”, except for stuff that the root user installs themselves. The Nix Store (where all the “derivations”/packages are stored) can be thought of more as a cache, just because a package is there doesn’t mean it’s used anywhere. Users don’t get to choose the cache “key” (i.e. the directory name in /nix/store) either - it is determined by all the build instructions and dependencies needed to build the package, and Nix doesn’t (well, at the very least shouldn’t) give users any control about the package build process after it starts.
When users install something, Nix fetches or builds that package into
/nix/store- which doesn’t affect other users in any way - and then just symlinks that particular package into some user-owned directory in their$PATH(e.g.$HOME/.local/share/nix/profiles/defaultor so), which also doesn’t affect root or other users in any way.So, basically, if some user installs malware only they are affected - the fact that it’s also in the shared
/nix/storeis irrelevant since there’s nothing in other user’s profiles or$PATHor whatever that references it in any way.The most likely vulnerability is something like this:
- The attacker guesses the nix store path of some package that root will use in the future (e.g. glibc from a more up-to-date Nixpkgs version compared to what root currently uses) - note that users can’t replace a derivation that’s already in the store, so they need to guess a future derivation path
- The attacker forces Nix to build that package from source instead of fetching from a trusted substituter
- The attacker finds a way to breach Nix’s sandbox during the build and inject their own backdoor into the resulting package ← this is the difficult part, there are currently no such known sandbox holes
- The attacker then waits until root starts using that package in the store, at which point the backdoor becomes actively ran as root.
I must add that this is theoretical and I don’t think has ever happened in practice on a multi-user system.
Sorry, I’m pretty bad at explaining stuff, hopefully it makes some sense :)
Note that any user listed in
nix.settings.trusted-userscan place arbitrary directories into the Nix store, and can choose the store key.Hello, thank you for the in depth explanation! I am not very familiar with nix, and yes I’m talking about the package manager. I have installed NixOs recently to toy around with it and see whether it could be a good solution to certain problems we have. I have determined that NixOs itself is not mature enough for what we need, but I do see value in the nix package manager that other solutions do not offer.
Regarding the 4 points you raise
- I’m not too worried about hash squatting, I guess there’s no way around it, but other ways of managing packages have way worse flaws than that.
- I do want my users to be able to build packages. In most cases I imagine we’ll use pre built packages, but often packages available are not optimized for certain architectures.
- How does the nix sandbox work? Just so I know what I’m working with. Do you know where I can read about it?
- Does root actively run installed packages at any point? If that is the case I’d be a bit wary to use nix for this purpose.
I’d like to note that the 4 points are not separate issues, they are all requirements in order for an attacker to get access.
I’m not too worried about hash squatting, I guess there’s no way around it, but other ways of managing packages have way worse flaws than that.
“hash squatting” is not really an issue on its own. Unless an attacker can breach the sandbox, given the same derivation hash, the resulting derivation output would be the same regardless of which user requested the build.
I do want my users to be able to build packages. In most cases I imagine we’ll use pre built packages, but often packages available are not optimized for certain architectures.
Actually, Nix without the ability to build derivations would just be useless. In Nix, everything is a derivation, even if you just want to have multiple packages installed into your user profile at once, behind the scenes it’s a derivation which simply combines binaries from multiple other derivations, which will not be available in the remote cache and needs to be “built” (in the most trivial sense possible) locally.
How does the nix sandbox work? Just so I know what I’m working with. Do you know where I can read about it?
You can think of the Nix sandbox as a kind of lightweight container, into which Nix mounts all the dependencies needed to build the derivation (including source code and such), and then runs the builder command. On Linux it is using User Namespaces (and a few other sandboxing tools), in a similar way to what Docker does (in fact there is some work to use
runcas a sandbox backend but it’s not usable yet). I don’t know if it’s described in a lot of detail anywhere, but a brief description is available in the manual.Does root actively run installed packages at any point? If that is the case I’d be a bit wary to use nix for this purpose.
Once again, just random packages in Nix Store are not “installed” in any real sense. In order for any user (incl. root) to install/run something, they have to be the one explicitly doing it, e.g. via
nix profile installornix shell. That also means that if you’re not using NixOS, you can actually just not do anything as a root user and then no code from/nix/storewill be executed with root privileges at all.To reiterate, unless your threat model involves sophisticated attackers[1], users installing packages with Nix is as secure as them just building code from source manually, or dropping binaries into their own
$HOME/.local/bin. One user installing a package into their profile does not affect other users or root in any way.From what I can tell from your other replies, you are setting up a common build server with your coworkers. In that case I’d say it’s 100% fine, both workplaces where I’ve been doing Nix work had a server like this, there were never any issues with privilege escalation or malware spreading between users.
The attackers need to have access to an unpatched 0-day Nix sandbox breach or a critical nix-daemon issue, which is very unlikely. If that makes you feel any safer, I know that Nix has been scanned for vulnerabilities by at least one frontier-class LLM with no such vulnerabilities discovered, so just a script-kiddie with access to ChatGPT won’t be able to find anything like that. ↩︎
Thank you, you reassured me quite a bit. I’ll have a good read of the manual.
It is not really a build server, it’s a compute server. I’ll have users installing hundreds of different software packages, most of them using incompatible libraries and thus I was thinking of using Nix. Alternatives would be LMOD, which however requires the administrator to install very single piece of software, which… We don’t really have a dedicated system administrator and I don’t really want to go through the compilation instructions of every single piece of software we use. Alternatively everyone could compile their own software or install it through something like conda, but that eats up a lot of storage space since every library would be duplicated across users.
Well, in that case remember to force everyone to use the same Nixpkgs version (otherwise you’ll still end up with a lot of duplicated packages), and run
nix-store --optimise; nix-collect-garbage -dregularly (ideally via cron or something).
If you mean Nix, the package manager, then no packages are actually installed as root. Every package is installed in the Nix store, but programs that need root privileges can’t be run by any user.
One of the (many) great things about Nix is that users don’t need root privileges to install packages.
Thank you for the explanation. Yes, I’m talking about the nix package manager, not about the operating system.
As far as I understand packages are built by less privileges users, but libraries and binaries are owned by the root user in the nix store.
https://nix.dev/manual/nix/2.34/installation/multi-user.html
What I wonder is wheter this could lead to root executing untrusted code. I will have several users installing packages and I can not trust that they will check every package. Thus I’d like to know whether I can trust the system to be resilient without careful attention.
If one user installs a modified version of a bin or lib it won’t collide with the others.
In theory it is safe. When a Nix package is built it isn’t “installed”. Unless root is running/installing random packages out of the Nix store there is no problem. As long as the user’s aren’t added to the
trusted-usersoption they shouldn’t be able to cause any problems for other users.However like any multi-user system you are sharing a Linux kernel. A kernel is a very complex piece of software with a huge attack surface. Privileged escalation vulnerabilities are commonly found. (This also applies to the nix-daemon, but it is a bit smaller attack surface but vulnerabilities are still occasionally found.) So you shouldn’t assume strong security isolation. I would say that a setup like this is acceptable for mostly-trusted people like coworkers or friends that are not expected to actively exploit vulnerabilities but definitely wouldn’t let random unknown users use the system.
So if you want strong isolation use a VM or separate hardware, but then you won’t be able to share the builds and packages defeating the point in this case.
Thank you for the clear explanation. This is good enough for me I guess then. What I am most afraid is not really a colleague building a malicious package on purpose, but rather them installing something which is then run as root and gets access to the whole system. I understand packages are built by unprivileged users, but in the end they are owned by root.
What I wonder is whether in the way it is possible that something is run as root without someone specifically deciding it should be.
Do you want explain a bit more?
Linux is Linux. If you’re using the standard install packages like Pam, then it’s built specifically for multiple users.
Elaborate more if you could.
What’s your point? Nix is meant for CI, but even still the defaults are set for multi-user operations.
Nix is absolutely not just for CI, the idea is to use it locally as well, so the CI runs in the same environment.
Are you talking about NixOS?
You can install post user nix home manager distributions which are separate from the shared nix environment (not sure if that’s the right term)
