/nix on a subvolume or bindmount

[chewblacka]

Hi, I would like to keep /nix separate from my root file system. Two ways I can accomplish this are to either:

A. Put it in it's own btrfs subvolume
B. Bind mount /nix at boot to another file system

Unfortunately both of these solutions lead to 2 issues when using the Determinate Systems nix-installer. Namely:

1. the volumes aren't mounted by the time systemd tries to execute the nix-daemon symlinks which point to a non-existent volume at this point in the boot, so the nix-daemon has to be started manually.
2. During uninstall the installer tries to revert the action "create directory /nix" which it can't do (bind mounted dir or subvolume can't just be deleted) so it reports an error.

Number 2 is pretty trivial (can be ignored), but ideally the installer would recognize during install that /nix was pre-existing, so during uninstall it would leave it place.
Number 1 I'm not sure about. My current fix is to put your installer in a bash wrapper script which copies the 2 systemd files (rather than symlink them). This works, but means I have to install / uninstall using my script only.

Longer term I think quite a few people might want to have /nix either on a subvolume or bindmount so this could be an issue worth fixing IMHO. Many thanks!

[Hoverbear]

Hi @chewblacka !

This is something I'd like to support too! In fact, I think it may be required for things like #389 .

One strategy here would be to have the CreateDirectory action detect if something is a mountpoint if a directory exists, then set its ActionState to ActionState::Skipped... however that does not solve the service ordering issue.

Another approach would be to add a --nix-mount option which ensures the /nix path is a mountpoint and adds the RequiresMountsFor systemd setting: https://www.freedesktop.org/software/systemd/man/systemd.unit.html#RequiresMountsFor=

[chewblacka]

Hi @Hoverbear many thanks for the reply! I've been tinkering with the symlinked systemd unit files today, and try as I might I can't get them to work. The systemd unit file already contains several RequiresMountsFor options. The problem is the unit file never gets seen by systemd as it's a symlink to an unmounted volume.

I even created an explicit systemd unit file to mount /nix (rather than via fstab), and then have that unit try and load the nix-daemon but that didn't work either. Maybe I was doing something wrong? But the simplest solution seems to be to simply copy the files rather than symlink them. I was hoping for a fancier solution but couldn't find one. Hope you have better luck!

Many thanks!

[Hoverbear]

Ah! That sounds really similar to what we do on the Steam Deck where we have a unit which reloads the units after the mounts:

nix-installer/src/planner/steam_deck.rs

Lines 180 to 198 in 4cc8326

	let ensure_symlinked_units_resolve_buf = format!(
	"\
	[Unit]\n\
	Description=Ensure Nix related units which are symlinked resolve\n\
	After=nix.mount\n\
	Requires=nix-directory.service\n\
	Requires=nix.mount\n\
	DefaultDependencies=no\n\
	\n\
	[Service]\n\
	Type=oneshot\n\
	RemainAfterExit=yes\n\
	ExecStart=/usr/bin/systemctl daemon-reload\n\
	ExecStart=/usr/bin/systemctl restart --no-block nix-daemon.socket\n\
	\n\
	[Install]\n\
	WantedBy=sysinit.target\n\
	"
	);

[chewblacka]

Many thanks @Hoverbear ! I followed your steam-deck install and copied your systemd units, which work great on a bind mount. Just a slight heads up. In your steam-deck install, in the nix.mount systemd unit, the following lines are placed in the [Unit] section which causes an error and the lines are ignored. Should go in the [Install] section I believe.

        RequiredBy=nix-daemon.service\n\
        RequiredBy=nix-daemon.socket\n\

[Hoverbear]

Great catch, thanks!

[Hoverbear]

Yeah, definitely a problem!

[bam80]

Progress?

[moonpiedumplings]

Similar issue here. I attempted to workaround this on my own my only having /nix/store be on a separate subvolume, and having /nix be on the main root subvolume. However, then the installer fails, presumably because /nix/store already exists.

I'll probably go the path of either copying over the systemd unit files, or the daemon-reload after mount, but being able to have /nix/store on a separate subvolume would be cleaner.

Proceed? ([Y]es/[n]o/[e]xplain): Y
 INFO Step: Create directory `/nix`
 INFO Step: Provision Nix
ERROR
   0: Install failure
   1: Error executing action
   2: Action `provision_nix` errored
   3: Action `move_unpacked_nix` errored
   4: Rename `/nix/temp-install-dir/nix-2.23.3-x86_64-linux/store/1rkhjf55x59w6qm1pbhf80ks2wjpg973-libcpuid-0.6.4` to `/nix/store/1rkhjf55x59w6qm1pbhf80ks2wjpg973-libcpuid-0.6.4`
   5: Cross-device link (os error 18)

Location:
   src/cli/subcommand/install.rs:253

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.

Consider reporting this error using this URL: https://github.com/DeterminateSystems/nix-installer/issues/new?title=%3Cautogenerated-issue%3E&body=%23%23+Error%0A%60%60%60%0AError%3A+%0A+++0%3A+Install+failure%0A+++1%3A+Error+executing+action%0A+++2%3A+Action+%60provision_nix%60+errored%0A+++3%3A+Action+%60move_unpacked_nix%60+errored%0A+++4%3A+Rename+%60%2Fnix%2Ftemp-install-dir%2Fnix-2.23.3-x86_64-linux%2Fstore%2F1rkhjf55x59w6qm1pbhf80ks2wjpg973-libcpuid-0.6.4%60+to+%60%2Fnix%2Fstore%2F1rkhjf55x59w6qm1pbhf80ks2wjpg973-libcpuid-0.6.4%60%0A+++5%3A+Cross-device+link+%28os+error+18%29%0A%60%60%60%0A%0A%23%23+Metadata%0A%7Ckey%7Cvalue%7C%0A%7C--%7C--%7C%0A%7C**version**%7C0.20.1%7C%0A%7C**os**%7Clinux%7C%0A%7C**arch**%7Cx86_64%7C%0A
Installation failure, offering to revert...
Nix uninstall plan (v0.20.1)

Planner: linux (with default settings)

Planned actions:
* Remove the directory tree in `/nix`
* Remove the directory `/nix`


Proceed? ([Y]es/[n]o/[e]xplain): n
Okay, didn't do anything! Bye!