nix-configs

KO | EN

gitlite — search

#go #golang #javascript #react #python #hacktoberfest #python3 #game #terraform #bash #boilerplate #cli

★ 22 Open GitHub ↗

My Nix ecosystem journey, from the first commit to currently running configs

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# nix-configs

Repository containing my personal Nix (NixOS, Home Manager etc.) configurations.

Basic structure:

- `modules/` - all modules live here, they MUST be turned off by default (side-effect free imports),
  - `modules/nixos` - NixOS specific modules
  - `modules/home-manager` - Home Manager modules
  - `modules/nix-darwin` - nix-darwin modules
  - `modules/shared` - modules & configs shared by more than 1 "target"
    - `modules/shared/universal` - universal modules, often defining options to be handled elsewhere
- `packages/`, some personal/in-house/out-of-band tools

Generally I aim to hide as much as possible behind `*.enable` options.

## Overview

This is an incomplete list of incorporated software/systems:

- disks: ZFS on LUKS through `disko`
- desktop: Sway (Wayland), as much as possible through Home Manager
- shell: Fish
- users: Home Manager
- MISSING: development environment

# Notes

## Tasks

Repository tasks are tracked as one file per task under [`docs/tasks/`](docs/tasks/README.md) —
a finished task is tagged `status: done` in frontmatter and gains a sibling `<task>.done.md`
solution file. See [docs/tasks/README.md](docs/tasks/README.md) for the full convention. (The
legacy root `TASKS.md` is being phased out.)

## Custom ISO installer

see https://bmcgee.ie/posts/2022/12/setting-up-my-new-laptop-nix-style/

```shell
# building the installer from packages/install-iso
set INSTALL_ISO_PATH "$(nom build '.#install-iso' --no-link --print-out-paths --print-build-logs)/iso/kdn-nixos-install-iso.iso"
sudo dd bs=1M if="$INSTALL_ISO_PATH" of=/dev/disk/by-id/usb-SanDisk_Cruzer_Blade_02000515031521144721-0:0 status=progress
sudo dd bs=1M if="$INSTALL_ISO_PATH" of=/dev/disk/by-id/usb-_Patriot_Memory_070133F17AC22052-0:0 status=progress
sudo dd bs=1M if="$INSTALL_ISO_PATH" of=/dev/disk/by-id/usb-Samsung_Portable_SSD_T5_1234567D585A-0:0 status=progress
scp "$INSTALL_ISO_PATH" root@kvm-fa56.lan.etra.net.int.kdn.im:/data/nixos-amd64.iso
# boot the machine and ssh into it
ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null kdn@kdn-nixos-install-iso
```

## Golden path for bootstrapping new physical machine

1. build the `install-iso` and boot it
2. prepare a new config at `host/${HOSTNAME}/default.nix`
    - put `kdn.profile.machine.baseline.enable = true`
    - look through `kdn.hardware.{gpu,cpu}.{intel,amd}`
    - set up `zramSwap` & `boot.tmp.tmpfsSize`
    - figure out missing `boot.initrd.{availableK,k}ernelModules`
3. boot the `install-iso`
4. set up a disk configuration gathering data from the running system,
    ```nix
    (
      # hashtag comment denotes defaults
      # /* */ are just comments
      let
        cfg = config.kdn.disks;
        d1 = "<DISK_1_NAME>-${config.networking.hostName}";
        d1Cfg = cfg.luks.volumes."${d1}";
        d2 = "<DISK_2_NAME>-${config.networking.hostName}";
        d2Cfg = cfg.luks.volumes."${d2}";
      in
      {
        kdn.disks.enable = true;
        #kdn.disks.zpool-main.name = "<HOSTNAME>-main";
        #kdn.disks.devices."${config.kdn.disks.defaults.bootDeviceName}" = { type = "gpt"; content.partitions.ESP = { size = "4096M"; ... }; };
        #kdn.disks.zpools."${cfg.zpool-main.name}" = { };
        #disko.devices.zpool."${cfg.zpool-main.name}".datasets = {
        # "${hostname}/nix-system/nix-store" = { mountpoint = "/nix/store"; ... };
        # "${hostname}/nix-system/nix-var" = { mountpoint = "/nix/var"; ... };
        #}

/* point it at the right detached `/boot` disk (USB flash drive) */
        kdn.disks.devices."${config.kdn.disks.defaults.bootDeviceName}".path = "/dev/disk/by-id/usb-<CORRECT_IDENTIFIER>";
        #disko.devices.disk."${config.kdn.disks.defaults.bootDeviceName}" = { content.type = "gpt"; content.partitions = { ... }; ... };
        #kdn.disks.devices."${d1}" = { type = "luks"; path = d1Cfg.targetSpec.path; };
        #disko.devices.disk."${d1}" = { type = "luks"; name = "${d1}-crypted"; ... };
        #kdn.disks.devices."${d2}" = { type = "luks"; path = d2Cfg.targetSpec.path; };
        #disko.devices.disk."${d1}" = { type = "luks"; name = "${d1}-crypted"; ... };

kdn.disks.luks.volumes."${d1}" = {
          #target.deviceKey = d1;
          targetSpec.path = "/dev/disk/by-id/<DISK_PATH>";
          uuid = "<uuidgen result>";
          #keyFile = "/tmp/${d1}.key";
          #header.deviceKey = config.kdn.disks.defaults.bootDeviceName;
          #header.partitionKey = d1;
          headerSpec.num = 2;
          #zpool.name = cfg.zpool-main.name;
        };

kdn.disks.luks.volumes."${d2}" = {
          #target.deviceKey = d2;
          targetSpec.path = "/dev/disk/by-id/<DISK_PATH>";
          uuid = "<uuidgen result>";
          #keyFile = "/tmp/${d1}.key";
          #header.deviceKey = config.kdn.disks.defaults.bootDeviceName;
          #header.partitionKey = d2;
          headerSpec.num = 3;
          #zpool.name = cfg.zpool-main.name;
        };

#kdn.disks.impermanence."sys/config".snapshots = true;
        #kdn.disks.impermanence."sys/cache".snapshots = false;
        #kdn.disks.impermanence."sys/data".snapshots = true;
        #kdn.disks.impermanence."sys/state".snapshots = false;
        #kdn.disks.impermanence."usr/config".snapshots = true;
        #kdn.disks.impermanence."usr/cache".snapshots = false;
        #kdn.disks.impermanence."usr/data".snapshots = true;
        #kdn.disks.impermanence."usr/state".snapshots = false;

/* just a single impermanence example goes below */
        #kdn.disks.impermanence."usr/data" = {
        #  #zpool.name = cfg.zpool-main.name;
        #  #zfsPrefix = "${config.networking.hostName}/impermanence";
        #  #zfsPath = "${zfsPrefix}/${"usr/data"}";
        #  imp.directories = [
        #    "/var/lib/libvirt/images"
        #  ];
        #  imp.users.root.directories = [
        #    # ...
        #  ];
        #  imp.users.kdn.directories = [
        #    ".local/share/atuin"
        #    ".local/share/nix"
        #    ".local/share/containers"
        #    "dev"
        #  ];
        #};
        #environment.persistence."/nix/persist/usr/data" = {
        #  enable = true;
        #  hideMounts = true;
        #  users.root.home = "/root";
        #} // cfg.impermanence."usr/data".imp;
        #disko.devices.zpool."${cfg.zpool-main.name}".datasets."${cfg.impermanence."usr/data".zfsPath}" = { ... };

#kdn.disks.tmpfs.size = "16M";
        #disko.devices.nodev."/" = { fsType = "tmpfs"; mountPoints = ["size=${cfg.tmpfs.size}" ... ]; };
        #disko.devices.nodev."/" = { fsType = "tmpfs"; mountPoints = ["size=${cfg.tmpfs.size}" ... ]; };
      }
    )
    ```
   - see [`hosts/pwet`](hosts/pwet/default.nix) for a different storage topology:
     - `boot` disk also holds the LUKS header and volume,
     - `zvol` is used as a LUKS header for the second `zpool`,

5. add all your required `environment.persistence` entries
6. run unified "deploy to install-iso" script utilizing nixos-anywhere and generating secrets automatically:
   ```shell
   set HOST_NAME "pwet"
   set HOST_CONNECTION "192.168.73.211"
   nix run ".#kdnctl" -- host -n "$HOST_NAME" -a "$HOST_CONNECTION" bootstrap
   ```
   - the script automates and therefore deprecates most of the follow-up steps
   - TODO: try and document `chroot` using `nixos-enter` from the install-iso
   - TODO: provide a "cleanup & reboot" script counterpart after `bootstrap`

7. former, deprecated steps (should be documented in `kdnctl host bootstrap` before removing):
    1. set up keyfiles for each disk:
        ```shell
        # this is fish shell
        set HOST_NAME faro
        set DISK_NAME virtual
        dd if=/dev/random bs=1 count=2048 of=/dev/stdout | pass insert --force --multiline "luks/$DISK_NAME-$HOST_NAME/keyfile"
        ```

2. run `nixos-anywhere` to deploy
        ```shell
        # this is fish shell
        set HOST_NAME faro
        set DISK_NAME virtual
        set HOST_CONNECTION root@192.168.73.79
        # build locally
        nixos-anywhere --phases disko,install --disk-encryption-keys "/tmp/$DISK_NAME-$HOST_NAME.key" "$(pass show "luks/$DISK_NAME-$HOST_NAME/keyfile" | psub)" --flake ".#$HOST_NAME" "$HOST_CONNECTION"

# build on the target machine
        nixos-anywhere --phases disko,install --build-on-remote --disk-encryption-keys "/tmp/$DISK_NAME-$HOST_NAME.key" "$(pass show "luks/$DISK_NAME-$HOST_NAME/keyfile" | psub)" --flake ".#$HOST_NAME" "$HOST_CONNECTION"
        ```
    3. set up either of for each disk:
        - (unattended) TPM2 unlock:
            ```shell
            ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null "$HOST_CONNECTION" sudo systemd-cryptenroll --unlock-key-file="/tmp/$DISK_NAME-$HOST_NAME.key" --tpm2-device=auto "/dev/disk/by-partlabel/$DISK_NAME-$HOST_NAME-header"
            ```
        - (attended) YubiKey FIDO2 (touch required, without PIN) unlock:
            ```shell
            ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null "$HOST_CONNECTION" sudo systemd-cryptenroll --unlock-key-file="/tmp/$DISK_NAME-$HOST_NAME.key" --fido2-device=auto --fido2-with-client-pin=false --fido2-with-user-verification=false "/dev/disk/by-partlabel/$DISK_NAME-$HOST_NAME-header"
            ```

4. add SSH key to `/.sops.yaml`

11. run on over ssh: `umount -R /mnt && zpool export -a && reboot now`

### partial recovery

decrypt volume using FIDO2:

```shell
set HOST_NAME faro
set DISK_NAME virtual
set HOST_CONNECTION root@192.168.73.79
set ENCRYPTED_DEVICE /dev/vdb
set ENCRYPTED_HEADER /dev/vda2
set ENCRYPTED_NEW_SLOT 3

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null "$HOST_CONNECTION" systemd-cryptsetup attach "$HOST_NAME-main-crypted" "$ENCRYPTED_DEVICE" - header="$ENCRYPTED_HEADER"
```

copying the keyfile to the machine:

```shell
scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null "$(pass show "luks/$DISK_NAME-$HOST_NAME/keyfile" | psub)" "$HOST_CONNECTION:/tmp/$DISK_NAME-$HOST_NAME.key"
```

generate & register a passphrase:

```shell
pass generate "luks/$DISK_NAME-$HOST_NAME/passphrase" 32
# copy it to device
scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null "$(pass show "luks/$DISK_NAME-$HOST_NAME/passphrase" | tr -d '\n' | psub)" "$HOST_CONNECTION:/tmp/$DISK_NAME-$HOST_NAME.passphrase"
# add it to the header
ssh -t -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null "$HOST_CONNECTION" cryptsetup luksAddKey --header="$ENCRYPTED_HEADER" --new-key-slot="$ENCRYPTED_NEW_SLOT" --key-file="/tmp/$DISK_NAME-$HOST_NAME.key" "$ENCRYPTED_DEVICE" "/tmp/$DISK_NAME-$HOST_NAME.passphrase"
# decrypt, paste the key into it
ssh -t -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null "$HOST_CONNECTION" cryptsetup open --header="$ENCRYPTED_HEADER" "$ENCRYPTED_DEVICE" "$HOST_NAME-main-crypted"
```

## Building on Hetzner Cloud from NixOS installer image

1. mount the NixOS installer image
2. run the build:
   ```
   APPLY=1 HOST="<HOST>" bash <(curl -L 'https://raw.githubusercontent.com/nazarewk-iac/nix-configs/main/hetzner.sh')
   ```

## Minimal NixOS bootstrap without custom installer

Used this snippet to gain initial access to nixos-avf (a NixOS VM on Android phone):

```nix
{ config, lib, pkgs, ... }:

{
  imports = [
    <nixos-avf/avf>
  ];

avf.defaultUser = "kdn";

system.stateVersion = "26.05";
  services.openssh.enable = true;
  services.openssh.ports = [2022 22];
  environment.systemPackages = with pkgs; [
    git
    vim
  ];

services.netbird.clients.priv.port = 0;

nix.settings = {
    show-trace = true;
    experimental-features = [
      "nix-command"
      "flakes"
    ];
  };
  nixpkgs.config = {
    allowAliases = true;
    allowUnfree = true;
  };

users.users.kdn = {
    uid = 31893;
    linger = true;
    openssh.authorizedKeys.keys = [
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIFngB2F2qfcXVbXkssSWozufmyc0n6akKYA8zgjNFdZ ssh@kdn.im"
    ];
  };
}
```

## Interaction between NixOS and Home Manager

- https://jdisaacs.com/blog/nixos-config/

## How to find out what uses the specific store path?

Find immediate parents/reverse dependencies: `nix-store --query --referrers <paths...>`.

Find the root using paths: `nix-store --query --roots <paths...>`.

## Fix nix store errors

Fix errors like `/nix/store/*-source not found`: `sudo nix-store --repair --verify --check-contents`.

## GitHub rate-limiting unauthenticated requests

see https://discourse.nixos.org/t/flakes-provide-github-api-token-for-rate-limiting/18609

add token to `~/.config/nix/nix.conf`:

```
access-tokens = github.com=github_pat_XXXX
```

## example of standalone Nix module

https://gist.github.com/nazarewk/8988facb6118f73d2db3d28b64463cba

## systemd-cryptenroll --fido2-with-user-presence=false doesn't work with YubiKey

YubiKey's FIDO2 applet seems to ALWAYS require touch, so it cannot be used for unattended unlock of LUKS volume:

```shell
> systemd-cryptenroll --unlock-key-file=$(cat /tmp/emmc-etra.key | psub) --fido2-device=auto --fido2-with-client-pin=false --fido2-with-user-verification=false --fido2-with-user-presence=false /dev/disk/by-partlabel/disk-boot-emmc-etra-header
Initializing FIDO2 credential on security token.
👆 (Hint: This might require confirmation of user presence on security token.)
Generating secret key on FIDO2 security token.
👆 Locking without user presence test requested, but FIDO2 device /dev/hidraw1 requires it, enabling.
New FIDO2 token enrolled as key slot 1.
```

# Keeping nixpkgs fork with arbitrary patches up to date

Using my own [`.flake.patches/update.py`](.flake.patches/update.py) script:

1. Create required `nixpkgs` inputs:
  - `nixpkgs-upstream` - desired upstream nixpkgs branch
  - `nixpkgs` - the fork you have access to that will be managed by the updater
  - add entries to [`.flake.patches/config.toml`](.flake.patches/config.toml)
2. run `nix run '.#nixpkgs-update' g:patches`

Most of it is wrapped in [`/nixpkgs-update.sh`](nixpkgs-update.sh) and top entries of [`/flake.nix`](flake.nix).

# TODOs

- [ ] TODO: evaluate https://github.com/Mic92/nix-fast-build
- [ ] TODO: evaluate https://github.com/nix-community/nix-eval-jobs
- [ ] TODO: try to integrate lanzaboote?

nix-configs

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README