VPS Architecture

    

One flake that describes a whole VPS — disks, firewall, secrets, every container service — plus a second machine that runs its CI, and the OpenTofu that creates both and publishes their DNS.

Ported from the Ansible repo it replaces. What that repo needed three layers for (docker's restart policy, a boot-time docker-services.sh sweep, and an Ansible converge run from a workstation) is one layer here: systemd units built from this flake. There is no state on the server that this repo does not describe, and nothing to remember to run.

.
├── configuration.nix     # the import list, nothing else
├── disk-config.nix       # disko layout + the local QEMU VM
├── flake.nix             # vps, vps-hetzner and runner-hetzner
├── .pre-commit-config.yaml  # the checks, shared by the commit hook and CI
├── .markdownlint-cli2.yaml  # what the markdown linter checks, and what it skips
├── .forgejo/workflows/   # CI and pages, run on the dedicated runner box
├── .github/workflows/    # CI and option packs, run on the GitHub mirror
├── docs/                 # the handbook — mdBook source in docs/src/
├── runner/               # the CI runner's own configuration.nix
├── tests/                # VM and network tests behind checks.*
├── packs/                # the Minecraft worlds' modpacks (packwiz), see the handbook
├── renovate.json5        # this repo's Renovate config
├── modules/
│   ├── options.nix       # infra.* — domain, tailnet IP, docker networks
│   ├── secrets.nix       # the sops keys (serenity's live with their modules)
│   ├── acme.nix          # one certificate, DNS-01 via Cloudflare
│   ├── firewall.nix      # nftables, including the container forward path
│   ├── services.nix      # sshd, tailscale, fail2ban, docker
│   ├── security.nix      # auditd rules
│   ├── postgres.nix      # host postgres + pgbouncer, pg_dumpall backups
│   ├── backups.nix       # restic → B2, daily + weekly quiescent minecraft
│   ├── image-archive.nix # docker save every image; restore one when a pull fails
│   ├── syncthing.nix     # tailnet-only, /home/hutao/syncthing
│   ├── tarpit.nix        # endlessh-go on 222/2022/22222, and its prometheus
│   ├── pages-pull.nix    # fetches published artifacts into the pages volume
│   ├── pages-hook.nix    # webhook receiver that triggers pages-pull
│   ├── renovate.nix      # daily Renovate run, on the host
│   ├── boot.nix hardware.nix ids.nix networking.nix nix.nix users.nix
│   ├── runner/           # everything only the CI runner has
│   └── containers/       # one module per service
└── tofu/                 # hcloud server + edge firewall, Cloudflare DNS

The handbook

Everything below is the short version. The long version — how a packet reaches a service, why the CI runner is a separate machine, what to do when a deploy aborts — is in docs/, published at https://docs.hu-tao.dev/.

nix run .#docs -- --open   # writes the mermaid assets first, then serves

Chapter	For
Network and trust boundaries	the three doors, input vs forward, why there is no private network
The tailnet policy	what a tailnet peer may reach, and why tags are the only lever
CI runner isolation	the seven layers, the one-way rule, the six-path allow-list
The service stack	what runs, and how each thing is reached
Deploying	the everyday path, and the circuit breaker behind it
CI and validation	what each forge checks and how to run the Minecraft startup test
Failure modes and recovery	what rolls back on its own and what does not
Runbook	the commands, in the order you want them
Working on the packs	the Minecraft modpacks: packwiz, sides, the dependency check

Those links go to the source, which renders fine in Forgejo — diagrams included, since they are mermaid and Forgejo draws them natively.

Playing Minecraft

Three worlds, one address each. Install the world's modpack in Prism Launcher (Add Instance → Import, paste the link) or the Modrinth App, ask to be whitelisted, then join.

World	Join at	Minecraft	Pack
world 1	mc.hu-tao.dev	26.1.2	mc.hu-tao.dev-26.1.2.mrpack
world 2	mc2.hu-tao.dev	1.21.1	mc2.hu-tao.dev-1.21.1.mrpack
world 3	mc3.hu-tao.dev	26.2	mc3.hu-tao.dev-26.2.mrpack
world 4	mc4.hu-tao.dev	26.3	mc4.hu-tao.dev-26.3.mrpack

A sleeping world wakes when you join: you are turned away once while it starts, so join again a minute later. The full guide, including what to do when the game will not start, is Playing in the handbook.

The separate Horizon trial runs the native Rust implementation of Java 26.3 at horizon.hu-tao.dev. Both client and server use pinned upstream binaries; the Fabric packs are separate.

Quick start

nix develop                       # the shell everything below assumes
deploy .#vps                      # the VPS
deploy .#runner-forgejo-runner    # the CI runner, via ProxyJump through the VPS

QEMU_OPTS="-vnc :0" nix run .#default    # the whole system, locally
nix develop -c pre-commit install        # once per clone

Two machines: the VPS holds every service, and a second box runs CI and is treated as hostile. See the introduction.

The things that will bite you

Six, all of them load-bearing, all of them silent when wrong.

A published container port lives in the forward chain, not input. It is DNAT'd in prerouting and then forwarded, so it never touches the input hook. Docker writes its own accepts into the ip filter table, and in nftables every table's chain runs — an accept over there cannot rescue a packet table inet nixos-fw drops. So a policy-drop forward chain with no rules leaves every container unreachable and without egress. When adding a service, its port goes in the forward allow-list; getting that backwards gives you a port the internet can reach that the firewall never authorised.

fail2ban's chain = DOCKER-USER does not translate. That is the right answer for iptables. With networking.nftables on, NixOS resolves banaction-allports to nftables-allports, where chain is only the nftables chain name — so DOCKER-USER would name an input-hooked chain and ban nothing that reaches a container. chain_hook = forward is the part that moves it. See the forgejo-ssh jail in modules/services.nix, and check the live result with nft list table inet f2b-table.

The forgejo container needs --log-opt tag=forgejo. The jail reads the journal, and without the tag the journal identifier is the container ID, which changes on every recreate. The jail then looks perfectly healthy and matches nothing.

A bind-mounted sops secret goes stale. A rendered secret's real path is under a generation directory and path only symlinks to it; docker resolves the symlink at mount time and holds that inode forever. dozzle-users.service and mailserver-dkim.service copy to a stable path first, which is why they exist.

A sops env file changes without restarting anything. sops.templates.<n>.path is a stable path, so rotating a value changes the file's content and nothing else — the unit text is byte-identical and switch-to-configuration finds no unit to restart. The container keeps the old value in its environment until something unrelated recreates it, which can be weeks. restartUnits on the template is the fix: sops-nix diffs the rendered file and restarts only on a real change, so no-op deploys still don't bounce the service. navidrome.env and pages-hook.env do this; the older env templates predate it.

The two firewalls must both allow a port. tofu/modules/hetzner-firewall is the edge and modules/firewall.nix is the host. Each is what survives a misconfiguration of the other, so when something is unreachable, check both.

Configuration

Set your secrets using SOPS + age.

mkdir -p ~/.sops-nix
age-keygen | tee ~/.sops-nix/key.txt > /dev/null
chmod 0600 ~/.sops-nix/key.txt

Then edit them:

SOPS_AGE_KEY_FILE=~/.sops-nix/key.txt nix develop -c sops secrets.yaml

secrets.example.yaml is the template. Every declared sops key must exist — most are in modules/secrets.nix, and the serenity keys sit next to their consumers in postgres.nix and containers/serenity-bot.nix. A missing one makes sops-install-secrets fail during activation — which on a fresh install means the machine boots with no credentials at all, its own login included.

Beyond what the Ansible vault held, this port needs:

Key	Why
grafana/admin_user, grafana/admin_password	anonymous Admin is off, so this is the only way in
kuma/healthcheck_url	the out-of-band status-page probe; a separate check from the backup one because they fail for different reasons
minecraft/rcon_password	RCON is loopback-only but it is still a remote console
minecraft2/rcon_password	the second world's console. A separate password on purpose — one leak should not reach both worlds
minecraft3/rcon_password	the third world's console, separate for the same reason
minecraft4/rcon_password	the fourth world's console, separate for the same reason
serenity/bot_token, serenity/ai_api_key	the Discord bot's gateway token and its DeepSeek key
searxng/secret_key	signs searxng's session cookies; upstream's default is the literal ultrasecretkey
searxng/admin_user, searxng/admin_password_hash	searxng has no accounts, so caddy's basic_auth is the whole access control. bcrypt, same shape as dozzle's
serenity/db_password	one password, two consumers: ALTER ROLE in postgres and the pgbouncer userlist, both rendered from this key
navidrome/lastfm/api_key, navidrome/lastfm/secret	one Last.fm application registration, reaching the container as ND_LASTFM_APIKEY / ND_LASTFM_SECRET. Enables scrobbling server-side; each user still links their own account under Personal Settings
renovate/token, renovate/github_com_token	Renovate's bot account and its github.com rate-limit token. Host-side on purpose, so the CI runner never holds a token that can write across every repo
forgejo/system_webhooks/pages_pull/secret	the HMAC the pages-hook receiver verifies. Must match the system webhook's Secret field, which is set by hand in Forgejo
syncthing/gui_password	the syncthing GUI login, declared rather than inherited from the old box's config

The Actions runner's credential is not in this set. It lives on the runner box, delivered through Hetzner user-data, and that host holds no age key at all — see Secrets.

acme_email is gone from the secret set: security.acme needs it at evaluation time and a registration contact is not a credential. It is infra.acmeEmail in modules/options.nix.

Not in this repo

Restore or regenerate these. Without them a clone will not deploy.

What	Where	Notes
secrets.yaml	committed but encrypted	Template is secrets.example.yaml; the age key it is encrypted to is not here
age private key	~/.sops-nix/key.txt, and /var/lib/sops-nix/key.txt on the host	Must be a recipient in .sops.yaml. Lose every recipient and the secrets are unreadable
restic password	backups/restic_password in secrets.yaml	The repository password is the encryption key. Lose it and every snapshot is permanently unrecoverable — keep a copy outside this repo
DKIM private key	email/dkim_private_key	Its public half is published from tofu/. Cannot be regenerated without republishing DNS, and a mismatch fails DKIM at every recipient
tofu/terraform.tfvars	gitignored	Template is tofu/terraform.tfvars.example
Mail store, forgejo repos, minecraft world	docker volumes under /var/lib/docker/volumes	Restore before the first boot of a rebuilt host, or the services initialise blank
Every postgres database	pg_dumpall under /var/backup/postgresql, inside the same restic repository	`zstd -d < all.sql.zstd \