Close the remaining brute-force and admin-exposure gaps

hutao commented

2026-09-24 20:19:22 +00:00

Owner

Follow-up to the caddy rate-limit change (a71aadb): the remaining brute-force and admin-exposure gaps, and the handbook brought up to date.

What changes

fail2ban never bans a docker address. The host's ignoreIP gains 172.16.0.0/12. forgejo-ssh bans into the forward chain, so a banned bridge address would take every container down (the 2026-09-05 outage). operations/recovery.md already said to keep private ranges there.
docker-mailserver's own fail2ban gets the same exemption, through a mounted fail2ban-jail.cf. Roundcube's IMAP logins come from a docker address, so six wrong webmail passwords in a week would have banned Roundcube and taken webmail down for everyone.
Password auth over git and the API is rate-limited. Authorization: Basic on git. is capped at 30/min per client IP. The login zone could not see it, and Forgejo has no throttle of its own.
kuma's admin login is tailnet-only. Caddy answers /socket.io/ with a 404 on the public status.. kuma's login travels inside that socket, so no HTTP rate limit can count attempts; the public status page never uses it.
Forgejo site administration is tailnet-only. /admin and /api/v1/admin/* return 404 publicly. Personal and org settings stay public.
Split DNS for the half-public names. Tailscale split DNS sends git. and status. lookups from tailnet devices to a CoreDNS on the vps, which answers with the tailnet address. Nothing is configured per device. The name list is computed in caddy.nix and has to agree with tofu's split_dns_subdomains.
Tailscale SSH is off on the vps. With split DNS it owned port 22 on the tailnet address and caught git@git.hu-tao.dev. Deploys and the runner's jump hop now use sshd on 2222.
HTTP/3 works over the tailnet too. Firefox kept the public side's Alt-Svc: h3=":443" after split DNS moved git. to the tailnet address, sent QUIC to udp 443 there, and Forgejo's webpack chunks failed with ChunkLoadError. The firewall now redirects tailscale0's udp 443 to caddy's tailnet listener, which publishes 8443/udp and speaks HTTP/3. The tailnet sites advertise h3=":443", because udp 8443 sent directly does not arrive (recorded as a known quirk). An interim commit removed the public advertisement instead; the later commits undo that.
Real client IPs reach the services (#30). 330 public requests to pages. got exactly 300 through and 30 × 429, so caddy counted the client's real address on every one. docker-mailserver's postfix jail had banned six public addresses. The firewall comment and network.md record it.
Handbook: access-control, services, recovery, tailnet, network, deploying and runbook chapters describe the above.

Verified

caddy validate passes on the generated Caddyfile.
Local caddy runs of the generated blocks returned 429s at the expected budgets. They 404'd every admin spelling Forgejo routes (//admin, /api//v1/admin/*) on the public copy, runner or not, and proxied them on the tailnet copy.
A local CoreDNS run answered only git. and status., and refused everything else.
tofu validate passes, and both hosts evaluate from a clean checkout.
Live after the deploys and tofu apply: the admin paths are reachable over the tailnet (desktop and phone) and 404 publicly, and public DNS is unchanged. Over the tailnet, the chunks that failed return 200 over h3, /admin 303 over h3; publicly over h3, /admin is 404.

After merge

Deployed from the branch and tofu apply'd already. The one thing not yet exercised is a runner deploy through ProxyJump=hutao@vps:2222.

Fixes #30

Follow-up to the caddy rate-limit change (a71aadb): the remaining brute-force and admin-exposure gaps, and the handbook brought up to date. ## What changes - **fail2ban never bans a docker address.** The host's `ignoreIP` gains `172.16.0.0/12`. `forgejo-ssh` bans into the forward chain, so a banned bridge address would take every container down (the 2026-09-05 outage). `operations/recovery.md` already said to keep private ranges there. - **docker-mailserver's own fail2ban gets the same exemption**, through a mounted `fail2ban-jail.cf`. Roundcube's IMAP logins come from a docker address, so six wrong webmail passwords in a week would have banned Roundcube and taken webmail down for everyone. - **Password auth over git and the API is rate-limited.** `Authorization: Basic` on `git.` is capped at 30/min per client IP. The login zone could not see it, and Forgejo has no throttle of its own. - **kuma's admin login is tailnet-only.** Caddy answers `/socket.io/` with a 404 on the public `status.`. kuma's login travels inside that socket, so no HTTP rate limit can count attempts; the public status page never uses it. - **Forgejo site administration is tailnet-only.** `/admin` and `/api/v1/admin/*` return 404 publicly. Personal and org settings stay public. - **Split DNS for the half-public names.** Tailscale split DNS sends `git.` and `status.` lookups from tailnet devices to a CoreDNS on the vps, which answers with the tailnet address. Nothing is configured per device. The name list is computed in caddy.nix and has to agree with tofu's `split_dns_subdomains`. - **Tailscale SSH is off on the vps.** With split DNS it owned port 22 on the tailnet address and caught `git@git.hu-tao.dev`. Deploys and the runner's jump hop now use sshd on 2222. - **HTTP/3 works over the tailnet too.** Firefox kept the public side's `Alt-Svc: h3=":443"` after split DNS moved `git.` to the tailnet address, sent QUIC to udp 443 there, and Forgejo's webpack chunks failed with ChunkLoadError. The firewall now redirects tailscale0's udp 443 to caddy's tailnet listener, which publishes 8443/udp and speaks HTTP/3. The tailnet sites advertise `h3=":443"`, because udp 8443 sent directly does not arrive (recorded as a known quirk). An interim commit removed the public advertisement instead; the later commits undo that. - **Real client IPs reach the services (#30).** 330 public requests to `pages.` got exactly 300 through and 30 × 429, so caddy counted the client's real address on every one. docker-mailserver's postfix jail had banned six public addresses. The firewall comment and network.md record it. - **Handbook:** access-control, services, recovery, tailnet, network, deploying and runbook chapters describe the above. ## Verified - `caddy validate` passes on the generated Caddyfile. - Local caddy runs of the generated blocks returned 429s at the expected budgets. They 404'd every admin spelling Forgejo routes (`//admin`, `/api//v1/admin/*`) on the public copy, runner or not, and proxied them on the tailnet copy. - A local CoreDNS run answered only `git.` and `status.`, and refused everything else. - `tofu validate` passes, and both hosts evaluate from a clean checkout. - Live after the deploys and `tofu apply`: the admin paths are reachable over the tailnet (desktop and phone) and 404 publicly, and public DNS is unchanged. Over the tailnet, the chunks that failed return 200 over h3, `/admin` 303 over h3; publicly over h3, `/admin` is 404. ## After merge Deployed from the branch and `tofu apply`'d already. The one thing not yet exercised is a runner deploy through `ProxyJump=hutao@vps:2222`. Fixes #30

hutao added 8 commits

2026-09-24 20:19:22 +00:00

fix(fail2ban): never ban a docker address 832690f844

The host's ignoreIP exempted no docker range, and the forgejo-ssh jail
bans into the FORWARD chain. A banned bridge address takes every
container down at once, as on 2026-09-05, and is never the attacker:
a connection that came through docker's userland proxy shows the
gateway as its source. operations/recovery.md already said to keep
private ranges in ignoreIP; 172.16.0.0/12 holds every docker network
here, and is now on the list.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

fix(mailserver): exempt docker addresses from its own fail2ban 1a1ff42d48

docker-mailserver's stock jail exempts only 127.0.0.1/8. Roundcube's
IMAP logins come from a docker address, so six wrong webmail passwords
in a week would ban Roundcube and take webmail down for everyone for a
week. A fail2ban-jail.cf mounted into its config dir adds the docker
and tailnet ranges to ignoreip; the 6-a-week, one-week-ban policy is
unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

fix(caddy): rate-limit password auth over git and the API 462aebb1a2

git over https and /api/v1 take a password in an Authorization: Basic
header and never touch /user/login, so the login zone cannot see them,
and Forgejo has no throttle of its own. A git.<domain>-basic zone caps
those requests at 30 a minute per client IP, exempt like the site-wide
zone so renovate's token pushes from this host still work.

Verified: a local caddy run of the zone returned 429 past its budget,
while unauthenticated and token-header requests were not counted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

fix(kuma): serve the admin login on the tailnet side only ea84e9381d

kuma's login is a message inside an open socket.io connection, so the
caddy rate limit sees one request however many passwords go through it,
and kuma's own limiter (20 a minute) was the only layer in front of the
admin account.

The public status page never uses socket.io: kuma 2.5.5 lists /status*
and / in noSocketIOPages and loads everything from /api/status-page/*.
So /socket.io/ on the public status.<domain> is only the admin login and
dashboard, and caddy now answers it with a 404. The same name is served
again on the tailnet listener with the socket open. Its rate-limit zones
carry a -tailnet suffix so the two copies keep separate counters.

A tailnet device reaches the admin side only if it resolves
status.<domain> to the tailnet address. Hosted Tailscale cannot publish a
per-name record (tailscale/tailscale#1543), so that is set up on the
client, not here.

Verified: caddy validate passes with both status blocks; a local run
answered /socket.io/ with 404 without reaching the upstream, while
/status/all and /api/status-page/* were proxied.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

docs: bring the handbook up to date with rate limiting and fail2ban 93b9ec084b

access-control.md gains a rate-limiting section: every site by default,
the site-wide, login and basic-auth zones, who is exempt and why, and the
second layer behind each one. It also covers kuma's socket.io and
docker-mailserver's fail2ban exemption. services.md and recovery.md stop
describing search. as the only rate-limited site, and recovery.md records
the 172.16.0.0/12 exemption its own failure table already asked for.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

fix(forgejo): serve site administration on the tailnet side only 1b5fd3de4d

/admin (the web admin panel) and /api/v1/admin/* (its API: 34 endpoints
on 16.0.5, users, orgs, runner registration tokens, cron, system
webhooks) are site-admin only and used by nothing automated here. On
the public git.<domain> caddy now answers them with a 404; the same name
is served again on the tailnet listener with them open, like
status.<domain>. Personal and org settings stay public. A stolen
password or session can still use what the account owns, but cannot
administer the instance from outside the tailnet.

The tailnet copy drops the runner restriction: the runner has no
tailscale (modules/runner/networking.nix) and the tailnet policy grants
this host to the owner's account alone.

The blocked-path emitter becomes a `handle` of its own. caddy orders
`handle` before `respond`, and git.<domain>'s upstream sits inside
handle blocks for the runner restriction, so the bare `respond` used for
kuma would never have run there.

Verified against the live instance: Forgejo also routes //admin, /admin/
and /api//v1/admin/* to the admin handlers. A local replay of the
generated git blocks against a stub upstream answered all of those with
404 on the public copy (runner or not), proxied them on the tailnet
copy, and left /user/settings, /api/v1/user, repo pages and the runner
API unchanged. `caddy validate` passes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

feat(tailscale): split DNS for the half-public names 1d828c7d50

git.<domain> and status.<domain> are now served twice, publicly and on
caddy's tailnet listener with their admin paths open. The tailnet-only
names never needed anything like this, because their public A record
already points at the tailnet address. These two must keep their public
address for everyone else, so tailnet devices had no way to reach the
admin side short of a per-device hosts entry.

Tailscale split DNS sends lookups for exactly those names, from tailnet
devices only, to a CoreDNS on the vps bound to the tailnet address. It
answers them with that address and refuses everything else. Nothing is
configured per device, phones included. The name list is computed in
caddy.nix from the sites that have both a public and a tailnet copy;
tofu's var.split_dns_subdomains must agree with it.

- modules/options.nix: infra.tailnetIPv4, a copy of tofu's tailnet_ipv4.
- CoreDNS waits for tailscale0 to carry the address before binding,
  within systemd's default start timeout, instead of crash-looping.
- The tailnet policy grants 53 to the vps and tests it.
- The OAuth client needs the DNS (write) scope next to Policy File
  before `tofu apply`; tofu/tailscale.tf says so.

If the resolver is down, tailnet devices most likely cannot resolve
these two names at all: split DNS sends them only there, and Tailscale
documents no fallback. The internet is unaffected and both names live on
this box anyway, so deploy the resolver before `tofu apply`.

Verified: a local CoreDNS run of the generated config answered git and
status with the tailnet address over udp and tcp, returned an empty
AAAA, and refused music.<domain> and google.com. The wait script exits
0 on the exact address and 1 on a prefix of it. `tofu validate` passes
and the rendered policy parses with 53 in the grant and the test.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

fix(tailscale): turn Tailscale SSH off on the vps

CI / Format, lint and evaluate (pull_request) Successful in 3m2s

Details

c07467a3d0

With --ssh on, tailscaled owns port 22 on the tailnet address. Once
split DNS sent git.<domain> to that address on tailnet devices,
`git@git.<domain>` landed on Tailscale SSH (banner SSH-2.0-Tailscale)
instead of forgejo's OpenSSH. sshd on 2222 already carries deploys, so
Tailscale SSH goes.

- modules/services.nix: extraSetFlags = [ "--ssh=false" ]. Dropping
  --ssh from extraUpFlags alone would change nothing on the running
  node, because `tailscale up` only runs on a fresh join; the
  tailscaled-set unit runs `tailscale set --ssh=false` on every start.
- flake.nix: the runner deploy jumps via hutao@vps:2222. A bare `vps`
  names no port and landed on 22, which was Tailscale SSH on the tailnet
  and would be forgejo's now.
- The tailnet policy drops its tag:vps ssh rule, which would grant
  nothing, and says why port 22 is forgejo's.
- The handbook's runbook, network, services, deploying and tailnet
  chapters stop describing Tailscale SSH as the way in.

This also removes the browser re-auth that runner deploys went through;
both deploys now use sshd on 2222 with an ordinary key.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

hutao referenced this pull request

2026-09-24 20:19:45 +00:00

Measure whether services see real client IPs or docker-proxy's address #30

hutao added 1 commit

2026-09-24 20:28:47 +00:00

fix(caddy): no HTTP/3 advertisement on half-public names

CI / Format, lint and evaluate (pull_request) Successful in 2m59s

Details

89d7074c97

caddy's public :443 sends `Alt-Svc: h3=":443"; ma=2592000`, and a
browser keeps that for 30 days. A device that learned it before split
DNS now tries HTTP/3 over UDP 443 at the tailnet address, where only TCP
443 is redirected to caddy's tailnet listener. Those requests hang, and
Forgejo's webpack chunk loads (dropzone, context-popup) failed with
ChunkLoadError on the issue page.

The public copies of git.<domain> and status.<domain>, the hosts in
splitDnsHosts, now drop the header, so no device learns a promise the
tailnet side cannot keep. The other public sites keep HTTP/3. UDP 443
over the tailnet not reaching caddy is left as is: if it did, it would
land on the public listener, where the admin paths are 404.

Verified: HTTP/3 to the tailnet address failed for all three chunks
while h2 and h1.1 returned 200; a local caddy run showed `header
-Alt-Svc` removes the advertisement for one site and leaves its
neighbour's; `caddy validate` passes and only the public git and status
blocks carry it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

hutao added 1 commit

2026-09-24 20:36:52 +00:00

fix(caddy): serve HTTP/3 on the tailnet listener instead of hiding it

CI / Format, lint and evaluate (pull_request) Successful in 2m54s

Details

751674340b

89d7074 stopped the public copies of git. and status. advertising
HTTP/3, so no browser would carry `h3=":443"` onto the tailnet address.
That worked by giving up HTTP/3 for public visitors. This makes the
tailnet side speak it instead and puts the advertisement back:

- firewall: `iifname tailscale0 udp dport 443 redirect to :8443`, in
  the same priority -110 prerouting chain as the tcp rules, so it runs
  ahead of docker's dstnat and never touches public traffic.
- caddy publishes 8443/udp, and the tailnet listener loses its
  `protocols h1 h2` restriction, which existed only because nothing
  carried udp there. It advertises `h3=":8443"`; the tailnet policy's
  8443 grant covers udp.
- `header -Alt-Svc` on the half-public public copies is removed. A
  cached `h3=":443"` now lands on the tailnet listener, the copy with
  the admin paths open.

network.md shows the udp rule and why; tailnet.md's split-DNS section
describes the route instead of the stopgap.

Verified: `caddy validate` passes with no servers block for the tailnet
listener; the ruleset derivation builds, which runs `nft --check` on it,
and carries the udp redirect beside the tcp ones. Not verifiable before
deploy: HTTP/3 over the tailnet end to end.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

hutao added 1 commit

2026-09-24 20:42:14 +00:00

fix(caddy): advertise HTTP/3 on 443 from the tailnet listener

CI / Format, lint and evaluate (pull_request) Successful in 2m54s

Details

8642d7b4e7

After 7516743 deployed, caddy's tailnet listener advertised its own
port, `h3=":8443"`. Measured over the tailnet: tcp 8443 direct and udp
443 through the firewall redirect both reached it (/admin -> 303, the
tailnet copy), but udp 8443 sent directly did not get through, so a
browser following the advertisement hung the way it did before.

The tailnet sites now override the header with `h3=":443"`, the port
every tailnet client already uses for these names and the udp path that
measurably works end to end. Why direct udp 8443 fails is still open;
nothing depends on it any more.

Verified: a local caddy run showed a site-level `header Alt-Svc`
replaces caddy's own value rather than adding a second one; `caddy
validate` passes and exactly the five tailnet sites carry it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

hutao referenced this pull request

2026-09-24 20:42:28 +00:00

Measure whether services see real client IPs or docker-proxy's address #30

hutao added 1 commit

2026-09-24 21:00:19 +00:00

docs(firewall): record that client addresses reach the services

CI / Format, lint and evaluate (pull_request) Successful in 2m52s

Details

fa265b4569

#30 asked whether docker's userland proxy, which the firewall comment
says carries a share of published-port traffic as input, hides client
IPs from caddy's rate limits and docker-mailserver's fail2ban. Measured
2026-09-24, it does not:

- caddy: 330 public requests to pages.<domain> from one client, exactly
  300 through and 30 answered 429, so every connection was counted
  against the client's real address, none exempt as private.
- mail: dovecot's log carries real public scanner addresses, and the
  postfix jail had banned six public addresses. The 172.18.x entries are
  containers on the proxy network, now exempt from bans.

firewall.nix says so under the PUBLISHED CONTAINER PORTS comment, and
network.md records the one quirk left over: udp 8443 sent directly over
the tailnet does not arrive, which nothing uses since the tailnet sites
advertise h3 on 443.

Fixes #30

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

hutao referenced this pull request

2026-09-24 21:00:54 +00:00

Measure whether services see real client IPs or docker-proxy's address #30

hutao merged commit 005f526d0e into main

2026-09-24 21:20:07 +00:00

hutao deleted branch fix/auth-hardening

2026-09-24 21:20:07 +00:00

hutao referenced this pull request from a commit

2026-09-24 21:20:09 +00:00

Merge pull request 'Close the remaining brute-force and admin-exposure gaps' (#29) from fix/auth-hardening into main

hutao referenced this pull request from a commit

2026-09-28 12:15:17 +00:00