Tailnet requests reach caddy as 172.18.0.1: the 443 to 8443 redirect bypasses docker's DNAT #43

New issue

Open

opened 2026-10-21 22:15:42 +00:00 by hutao · 0 comments

hutao commented

2026-10-21 22:15:42 +00:00

Owner

Every request over the tailnet reaches caddy from 172.18.0.1, the proxy bridge's gateway, instead of from the client. Follows on from #30, which measured the public ports.

Cause

nixos-fw's prerouting chain rewrites tailnet https onto caddy's second listener:

iifname "tailscale0" tcp dport 443 redirect to :8443
iifname "tailscale0" udp dport 443 redirect to :8443

redirect is a NAT rule, and netfilter performs NAT for a connection only once. This one runs first (priority dstnat - 10), so docker's DNAT for published port 8443 never applies. The connection is delivered locally to docker-proxy on :8443, which opens a second connection to caddy using the bridge gateway as its source.

Captured on the vps at 2026-10-22 01:14 EEST:

100.109.115.12:8443  <- 100.108.24.65:58338  docker-proxy pid 5147   (tailnet client)
172.18.0.1:57840     -> 172.18.0.10:8443     docker-proxy pid 5147   (onward connection to caddy)

Caddy's access log for mail. and music. shows "remote_ip":"172.18.0.1","client_ip":"172.18.0.1".

Effect

Applies to every tailnet vhost: dozzle, grafana, music, search, syncthing, webmail, and the admin side of git and status.

caddy's per-client rate limits put every tailnet client in one bucket
Roundcube's login rate limit (mod_remoteip trusts caddy's XFF, and that header now says 172.18.0.1) is likewise shared
the client-address access log from 8e137e9 does not record any address for tailnet traffic

Public 443 does not pass through the redirect and is not affected by this issue.

Fix options (none tested)

Publish caddy's tailnet listener on the tailscale address (100.109.115.12:443:8443, tcp and udp) and remove the redirect, so docker's DNAT handles the connection and keeps the source. Unknowns: whether docker accepts this alongside the existing 0.0.0.0:443 binding, and the tailscale address must exist before caddy starts. Validate locally, not on the vps.
Disable the userland proxy and DNAT straight to caddy. This needs a pinned caddy address, and pinning the proxy network's subnet means recreating that network.
Run caddy with host networking. This also fixes the IPv6 gap documented in docs/src/architecture/access-control.md.

Update docs/src/architecture/access-control.md ("Client addresses") in the same branch as the fix.

Every request over the tailnet reaches caddy from `172.18.0.1`, the proxy bridge's gateway, instead of from the client. Follows on from #30, which measured the public ports. ## Cause `nixos-fw`'s prerouting chain rewrites tailnet https onto caddy's second listener: ``` iifname "tailscale0" tcp dport 443 redirect to :8443 iifname "tailscale0" udp dport 443 redirect to :8443 ``` `redirect` is a NAT rule, and netfilter performs NAT for a connection only once. This one runs first (priority `dstnat - 10`), so docker's DNAT for published port 8443 never applies. The connection is delivered locally to `docker-proxy` on :8443, which opens a second connection to caddy using the bridge gateway as its source. Captured on the vps at 2026-10-22 01:14 EEST: ``` 100.109.115.12:8443 <- 100.108.24.65:58338 docker-proxy pid 5147 (tailnet client) 172.18.0.1:57840 -> 172.18.0.10:8443 docker-proxy pid 5147 (onward connection to caddy) ``` Caddy's access log for mail. and music. shows `"remote_ip":"172.18.0.1","client_ip":"172.18.0.1"`. ## Effect Applies to every tailnet vhost: dozzle, grafana, music, search, syncthing, webmail, and the admin side of git and status. - caddy's per-client rate limits put every tailnet client in one bucket - Roundcube's login rate limit (mod_remoteip trusts caddy's XFF, and that header now says 172.18.0.1) is likewise shared - the client-address access log from 8e137e9 does not record any address for tailnet traffic Public 443 does not pass through the redirect and is not affected by this issue. ## Fix options (none tested) 1. Publish caddy's tailnet listener on the tailscale address (`100.109.115.12:443:8443`, tcp and udp) and remove the redirect, so docker's DNAT handles the connection and keeps the source. Unknowns: whether docker accepts this alongside the existing `0.0.0.0:443` binding, and the tailscale address must exist before caddy starts. Validate locally, not on the vps. 2. Disable the userland proxy and DNAT straight to caddy. This needs a pinned caddy address, and pinning the proxy network's subnet means recreating that network. 3. Run caddy with host networking. This also fixes the IPv6 gap documented in `docs/src/architecture/access-control.md`. Update `docs/src/architecture/access-control.md` ("Client addresses") in the same branch as the fix.

hutao added the

area/network

kind

bug

labels

2026-10-25 21:40:39 +00:00