Measure whether services see real client IPs or docker-proxy's address #30

New issue

Closed

opened 2026-09-24 20:19:45 +00:00 by hutao · 2 comments

hutao commented

2026-09-24 20:19:45 +00:00

Owner

modules/firewall.nix (PUBLISHED CONTAINER PORTS) records that a large share of internet traffic to published ports arrives through docker's userland proxy, whose connections show the bridge gateway as their source instead of the client.

Where that happens, per-client-IP controls are keyed on the wrong address:

caddy's rate limits: a masked client shares one bucket with every other masked client. It is exempt from the site-wide zone and shares the login zone with the others.
docker-mailserver's fail2ban (IMAP 993, SMTP 465/587): since #29 a masked address is exempt from bans, so it cannot lock everyone out, but it cannot be banned either.

Nothing has measured how much traffic is affected. To find out, on the vps:

docker exec mailserver fail2ban-client status dovecot
docker exec mailserver sh -c "grep -ohE 'rip=[0-9a-f.:]+' /var/log/mail/mail.log | sort | uniq -c | sort -rn | head -15"
sudo ss -Htnp state established '( sport = :443 or sport = :993 or sport = :587 or sport = :22 )' | awk '{print $4, $NF}' | sed -E 's/:[0-9]+ / /' | sort | uniq -c | sort -rn | head -20

Public addresses in the rip= list and no docker-proxy in the last command means real IPs arrive and this can be closed. Otherwise the fix is making docker preserve the source address, which needs care: the firewall comment describes the half-working failure mode that change can cause.

`modules/firewall.nix` (PUBLISHED CONTAINER PORTS) records that a large share of internet traffic to published ports arrives through docker's userland proxy, whose connections show the bridge gateway as their source instead of the client. Where that happens, per-client-IP controls are keyed on the wrong address: - caddy's rate limits: a masked client shares one bucket with every other masked client. It is exempt from the site-wide zone and shares the login zone with the others. - docker-mailserver's fail2ban (IMAP 993, SMTP 465/587): since #29 a masked address is exempt from bans, so it cannot lock everyone out, but it cannot be banned either. Nothing has measured how much traffic is affected. To find out, on the vps: ```sh docker exec mailserver fail2ban-client status dovecot docker exec mailserver sh -c "grep -ohE 'rip=[0-9a-f.:]+' /var/log/mail/mail.log | sort | uniq -c | sort -rn | head -15" sudo ss -Htnp state established '( sport = :443 or sport = :993 or sport = :587 or sport = :22 )' | awk '{print $4, $NF}' | sed -E 's/:[0-9]+ / /' | sort | uniq -c | sort -rn | head -20 ``` Public addresses in the `rip=` list and no `docker-proxy` in the last command means real IPs arrive and this can be closed. Otherwise the fix is making docker preserve the source address, which needs care: the firewall comment describes the half-working failure mode that change can cause.

hutao commented

2026-09-24 20:42:28 +00:00

Author

Owner

Related finding, measured over the tailnet after caddy's tailnet listener started publishing 8443/udp (#29):

path	result
tcp 8443 direct	303 (reaches the listener)
udp 443, via `iifname tailscale0 udp dport 443 redirect to :8443`	303 over h3
udp 8443 direct	times out

So caddy speaks HTTP/3 on 8443 and the redirected path delivers to it, but udp sent straight to the published port does not arrive. It is likely the same docker port-publishing behaviour as above. #29 works around it by advertising h3=":443" from the tailnet sites. To pin it down on the vps:

docker port caddy
sudo nft list ruleset | grep -n -B2 -A2 8443

Related finding, measured over the tailnet after caddy's tailnet listener started publishing 8443/udp (#29): | path | result | |---|---| | tcp 8443 direct | 303 (reaches the listener) | | udp 443, via `iifname tailscale0 udp dport 443 redirect to :8443` | 303 over h3 | | udp 8443 direct | times out | So caddy speaks HTTP/3 on 8443 and the redirected path delivers to it, but udp sent straight to the published port does not arrive. It is likely the same docker port-publishing behaviour as above. #29 works around it by advertising `h3=":443"` from the tailnet sites. To pin it down on the vps: ```sh docker port caddy sudo nft list ruleset | grep -n -B2 -A2 8443 ```

hutao referenced this issue from a commit

2026-09-24 21:00:19 +00:00

docs(firewall): record that client addresses reach the services

hutao referenced this issue from a pull request that will close it,

2026-09-24 21:00:43 +00:00

Close the remaining brute-force and admin-exposure gaps #29

hutao commented

2026-09-24 21:00:54 +00:00

Author

Owner

Answered: real client addresses reach both caddy and the mail server.

caddy: 330 public requests to pages. from one client, exactly 300 through and 30 answered 429. Every connection was counted against the client's real address. A relay-masked connection would have looked private and been exempt.
mail: dovecot's log carries real public scanner addresses (e.g. 107.150.111.167, 162.243.51.251), and the postfix jail had banned six public addresses. /var/log/mail.log links to /var/log/mail/mail.log, so the jails read the real log. The dovecot jail's zero failures are genuine: the whole log has two authentication failures, because scanners mostly probe and leave.
The 172.18.x entries are containers on the proxy network, now exempt from bans (#29).

The udp 8443 quirk is recorded in network.md; nothing uses that path. Closing with #29 (fa265b4).

Answered: real client addresses reach both caddy and the mail server. - **caddy:** 330 public requests to `pages.` from one client, exactly 300 through and 30 answered 429. Every connection was counted against the client's real address. A relay-masked connection would have looked private and been exempt. - **mail:** dovecot's log carries real public scanner addresses (e.g. `107.150.111.167`, `162.243.51.251`), and the postfix jail had banned six public addresses. `/var/log/mail.log` links to `/var/log/mail/mail.log`, so the jails read the real log. The dovecot jail's zero failures are genuine: the whole log has two authentication failures, because scanners mostly probe and leave. - The `172.18.x` entries are containers on the proxy network, now exempt from bans (#29). The udp 8443 quirk is recorded in network.md; nothing uses that path. Closing with #29 (fa265b4).

hutao closed this issue

2026-09-24 21:20:07 +00:00

hutao referenced this issue from a commit

2026-09-28 12:15:17 +00:00

docs(firewall): record that client addresses reach the services

hutao added the

labels