Use postfix options for routing mails through tunnel #45

Closed
esavkin wants to merge 15 commits from enable-mail-proxy-and-tunnel into master
Owner

Testable solution (wireguard) was replaced by ipsec/gre strongswan, which is not possible to test safely.

Testable solution (wireguard) was replaced by ipsec/gre strongswan, which is not possible to test safely.
esavkin added 1 commit 2024-08-22 12:26:24 +08:00
56577193f0 Use postfix options for routing mails through ssh tunnel
Signed-off-by: Egor Savkin <es@m-labs.hk>
sb10q reviewed 2024-08-22 13:13:27 +08:00
@ -1179,0 +1193,4 @@
@m-labs.hk :
@m-labs.ph :
@193thz.com :
@malloctech.fr :
Owner

Does postfix require all domains to be listed there, even those without a relay host?

Does postfix require all domains to be listed there, even those without a relay host?
Author
Owner

As in examples at https://www.postfix.org/transport.5.html , a wildcard is probably a better way

As in examples at https://www.postfix.org/transport.5.html , a wildcard is probably a better way
esavkin marked this conversation as resolved
esavkin added 1 commit 2024-08-22 13:21:33 +08:00
esavkin added 1 commit 2024-08-23 17:18:59 +08:00
68d8deed73 Use tsocks to wrap socks and add sock transport type
Signed-off-by: Egor Savkin <es@m-labs.hk>
esavkin added 1 commit 2024-08-26 11:59:28 +08:00
a2d7b847a9 Use proxychains-ng instead of tsocks
Signed-off-by: Egor Savkin <es@m-labs.hk>
esavkin changed title from Use postfix options for routing mails through ssh tunnel to WIP: Use postfix options for routing mails through ssh tunnel 2024-08-28 17:24:12 +08:00
esavkin added 1 commit 2024-08-28 17:25:07 +08:00
684c63bb49 WIP: Use gre/ipsec instead of proxy
Signed-off-by: Egor Savkin <es@m-labs.hk>
sb10q reviewed 2024-08-28 21:28:26 +08:00
@ -219,0 +235,4 @@
address = "0.0.0.0";
prefixLength = 0;
via = "5.78.86.156";
options.table = "2";
Owner

Already in use, please pay attention.

Already in use, please pay attention.
sb10q reviewed 2024-08-28 21:28:57 +08:00
@ -1190,0 +1245,4 @@
services.postfix = {
config = {
sender_dependent_relayhost_maps = "hash:/etc/postfix/sender_relay";
postscreen_upstream_proxy_protocol = "haproxy";
Owner

What?

What?
esavkin marked this conversation as resolved
sb10q reviewed 2024-08-29 11:04:53 +08:00
@ -219,0 +223,4 @@
ttl = 255;
type = "tun";
};
interfaces.intl0 = {
Owner

call it trump0, he's the one who started this shit.

call it ``trump0``, he's the one who started this shit.
esavkin marked this conversation as resolved
esavkin force-pushed enable-mail-proxy-and-tunnel from 684c63bb49 to 568d6ccfb1 2024-08-29 17:28:25 +08:00 Compare
esavkin force-pushed enable-mail-proxy-and-tunnel from 568d6ccfb1 to 6a46388d68 2024-09-04 17:24:54 +08:00 Compare
esavkin changed title from WIP: Use postfix options for routing mails through ssh tunnel to WIP: Use postfix options for routing mails through tunnel 2024-09-05 15:40:47 +08:00
esavkin force-pushed enable-mail-proxy-and-tunnel from 6a46388d68 to ebe55e2fa6 2024-09-06 17:31:28 +08:00 Compare
sb10q reviewed 2024-09-06 17:32:35 +08:00
@ -230,0 +237,4 @@
interfaces.intl0 = {
ipv4.addresses = [
{
address = "10.42.0.2";
Owner

That's a private class A address. Again this is never going to work.

That's a private class A address. Again this is never going to work.
esavkin marked this conversation as resolved
esavkin added 1 commit 2024-09-09 15:12:39 +08:00
Owner

AX.25 is also in the kernel. Why not use that instead?

AX.25 is also in the kernel. Why not use that instead?
esavkin reviewed 2024-09-10 11:37:51 +08:00
@ -249,0 +256,4 @@
publicKey = "4RozbGZ9ENCjvJXGMB5aK1oqyZfD4UCarEHjSckwVGI=";
allowedIPs = [ "0.0.0.0/0" ];
endpoint = "5.78.86.156:51820";
persistentKeepalive = 25;
Author
Owner

TODO: needs disabling routing tables (Table = off in wg conf file), so it will open interface and let apps choose to use interface instead of forwarding all the traffic

TODO: needs disabling routing tables (`Table = off` in wg conf file), so it will open interface and let apps choose to use interface instead of forwarding all the traffic
Owner

You need to set up the policy-based routing for the interface choosing part. As has been done before with existing altnet and HKBN connections.

You need to set up the policy-based routing for the interface choosing part. As has been done before with existing altnet and HKBN connections.
esavkin marked this conversation as resolved
sb10q reviewed 2024-09-10 15:36:06 +08:00
@ -248,1 +248,4 @@
};
wireguard.interfaces = {
intl0 = {
ips = [ "10.42.0.2/32" ];
Owner

I think you'd want /31 and then route the default traffic on table 3 through 10.42.0.1 (which would be the VPS)?

I suppose you are then using a regular NAT on the VPS to forward that to the internet and also do the port redirections? Sounds hacky but should work.

I think you'd want /31 and then route the default traffic on table 3 through 10.42.0.1 (which would be the VPS)? I suppose you are then using a regular NAT on the VPS to forward that to the internet and also do the port redirections? Sounds hacky but should work.
Author
Owner

This IP is to acquire.
But for now its a draft anyway, I'll test and update.

This IP is to acquire. But for now its a draft anyway, I'll test and update.
Owner

If you use /32 then you can't reach any other hosts.

If you use /32 then you can't reach any other hosts.
Owner

Also with a /31 the last digits should be .0 and .1, .2 is not going to work.

Also with a /31 the last digits should be .0 and .1, .2 is not going to work.
Owner

This IP is to acquire.

What does that mean anyway?

> This IP is to acquire. What does that mean anyway?
Author
Owner

It corresponds to Interface.Address in the wg.conf, which results in ip -4 address add 10.42.0.2/32 dev intl0. Routes are done separately.

It corresponds to Interface.Address in the wg.conf, which results in `ip -4 address add 10.42.0.2/32 dev intl0`. Routes are done separately.
Owner

And to where will you route? I repeat: with a /32 address you cannot reach any other host.

And to where will you route? I repeat: with a /32 address you cannot reach any other host.
Author
Owner

For example ip route add 10.42.0.0/30 dev wg0.

The current problem with nix configuration is to ensure that it doesn't route too much by default, as it happens wg.conf (but it can be disabled by disabling tables).

For example `ip route add 10.42.0.0/30 dev wg0`. The current problem with nix configuration is to ensure that it doesn't route too much by default, as it happens wg.conf (but it can be disabled by disabling tables).
Owner

And why not just set the correct netmask in the first place?

Regarding the routing tables: You complain about strongswan (despite having been provided with fully working configuration files where you just had to change the IP addresses), but you should note that it doesn't mess up the routing tables. Your private network/NAT hack would also work with strongswan.

And why not just set the correct netmask in the first place? Regarding the routing tables: You complain about strongswan (despite having been provided with fully working configuration files where you just had to change the IP addresses), but you should note that it doesn't mess up the routing tables. Your private network/NAT hack would also work with strongswan.
esavkin marked this conversation as resolved
esavkin added 1 commit 2024-09-11 11:04:42 +08:00
5e13fc0bc2 Apply tested client configuration
Adds an additional route, but doesn't enforce it so other apps will remain the same, but smtp can use tunnel for sending. Also sends replies through the tunnel if connection arrives on the tunnel.
Better have something tested and working before I start doing "perfect".

Signed-off-by: Egor Savkin <es@m-labs.hk>
esavkin added 1 commit 2024-09-11 11:28:10 +08:00
2f1c11d779 Ip rules instead of iptables tracking
Signed-off-by: Egor Savkin <es@m-labs.hk>
esavkin added 1 commit 2024-09-13 15:06:37 +08:00
6382326316 Use IPv6 for WG transport to decrease latency by 20%
Signed-off-by: Egor Savkin <es@m-labs.hk>
esavkin force-pushed enable-mail-proxy-and-tunnel from 6382326316 to addc202345 2024-09-13 16:48:01 +08:00 Compare
esavkin added 1 commit 2024-09-20 17:36:51 +08:00
c7dff35280 Return swan into the zoo
Signed-off-by: Egor Savkin <es@m-labs.hk>
esavkin added 1 commit 2024-09-23 13:36:25 +08:00
12af7ce0ad Add virtual ips for the gre tunnel
Signed-off-by: Egor Savkin <es@m-labs.hk>
esavkin force-pushed enable-mail-proxy-and-tunnel from 12af7ce0ad to b3c97d0fb5 2024-09-30 13:29:15 +08:00 Compare
esavkin force-pushed enable-mail-proxy-and-tunnel from b3c97d0fb5 to 451328c28b 2024-10-04 15:26:02 +08:00 Compare
esavkin changed title from WIP: Use postfix options for routing mails through tunnel to Use postfix options for routing mails through tunnel 2024-10-04 15:26:25 +08:00
Owner

which is not possible to test safely.

Stop whining and listen to the instructions I gave you on several occasions to test this without taking out the server connection.

> which is not possible to test safely. Stop whining and listen to the instructions I gave you on several occasions to test this without taking out the server connection.
Owner

Your comment is just like saying that website changes cannot be tested before deployment because HTTP servers do not work behind NAT. I gave you at least four ways to make it work for testing in the case of GRE.

Your comment is just like saying that website changes cannot be tested before deployment because HTTP servers do not work behind NAT. I gave you at least four ways to make it work for testing in the case of GRE.
esavkin added 1 commit 2024-10-07 16:44:04 +08:00
95ee041e13 Fix postfix settings so it should load successfully
Signed-off-by: Egor Savkin <es@m-labs.hk>
Author
Owner

Well indeed I cannot allocate a global IPv4 for myself.
I checked with IPv6 - tunnel works, once loaded just needs a few iptables rules on intl side.
Also checked that postfix builds and loads successfully and accepts connections from both interfaces.
That enforceful advice of doing the "seemless" (?) routing using GRE was totally misleading. NAT + a few iptables rules is the only possible way since most cloud providers do not allow global IPv4 subnets (Hetzner has this feature on Robot, which is VDS, which is overkill for a simple port forwarding since easier, more straightforward, more common and cheaper solutions exist).

Well indeed I cannot allocate a global IPv4 for myself. I checked with IPv6 - tunnel works, once loaded just needs a few iptables rules on intl side. Also checked that postfix builds and loads successfully and accepts connections from both interfaces. That enforceful advice of doing the "seemless" (?) routing using GRE was totally misleading. NAT + a few iptables rules is the only possible way since most cloud providers do not allow global IPv4 subnets (Hetzner has this feature on Robot, which is VDS, which is overkill for a simple port forwarding since easier, more straightforward, more common and cheaper solutions exist).
Owner

Well indeed I cannot allocate a global IPv4 for myself.

Cannot do this, cannot do that... Anyway this is only one of the several ways of testing it, and as I told you multiple times and as you could have figured out by studying the contents of this repo, we in fact have VLAN switch ports with additional public IPv4s for your convenience.

That enforceful advice of doing the "seemless" (?) routing using GRE was totally misleading.

GRE vs. Wireguard and IP range vs. NAT have nothing to do with each other, and all four combinations of these are possible. You just keep whining whilst demonstrating your lack of understanding of computer networks.

> Well indeed I cannot allocate a global IPv4 for myself. Cannot do this, cannot do that... Anyway this is only one of the several ways of testing it, and as I told you multiple times and as you could have figured out by studying the contents of this repo, we in fact have VLAN switch ports with additional public IPv4s for your convenience. > That enforceful advice of doing the "seemless" (?) routing using GRE was totally misleading. GRE vs. Wireguard and IP range vs. NAT have nothing to do with each other, and all four combinations of these are possible. You just keep whining whilst demonstrating your lack of understanding of computer networks.
esavkin added 1 commit 2024-10-08 16:06:06 +08:00
sb10q reviewed 2024-10-08 16:24:39 +08:00
@ -1208,2 +1261,4 @@
};
services.postfix.mapFiles.sender_transport = pkgs.writeText "sender_transport" ''
@m-labs-intl.com smtptun:
Owner

"smtptun" is quite a confusing name. You'd think it's some postfix internal option. Make it more specific and explicit.

"smtptun" is quite a confusing name. You'd think it's some postfix internal option. Make it more specific and explicit.
sb10q marked this conversation as resolved
esavkin force-pushed enable-mail-proxy-and-tunnel from 02b0095e81 to ed9b79a2d7 2024-10-08 16:28:31 +08:00 Compare
esavkin force-pushed enable-mail-proxy-and-tunnel from ed9b79a2d7 to 4d7e836f07 2024-10-09 11:15:48 +08:00 Compare
esavkin added 1 commit 2024-10-10 12:37:21 +08:00
0ff90d54a9 Stop rejecting packages from the tunnel
Appears that firewall rejects packages before they are getting unwrapped by GRE

Signed-off-by: Egor Savkin <es@m-labs.hk>
esavkin added 1 commit 2024-10-10 13:23:52 +08:00
6d2f4d136e Fix packages being not encrypted
Signed-off-by: Egor Savkin <es@m-labs.hk>
Owner

They're packets to/from the public internet. They don't really need to be encrypted. That's what AH is for.

They're packets to/from the public internet. They don't really need to be encrypted. That's what AH is for.
esavkin force-pushed enable-mail-proxy-and-tunnel from 6d2f4d136e to 0ff90d54a9 2024-10-10 13:47:34 +08:00 Compare
esavkin force-pushed enable-mail-proxy-and-tunnel from 0ff90d54a9 to 60903e955f 2024-10-10 15:52:33 +08:00 Compare
Author
Owner

Appears that firewall indeed does reject the AH+GRE packets from intl host by default - it happens before these packages get unwrapped to other interfaces.
Latest commit with iptables extra rules solves this issue.

Appears that firewall indeed **does** reject the AH+GRE packets from intl host by default - it happens before these packages get unwrapped to other interfaces. Latest commit with iptables extra rules solves this issue.
sb10q reviewed 2024-10-14 14:13:29 +08:00
@ -97,0 +99,4 @@
iptables -A INPUT -s 5.78.86.156 -p ah -j ACCEPT
iptables -A OUTPUT -d 5.78.86.156 -p gre -j ACCEPT
iptables -A OUTPUT -d 5.78.86.156 -p ah -j ACCEPT
'';
Owner

Still zero explanation why this would be required for your tunnel only and not altnet, nor why it works without it anyway.

Still zero explanation why this would be required for your tunnel only and not altnet, nor why it works without it anyway.
Author
Owner

Most likely because intl is NATed, and altnet is not, which interferes with conntrack rules in iptables.

Most likely because intl is NATed, and altnet is not, which interferes with conntrack rules in iptables.
Owner

This makes no sense.

This makes no sense.
Owner

How do you trigger a problem which is addressed by these iptables commands?

How do you trigger a problem which is addressed by these iptables commands?
Author
Owner

Just send the ping or telnet from intl to the hk end's NATed address. It gets rejected (in the logs) on the hk side, and then it sends destination/port unreachable to the unNATed intl (it's done by iptables rules).
Here iptables table with counters (ip6tables -L -v -n):

Chain nixos-fw (1 references)
 pkts bytes target     prot opt in     out     source               destination
   88 10216 nixos-fw-accept  0    --  lo     *       ::/0                 ::/0
 3443  433K nixos-fw-accept  0    --  *      *       ::/0                 ::/0                 ctstate RELATED,ESTABLISHED
    1    80 nixos-fw-accept  6    --  *      *       ::/0                 ::/0                 tcp dpt:22
... A BUNCH OF SELECTED TCP/UDP PORTS ACCEPTED
    0     0 nixos-fw-accept  17   --  *      *       ::/0                 ::/0                 udp dpt:5353
    0     0 DROP       58   --  *      *       ::/0                 ::/0                 ipv6-icmptype 137
    0     0 DROP       58   --  *      *       ::/0                 ::/0                 ipv6-icmptype 139
  108  7576 nixos-fw-accept  58   --  *      *       ::/0                 ::/0
    0     0 nixos-fw-accept  17   --  *      *       ::/0                 fe80::/64            udp dpt:546
  122 20256 nixos-fw-log-refuse  0    --  *      *       ::/0                 ::/0

So by default iptables in nixos firewall indeed rejects any IP packets with custom headers (like that with AH and GRE), and as it seen in logs in the last line.
The other way works because it falls into ctstate RELATED,ESTABLISHED rule, and once hk host sends to intl at least one packet conntrack will mark intl->hk packets as related or established and thus iptables will accept it.

I guess in case of altnet it doesn't fail because it gets unwrapped earlier, or conntrack is able to do the tracking properly (unlike in case of NAT).

Just send the ping or telnet from intl to the hk end's NATed address. It gets rejected (in the logs) on the hk side, and then it sends destination/port unreachable to the unNATed intl (it's done by iptables rules). Here iptables table with counters (`ip6tables -L -v -n`): ``` Chain nixos-fw (1 references) pkts bytes target prot opt in out source destination 88 10216 nixos-fw-accept 0 -- lo * ::/0 ::/0 3443 433K nixos-fw-accept 0 -- * * ::/0 ::/0 ctstate RELATED,ESTABLISHED 1 80 nixos-fw-accept 6 -- * * ::/0 ::/0 tcp dpt:22 ... A BUNCH OF SELECTED TCP/UDP PORTS ACCEPTED 0 0 nixos-fw-accept 17 -- * * ::/0 ::/0 udp dpt:5353 0 0 DROP 58 -- * * ::/0 ::/0 ipv6-icmptype 137 0 0 DROP 58 -- * * ::/0 ::/0 ipv6-icmptype 139 108 7576 nixos-fw-accept 58 -- * * ::/0 ::/0 0 0 nixos-fw-accept 17 -- * * ::/0 fe80::/64 udp dpt:546 122 20256 nixos-fw-log-refuse 0 -- * * ::/0 ::/0 ``` So by default iptables in nixos firewall indeed rejects any IP packets with custom headers (like that with AH and GRE), and as it seen in logs in the last line. The other way works because it falls into `ctstate RELATED,ESTABLISHED` rule, and once hk host sends to intl at least one packet conntrack will mark intl->hk packets as related or established and thus iptables will accept it. I guess in case of altnet it doesn't fail because it gets unwrapped earlier, or conntrack is able to do the tracking properly (unlike in case of NAT).
sb10q marked this conversation as resolved
sb10q reviewed 2024-10-14 14:15:04 +08:00
@ -1252,0 +1265,4 @@
ExecStart = "${pkgs.postfix}/sbin/postmap /var/lib/postfix/conf/sender_transport";
};
wantedBy = [ "multi-user.target" ];
};
Owner

Is this really required? I would expect NixOS to deal by itself with services.postfix.mapFiles without requiring any additional user-defined systemd services.

Is this really required? I would expect NixOS to deal by itself with ``services.postfix.mapFiles`` without requiring any additional user-defined systemd services.
Owner

https://github.com/NixOS/nixpkgs/blob/nixos-24.05/nixos/modules/services/mail/postfix.nix#L752-L755

Does this not work for some reason, or is it your usual sloppiness?

https://github.com/NixOS/nixpkgs/blob/nixos-24.05/nixos/modules/services/mail/postfix.nix#L752-L755 Does this not work for some reason, or is it your usual sloppiness?
sb10q marked this conversation as resolved
sb10q reviewed 2024-10-14 14:26:43 +08:00
@ -1261,1 +1278,4 @@
} // (import /etc/nixos/secret/email_settings.nix);
services.postfix = {
config = {
sender_dependent_default_transport_maps = "hash:/var/lib/postfix/conf/sender_transport";
Owner

I have not looked into it, but considering your sloppy code above, this looks suspicious and I recommend you review this and double check that this is the right way of doing it.

I have not looked into it, but considering your sloppy code above, this looks suspicious and I recommend you review this and double check that this is the right way of doing it.
sb10q marked this conversation as resolved
sb10q reviewed 2024-10-14 14:27:27 +08:00
@ -1262,0 +1286,4 @@
args = [
"-o" "smtp_bind_address=10.47.3.1"
"-o" "inet_interfaces=10.47.3.1"
"-o" "inet_protocols=ipv4"
Owner

Are the last two lines necessary, considering the first line?

Are the last two lines necessary, considering the first line?
esavkin added 1 commit 2024-10-15 17:41:32 +08:00
2ee23bc03a Optimize new fw rules and tweak postfix
Signed-off-by: Egor Savkin <es@m-labs.hk>
sb10q reviewed 2024-10-15 21:14:08 +08:00
@ -1261,1 +1264,4 @@
} // (import /etc/nixos/secret/email_settings.nix);
services.postfix = {
mapFiles.sender_transport = pkgs.writeText "sender_transport" ''
@m-labs-intl.com intltunnel:
Owner

In https://www.postfix.org/transport.5.html I don't see the syntax @domain. Just domain. What is going on?

In https://www.postfix.org/transport.5.html I don't see the syntax ``@domain``. Just ``domain``. What is going on?
Author
Owner

I checked previously and it just didn't work until I used @domain

I checked previously and it just didn't work until I used `@domain`
Owner

Why the apparent discrepancy with the documentation? How carefully did you check?

Why the apparent discrepancy with the documentation? How carefully did you check?
sb10q reviewed 2024-10-15 21:14:25 +08:00
@ -1262,0 +1276,4 @@
args = [
"-o" "inet_interfaces=10.47.3.1"
"-o" "smtp_helo_name=mail.m-labs-intl.com"
"-o" "inet_protocols=ipv4"
Owner

Is inet_protocols=ipv4 still needed?

Is inet_protocols=ipv4 still needed?
Author
Owner

Yes, otherwise it tries to use ipv6

Yes, otherwise it tries to use ipv6
Owner

Despite net_interfaces=10.47.3.1 ?

Despite net_interfaces=10.47.3.1 ?
Author
Owner

Yes

Yes
sb10q reviewed 2024-10-15 21:20:55 +08:00
@ -1262,0 +1266,4 @@
mapFiles.sender_transport = pkgs.writeText "sender_transport" ''
@m-labs-intl.com intltunnel:
* :
'';
Owner

Is the line * : needed?

Is the line ``* :`` needed?
sb10q reviewed 2024-10-15 21:23:55 +08:00
@ -97,0 +97,4 @@
extraCommands = ''
iptables -A INPUT -s 5.78.86.156 -p gre -j ACCEPT
iptables -A INPUT -s 5.78.86.156 -p ah -j ACCEPT
'';
Owner

This is piling up iptables rules at each firewall activation. You need to clear them somewhere...

This is piling up iptables rules at each firewall activation. You need to clear them somewhere...
Author
Owner

I didn't notice rule duplication on rebuild switch, I guess in between they just flush all -> forbid all -> flush -> apply new rules.

I didn't notice rule duplication on rebuild switch, I guess in between they just flush all -> forbid all -> flush -> apply new rules.
Owner

nixos-rebuild switch only restarts units which have changed.

nixos-rebuild switch only restarts units which have changed.
esavkin force-pushed enable-mail-proxy-and-tunnel from 2ee23bc03a to 8ff15e4aba 2024-10-17 11:59:09 +08:00 Compare
sb10q reviewed 2024-10-17 12:00:49 +08:00
@ -93,3 +93,3 @@
allowedTCPPorts = [ 53 80 443 2222 7402 ];
allowedUDPPorts = [ 53 67 500 4500 ];
trustedInterfaces = [ netifLan ];
trustedInterfaces = [ netifLan netifUSA ];
Owner

No.

No.
esavkin force-pushed enable-mail-proxy-and-tunnel from 8ff15e4aba to 785777eb0e 2024-10-17 12:01:34 +08:00 Compare
esavkin closed this pull request 2024-10-18 17:42:42 +08:00
esavkin deleted branch enable-mail-proxy-and-tunnel 2024-10-22 11:25:00 +08:00

Pull request closed

Sign in to join this conversation.
No reviewers
No Label
No Milestone
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: M-Labs/it-infra#45
No description provided.