Iptables is blocking OpenVPN traffic - how do I fix it?
I'm trying to set up an OpenVPN VPN, which will carry some (but not all) traffic from the clients to the internet via the OpenVPN server.
My OpenVPN server has a public IP on eth0, and is using tap0 to create a local network, 192.168.2.x. I have a client which connects from local IP 192.168.1.101 and gets VPN IP 192.168.2.3.
On the server, I ran:
iptables -A INPUT -i tap+ -j ACCEPT
iptables -A FORWARD -i tap+ -j ACCEPT
iptables -t nat -A POSTROUTING -s 192.168.2.0/24 -o eth0 -j MASQUERADE
On the client, the default remains to route via 192.168.1.1. In order to point it to 192.168.2.1 for HTTP, I ran
ip rule add fwmark 0x50 table 200
ip route add table 200 default via 192.168.2.1
iptables -t mangle -A OUTPUT -j MARK -p tcp --dport 80 --set-mark 80
Now, if I try accessing a website on the client (say, wget google.com), it just hangs there. On the server, I can see
$ sudo tcpdump -n -i tap0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap0, link-type EN10MB (Ethernet), capture size 96 bytes
05:39:07.928358 IP 192.168.1.101.34941 > 74.125.67.100.80: S 4254520618:4254520618(0) win 5840 <mss 5="" 558838="" 1334,sackok,timestamp="" 0,nop,wscale="">05:39:10.751921 IP 192.168.1.101.34941 > 74.125.67.100.80: S 4254520618:4254520618(0) win 5840</mss>
Where 74.125.67.100 is the IP it gets for google.com .
Why isn't the MASQUERADE working? More precisely, I see that the source showing up as 192.168.1.101 – shouldn't there be something to indicate that it came from the VPN?
16 Replies
net.ipv4.ip_forward should be set to 1 with sysctl.
Sorry again if this is too obvious, because most of your post is actualy way over my little head :-)
Hope it helps…
$ cat /proc/sys/net/ipv4/ip_forward
1
@mikeage:
iptables -t nat -A POSTROUTING -s 192.168.2.0/24 -o eth0 -j MASQUERADE
05:39:07.928358 IP 192.168.1.101.34941 > 74.125.67.100.80: S 4254520618:4254520618(0) win 5840
Your iptables line is looking for packets originating from 192.168.2.0/24 and going to eth0, while the source of the packet is actually 192.168.1.101. Changing the iptables line to match 192.168.1.0/24 should clear it up.
As far as the server is concerned, the only fixed address is 192.168.2.2 . How can I have the packets coming from the VPN come from that source?
EDIT: This also doesn't appear to be working:
# sudo iptables -t nat -L -v
Chain PREROUTING (policy ACCEPT 169K packets, 8922K bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT tcp -- any any linode.mikeage.net anywhere tcp dpt:www
Chain POSTROUTING (policy ACCEPT 1109K packets, 70M bytes)
pkts bytes target prot opt in out source destination
34 2040 MASQUERADE all -- any eth0 192.168.1.0/24 anywhere
Chain OUTPUT (policy ACCEPT 1109K packets, 70M bytes)
pkts bytes target prot opt in out source destination
Although no connection can be established on outgoing port 80 [maybe the problem is with the return?]
edit: I'm asking because I see you're using bridge mode (why?), while routing mode is usually recommended.
Anyway, you're using bridged mode, yet we see no mention of br0 in your config. Is forwarding on on the br0 interface?
I have no fundamental interest in routing vs. bridging; I have no network on the server to speak of (it's just my linode). I did switch from dev tun to dev tap a while ago, since I found the /30 addressing to be quite confusing.
@mikeage:
Hoopycat – do you know why this is? 192.168.1.101 is the random local IP address that my client has; tomorrow, it could be anything else.
As far as the server is concerned, the only fixed address is 192.168.2.2 . How can I have the packets coming from the VPN come from that source?
You can enable NAT on the VPN client (192.168.2.2). Then, the packets will appear to be coming from 192.168.2.2 when they hit your server. However, you'd then be NATting traffic twice, which is somewhat silly. (Remember, the packet originated at 192.168.1.101; 192.168.2.2 is merely an intermediate router.)
> EDIT: This also doesn't appear to be working:
# sudo iptables -t nat -L -v Chain PREROUTING (policy ACCEPT 169K packets, 8922K bytes) pkts bytes target prot opt in out source destination 0 0 ACCEPT tcp -- any any linode.mikeage.net anywhere tcp dpt:www Chain POSTROUTING (policy ACCEPT 1109K packets, 70M bytes) pkts bytes target prot opt in out source destination 34 2040 MASQUERADE all -- any eth0 192.168.1.0/24 anywhere Chain OUTPUT (policy ACCEPT 1109K packets, 70M bytes) pkts bytes target prot opt in out source destination
Although no connection can be established on outgoing port 80 [maybe the problem is with the return?]
Hmmm… if you tcpdump eth0, are you seeing the traffic going out with the right addresses? I'm wondering if having that ACCEPT in PREROUTING is messing things up and causing it to skip the POSTROUTING… a skim through the man page doesn't indicate anything for sure, but a SNAT in POSTROUTING will cause it to ignore any further rules, so that'd be my next thing to try
@hoopycat:
You can enable NAT on the VPN client (192.168.2.2). Then, the packets will appear to be coming from 192.168.2.2 when they hit your server. However, you'd then be NATting traffic twice, which is somewhat silly. (Remember, the packet originated at 192.168.1.101; 192.168.2.2 is merely an intermediate router.)
True, but my concern is that in defining the rules, I can easily add 192.168.2.x to my iptables setup. I don't know how I can know ahead of time that the VPN will be passing traffic through from 192.168.1.101. Maybe tomorrow I'll connect to my VPN from a very generous public wifi hotspot, and I'd have a regular public IP?
@hoopycat:
Hmmm… if you tcpdump eth0, are you seeing the traffic going out with the right addresses? I'm wondering if having that ACCEPT in PREROUTING is messing things up and causing it to skip the POSTROUTING… a skim through the man page doesn't indicate anything for sure, but a SNAT in POSTROUTING will cause it to ignore any further rules, so that'd be my next thing to try
:-)
Nope
# iptables -t nat -F
# iptables -t nat -A POSTROUTING -s 192.168.1.0/24 -o eth0 -j MASQUERADE
# iptables -t nat -L -v
Chain PREROUTING (policy ACCEPT 178K packets, 9359K bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 1151K packets, 73M bytes)
pkts bytes target prot opt in out source destination
1 60 MASQUERADE all -- any eth0 192.168.1.0/24 anywhere
Chain OUTPUT (policy ACCEPT 1151K packets, 73M bytes)
pkts bytes target prot opt in out source destination
And then I ran telnet google.com 80 from my client, while monitoring the server…
# tcpdump -i tap0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap0, link-type EN10MB (Ethernet), capture size 96 bytes
17:37:44.541139 IP 192.168.1.101.60907 > cg-in-f100.google.com.www: S 1762522301:1762522301(0) win 5840 <mss 5="" 11338016="" 1334,sackok,timestamp="" 0,nop,wscale="">17:37:47.442837 IP 192.168.1.101.60907 > cg-in-f100.google.com.www: S 1762522301:1762522301(0) win 5840</mss>
@mikeage:
True, but my concern is that in defining the rules, I can easily add 192.168.2.x to my iptables setup. I don't know how I can know ahead of time that the VPN will be passing traffic through from 192.168.1.101. Maybe tomorrow I'll connect to my VPN from a very generous public wifi hotspot, and I'd have a regular public IP?
Right now, you've got two machines involved at the endpoint: your workstation (192.168.1.101) and your VPN gateway (192.168.2.2). If you connect straight from your workstation (or laptop, or whatever), your traffic should be originating from 192.168.2.0/24 or whatever the OpenVPN stuff is assigning.
@mikeage:
And then I ran telnet google.com 80 from my client, while monitoring the server…
# tcpdump -i tap0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tap0, link-type EN10MB (Ethernet), capture size 96 bytes 17:37:44.541139 IP 192.168.1.101.60907 > cg-in-f100.google.com.www: S 1762522301:1762522301(0) win 5840 <mss 5="" 11338016="" 1334,sackok,timestamp="" 0,nop,wscale="">17:37:47.442837 IP 192.168.1.101.60907 > cg-in-f100.google.com.www: S 1762522301:1762522301(0) win 5840</mss>
How about on the eth0 side? Do you see it as 192.168.1.101 or something else?
(For the record, I do use OpenVPN, but not NAT; for NAT-like taste with half the calories, I use ssh -D. I'm a wee bit lazy like that.
@hoopycat:
Right now, you've got two machines involved at the endpoint: your workstation (192.168.1.101) and your VPN gateway (192.168.2.2). If you connect straight from your workstation (or laptop, or whatever), your traffic should be originating from 192.168.2.0/24 or whatever the OpenVPN stuff is assigning.
192.168.2.2 is the address my laptop gets from the VPN. The VPN server (aka my linode) is 192.168.2.1.
@hoopycat:
How about on the eth0 side? Do you see it as 192.168.1.101 or something else?
Since you asked
$ sudo tcpdump -i eth0 -n | grep 74.125.45.100
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
20:47:18.039794 IP 209.123.162.201.50604 > 74.125.45.100.80: S 1187533034:1187533034(0) win 5840 <mss 6="" 13374642="" 1334,sackok,timestamp="" 0,nop,wscale="">20:47:18.074996 IP 74.125.45.100.80 > 209.123.162.201.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 6="" 1990835218="" 1430,sackok,timestamp="" 13374642,nop,wscale="">20:47:18.075020 IP 74.125.45.100.80 > 192.168.1.100.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 6="" 1990835218="" 1430,sackok,timestamp="" 13374642,nop,wscale="">20:47:18.483612 IP 74.125.45.100.80 > 209.123.162.201.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 6="" 1990835627="" 1430,sackok,timestamp="" 13374642,nop,wscale="">20:47:18.483636 IP 74.125.45.100.80 > 192.168.1.100.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 6="" 1990835627="" 1430,sackok,timestamp="" 13374642,nop,wscale="">20:47:19.084976 IP 74.125.45.100.80 > 209.123.162.201.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 6="" 1990836228="" 1430,sackok,timestamp="" 13374642,nop,wscale="">20:47:19.085019 IP 74.125.45.100.80 > 192.168.1.100.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 6="" 1990836228="" 1430,sackok,timestamp="" 13374642,nop,wscale="">20:47:20.284925 IP 74.125.45.100.80 > 209.123.162.201.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 6="" 1990837428="" 1430,sackok,timestamp="" 13374642,nop,wscale="">20:47:20.284994 IP 74.125.45.100.80 > 192.168.1.100.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 6="" 1990837428="" 1430,sackok,timestamp="" 13374642,nop,wscale="">20:47:20.867654 IP 209.123.162.201.50604 > 74.125.45.100.80: S 1187533034:1187533034(0) win 5840 <mss 6="" 13375392="" 1334,sackok,timestamp="" 0,nop,wscale="">20:47:20.891638 IP 74.125.45.100.80 > 209.123.162.201.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 6="" 1990838035="" 1430,sackok,timestamp="" 13374642,nop,wscale="">20:47:20.891663 IP 74.125.45.100.80 > 192.168.1.100.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 6="" 1990838035="" 1430,sackok,timestamp="" 13374642,nop,wscale="">20:47:21.884928 IP 74.125.45.100.80 > 209.123.162.201.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 6="" 1990839028="" 1430,sackok,timestamp="" 13374642,nop,wscale="">20:47:21.884955 IP 74.125.45.100.80 > 192.168.1.100.50604: S 514506911:514506911(0) ack 1187533035 win 5672 <mss 6="" 1990839028="" 1430,sackok,timestamp="" 13374642,nop,wscale="">$ sudo iptables -L -t nat -v
Chain PREROUTING (policy ACCEPT 181K packets, 9535K bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 1157K packets, 74M bytes)
pkts bytes target prot opt in out source destination
3 180 MASQUERADE all -- any eth0 192.168.1.0/24 anywhere
Chain OUTPUT (policy ACCEPT 1157K packets, 74M bytes)
pkts bytes target prot opt in out source destination</mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss></mss>
It may not be obvious at first from the logs, but the traffic here was in bursts (retries?). I'm really leaning towards a problem with the return traffic, but I don't know what else needs to be set up to have it re-masquerade back. I'm also not sure if I should double NAT it… what do you think?
I've also used SOCKS5 proxies (or just an SSH tunnel to 3128): my goal here is to set up a PC to always use a transparent squid proxy (to prevent direct access to the internet).
$ sudo iptables -v -L
Chain INPUT (policy ACCEPT 18433 packets, 20M bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy DROP 29 packets, 1740 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- eth0 tap+ anywhere anywhere state RELATED,ESTABLISHED
6 360 ACCEPT all -- tap+ eth0 anywhere anywhere
26 1560 LOG all -- any any anywhere anywhere LOG level warning
Chain OUTPUT (policy ACCEPT 13868 packets, 2891K bytes)
pkts bytes target prot opt in out source destination
$ sudo iptables -v -L -t nat
Chain PREROUTING (policy ACCEPT 612 packets, 30485 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 2541 packets, 162K bytes)
pkts bytes target prot opt in out source destination
646 41013 MASQUERADE all -- any eth0 anywhere anywhere
Chain OUTPUT (policy ACCEPT 3268 packets, 208K bytes)
pkts bytes target prot opt in out source destination
I did set up logging on the forwarded packets that were not being accepted:
$ dmesg | tail
IN=eth0 OUT=eth0 SRC=74.125.45.100 DST=192.168.1.100 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=19188 PROTO=TCP SPT=80 DPT=33708 WINDOW=5672 RES=0x00 ACK SYN URGP=0
device eth0 entered promiscuous mode
IN=eth0 OUT=eth0 SRC=74.125.45.100 DST=192.168.1.100 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=19108 PROTO=TCP SPT=80 DPT=33709 WINDOW=5672 RES=0x00 ACK SYN URGP=0
IN=eth0 OUT=eth0 SRC=74.125.45.100 DST=192.168.1.100 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=19109 PROTO=TCP SPT=80 DPT=33709 WINDOW=5672 RES=0x00 ACK SYN URGP=0
IN=eth0 OUT=eth0 SRC=74.125.45.100 DST=192.168.1.100 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=19110 PROTO=TCP SPT=80 DPT=33709 WINDOW=5672 RES=0x00 ACK SYN URGP=0
IN=eth0 OUT=eth0 SRC=74.125.45.100 DST=192.168.1.100 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=19111 PROTO=TCP SPT=80 DPT=33709 WINDOW=5672 RES=0x00 ACK SYN URGP=0
IN=eth0 OUT=eth0 SRC=74.125.45.100 DST=192.168.1.100 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=19112 PROTO=TCP SPT=80 DPT=33709 WINDOW=5672 RES=0x00 ACK SYN URGP=0
IN=eth0 OUT=eth0 SRC=74.125.45.100 DST=192.168.1.100 LEN=60 TOS=0x00 PREC=0x00 TTL=54 ID=19113 PROTO=TCP SPT=80 DPT=33709 WINDOW=5672 RES=0x00 ACK SYN URGP=0
[btw, I did switch my test machine from 192.168.1.101 to 192.168.1.100.]
If I'm reading this correctly, the packets returning from google's server are being sent to 192.168.1.100, but my VPS doesn't know how to get there (since it only recognizes the machine called 192.168.1.100 by the name of 192.168.2.2)
Maybe I do need double NAT?
I followed the instructions here, and everything works!
Brief summary:
SNAT to rewrite the source address to 192.168.2.5
disable rp_filter so it won't be rejected.
echo 1 > /proc/sys/net/ipv4/conf/tun0/forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat –append POSTROUTING -s 192.168.44.0/29 -o eth0 -j MASQUERADE
I also had to enable the appropriate option in OpenVPN (obviously).
Also, I used TUN and not TAP…