Community discussions

MikroTik App
 
cdoc
just joined
Topic Author
Posts: 3
Joined: Sun Oct 29, 2023 4:38 pm

Transparent bridge, mangle and prerouting

Thu Feb 29, 2024 12:04 am

Hello,

I am trying to create a configuration that will have the following characteristics but I am unsure if I am on the right way.

Server-side:
  • I have a CHR in KVM on a remote dedicated server.
  • The dedicated server hosts a web service of mine.
  • The dedicated server has iptables firewall that allow traffic from the internet only to Winbox and Wireguard ports and forwards it to the CHR.
Client-side:
  • A hEX Mikrotik router will be shipped to each of my clients.
  • The hEX Mikrotik router must be a transparent bridge (I guess?), due to lack of knowledge of my clients' networks. The only difference between clients is for DHCP/Static network. They will be provided with a pre-configured hEX router according to their network setup.
  • The hEX Mikrotik will have a IPIP-over-Wireguard with my CHR.
  • The hEX Mikrotik router will be placed between their ISP router and their switch, so all traffic passes through it.
  • Now for the tricky part, I need it to mark the connections/packets to my server (excluding the Winbox & Wireguard ports), so that when the router catches traffic to my server's IP address, it will route it through the IPIP tunnel.

The point is to allow my clients use my web service (HOST MACHINE), strictly only via the established tunnel (CHR), and any other traffic should be routed through their ISP router regardless of their network setup (DHCP / Static). A sort of "buy my router to use my free service" thing.

I've done some things so far with a kind of success, but as a noob on networking stuff and Mikrotik in general, I would like some help to complete it because the only thing that works for now is the IPIP over Wireguard. My mangle rules are marking the packets correctly when I ping from within the Mikrotik itself (output), but they do not work at all when connecting my laptop with a cable and try to ping my server's public IP from my laptop (prerouting???).

*** The reason I'm trying IPIP over Wireguard is to test the stability of my connections. If I use only Wireguard, I get disconnections on each re-key and it does not comply with my short keepalive. It keeps disconnecting my client's web browser's websocket connection each time. ***

Below is my configuration for a DHCP setup (there might be some input/accept rules that don't matter for now, they're left-overs of some tests I'm doing).
Any help will be appreciated.

If anything is missing, I can provide it right away.
/interface bridge
add name=bridge1
/interface ipip
add local-address=10.10.10.2 name=ipip-tunnel-client remote-address=10.10.10.1
/interface wireguard
add listen-port=13231 mtu=1420 name=wireguard-client
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge1 interface=ether1
add bridge=bridge1 interface=ether2
add bridge=bridge1 interface=ether3
add bridge=bridge1 interface=ether4
add bridge=bridge1 interface=ether5
/interface bridge settings
set use-ip-firewall=yes use-ip-firewall-for-vlan=yes
/interface wireguard peers
add allowed-address=0.0.0.0/0 endpoint-address=SERVER_IP_ADDRESS_HERE endpoint-port=13231 interface=wireguard-client public-key=\
    "MY_PUBLIC_KEY_HERE"


/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/ip address
add address=10.10.10.2/24 interface=wireguard-client network=10.10.10.0
add address=10.10.11.2/24 interface=ipip-tunnel-client network=10.10.11.0
/ip dhcp-client
# DHCP client can not run on slave or passthrough interface!
add interface=ether1
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/ip firewall filter
add action=accept chain=input in-interface=wireguard-client
add action=accept chain=input dst-port=8291 protocol=tcp
add action=accept chain=input protocol=icmp
/ip firewall mangle
add action=mark-connection chain=prerouting dst-address=SERVER_IP_ADDRESS_HERE dst-port=!5182,8291 new-connection-mark=conn_to_server_non_wg_wb passthrough=yes \
    protocol=tcp
add action=mark-connection chain=output dst-address=SERVER_IP_ADDRESS_HERE dst-port=!5182,8291 new-connection-mark=conn_to_server_non_wg_wb passthrough=yes \
    protocol=tcp
add action=mark-connection chain=output log=yes new-connection-mark=conn_to_server_non_wg_wb passthrough=yes protocol=icmp
add action=mark-connection chain=prerouting log=yes new-connection-mark=conn_to_server_non_wg_wb passthrough=yes protocol=icmp
add action=mark-connection chain=prerouting dst-address=SERVER_IP_ADDRESS_HERE dst-port=!5182,8291 new-connection-mark=conn_to_server_non_wg_wb passthrough=yes \
    protocol=udp
add action=mark-connection chain=output dst-address=SERVER_IP_ADDRESS_HERE dst-port=!5182,8291 new-connection-mark=conn_to_server_non_wg_wb passthrough=yes \
    protocol=udp
add action=mark-routing chain=prerouting new-routing-mark=TUNNEL-ROUTE passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat dst-address=10.10.10.0/24 out-interface=wireguard-client src-address=10.10.10.0/24
add action=masquerade chain=srcnat dst-address=10.10.11.0/24 out-interface=ipip-tunnel-client src-address=10.10.11.0/24
add action=masquerade chain=srcnat connection-mark=conn_to_SERVER_IP_ADDRESS_HERE packet-mark=pkt_to_SERVER_IP_ADDRESS_HERE
/ip route
add disabled=no distance=1 dst-address=SERVER_IP_ADDRESS_HERE/32 gateway=ipip-tunnel-client pref-src="" routing-table=TUNNEL-ROUTE scope=30 suppress-hw-offload=\
    no target-scope=10
    

/routing table
add fib name=TUNNEL-ROUTE
 
rplant
Member
Member
Posts: 338
Joined: Fri Sep 29, 2017 11:42 am

Re: Transparent bridge, mangle and prerouting

Thu Feb 29, 2024 6:52 am

Hi,

My thought is that you might be better with a different router that has a different switch chip. (eg. A Hap AC^2)

With these switch chips, you can set the bridge to be fully hardware forwarding, using switch rules to kick selected switched packets to the
CPU. (It then hits the bridge configuration of the CPU, and you can kick it again into the router section)

So the CPU is only doing work for those packets.
And the CPU on the Hap AC^2 is quite a bit faster than the hex.

If multiple vlans are being switched, things are possibly less easy.
 
cdoc
just joined
Topic Author
Posts: 3
Joined: Sun Oct 29, 2023 4:38 pm

Re: Transparent bridge, mangle and prerouting

Thu Feb 29, 2024 9:41 am

That would be a disaster for me... I already have a good amount of hEXes in my stock. :(

Could you think a workaround maybe?
Hi,

My thought is that you might be better with a different router that has a different switch chip. (eg. A Hap AC^2)

With these switch chips, you can set the bridge to be fully hardware forwarding, using switch rules to kick selected switched packets to the
CPU. (It then hits the bridge configuration of the CPU, and you can kick it again into the router section)

So the CPU is only doing work for those packets.
And the CPU on the Hap AC^2 is quite a bit faster than the hex.

If multiple vlans are being switched, things are possibly less easy.
 
rplant
Member
Member
Posts: 338
Joined: Fri Sep 29, 2017 11:42 am

Re: Transparent bridge, mangle and prerouting

Fri Mar 01, 2024 2:36 am

Sorry for the amount of text here.

My thoughts.

1. You likely need to take some care with your customers networks.
2. The VPN connection to your server should look like a WAN interface to
your client,
- ie. Your clients can connect to your server.
- Your server can't open connections to devices on your clients network.
- If you have nat forwarding rules on the vpn, you should let your
clients know they are there, and what they are.
3. The address(es) to be redirected should be to a private IP address,
so if something goes wrong and packets destined for your server leave
via the normal gateway, the ISP will just drop them.

4. Your CHR should also treat the wireguard tunnel as a Wan interface,
and simply port forward (dst-nat) http(s) connections to the local
wireguard IP address to your web server.
Reply to pings over the wireguard tunnel.
Pretty much everything else should be dropped.

5. Please attempt to get rid of the IPIP tunnel, and just use wireguard.

A possibility.

Set the wireguard keepalive on the hex's to 25 S or so. On the CHR
and the Hex's under IP Firewall connection tracking set the UDP
timeout to 30S (Should be default on recent versions of RouterOS, but
was 10S for a long time)


How to:

Backup your current config.
Note: This is for something close to your stated requirements.

Start with the default configuration on the Hex using a recent version
of RouterOS.

And perform initial testing, using the Hex inside a trial network,
with your Laptop using the Hex as its default gateway.

To see that traffic from Laptop that should go via Wireguard/IPIP does, and
the rest goes via the main router.

(Making it do this from the bridge comes later, and is quite simple, but expensive)

From default config:
  • Disable IPv6 on the Hex
  • Move ether1 to the bridge
  • Turn off the dhcp server
  • Move the dhcp client to the bridge
  • Temporarily disable the firewall filter rule that drops invalid packets
  • At this time leave the 192.168.88.1/24 IP address on the Unit
  • Give the Unit a suitable Identity, Username/Password etc
Create your Wireguard, (and if required IPIP configuration).
Limit the Wireguard allowed addresses to only the IP Addresses that
are going to be redirected.
In fact, as mentioned earlier, ideally just use the CHR's wireguard IP
address as the only allowed IP address.
Configure the Hex's wireguard IP address.

If for example you have 10.10.10.2/24 on your hex and 10.10.10.1/24 on
the CHR you only need 10.10.10.1 on the hex for the Peers allowed IP
address(es), and you don't need any routes added. (As a route will
automatically be added)

Otherwise make a routing entry for addresses that must go via Wireguard (in main)

Add both Wireguard (and the IPIP) to the Interface List as WAN.

Plug the hex into your trial client network, (Use Ether1)
Plug your Laptop into any of the other Ports on the Hex.

The Hex should get an IP address on the local network.

Unfortunately you don't know what it is, however the Hex should show up in
Winbox as 192.168.88.1/24, and you should be able to connect via it's
mac address, and find the address it has received from its DHCP server.
(Or set it up statically if no DHCP server) You can then disable the
192.168.88.1 IP address (or not).

Now configure your laptop with the Hex's IP address as its default gateway.

Test that traffic that should go via WG/IPIP does.
Test that rest of traffic goes via default gateway.

Adjust configuration until it works solidly.
You likely need MSS adjustment applied to the wireguard/IPIP traffic?

Once the above is configured and working, you can then rejig the bridge
to redirect the needed packets to the routing engine. Unfortunately this
is expensive.

Disable the Hardware offload on Port ether1 on the bridge.

Create a bridge nat rule
#Template (a more specific example below)
/interface bridge nat
add action=redirect chain=dstnat comment="kick incoming for web server IP into router" mac-protocol=ip \
   dst-address=your-server-addr/32 in-interface=!ether1
#For the above config: (10.10.10.1 as wireguard peer, port forwarding to web server)
/interface bridge nat
add action=redirect chain=dstnat comment="kick incoming for web server IP into router" dst-address=10.10.10.1/32 \
    mac-protocol=ip in-interface=!ether1
You can likely re-enable the Invalid Packet firewall rule that was
disabled earlier at this time.

You should now reconfigure the default gateway on your laptop to be the normal main gateway.

Check it is working.

Some alternate options to using the bridge NAT rule.

The bridge NAT rule is expensive in that it has to examine every
packet coming through the Hex. (ie. Coming from or going to the main
gateway)

Unfortunately these alternate options are likely to be PER client.

If you can setup DHCP option 121 (classless routing) on the client
network, you can get the clients to go via the hex to get to
10.10.10.1

If you can set a DNS entry on the client network, you could have the
name point to the IP address of the hex (which would be configured to
port forward port 443/80 to 10.10.10.1)

If only a small number of client PC's need to connect to your server,
you could configure static routes on them pointing at the Hex.
Or use the hex as their default gateway.
Disable or alter the invalid packet rule if using as default gateway.

Who is online

Users browsing this forum: kulsa and 26 guests