InterVLAN routing not working as expected

cjseagraves · Thu Apr 04, 2024 12:41 am

Hey everyone,
First time poster here and brand new to Mikrotik. I’m attempting to set myself up for nice home-lab capabilities as I grow into this equipment and more in order to learn along with it. I have great respect for this community and know that someone sharp out there can catch what I am not seeing or perhaps not understanding. Here’s my current setup (with explanation below):

Picture of topology: https://ibb.co/x1jv1t2

"Core Switch" SW-GR config:

# 2024-04-03 16:46:39 by RouterOS 7.14.1
# software id = **ELIDED**
#
# model = CRS310-8G+2S+
# serial number = **ELIDED**
/interface bridge
add admin-mac= **ELIDED** auto-mac=no comment=defconf frame-types=\
    admit-only-vlan-tagged name=bridge port-cost-mode=short vlan-filtering=\
    yes
/interface vlan
add comment="MANAGEMENT SVI" interface=bridge name=VLAN25 vlan-id=25
add comment="WLAN SVI" interface=bridge name=VLAN50 vlan-id=50
add comment="LAN SVI" interface=bridge name=VLAN60 vlan-id=60
add comment="WAN SVI" interface=bridge name=VLAN70 vlan-id=70
/interface list
add comment="Wired list VLAN 60" name=Wired
add comment="WAN list VLAN 70" name=WAN
add comment="Trunk list" name=TRUNK
add comment="Management list VLAN 25" name=MANAGEMENT
add comment="Wireless list VLAN 50" name=Wireless
/ip hotspot profile
set [ find default=yes ] html-directory=hotspot
/ip pool
add name=dhcp_pool_VLAN50 ranges=192.168.50.50-192.168.50.254
add name=dhcp_pool_VLAN60 ranges=192.168.60.50-192.168.60.254
/ip dhcp-server
add address-pool=dhcp_pool_VLAN50 interface=VLAN50 lease-time=1h name=\
    "WLAN DHCP"
add address-pool=dhcp_pool_VLAN60 interface=VLAN60 lease-time=1h name=\
    "LAN DHCP"
/interface bridge port
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    interface=MANAGEMENT pvid=25
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    interface=Wired pvid=60
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    interface=Wireless pvid=50
add bridge=bridge frame-types=admit-only-vlan-tagged interface=TRUNK
add bridge=bridge frame-types=admit-only-untagged-and-priority-tagged \
    interface=WAN pvid=70
/interface bridge settings
set allow-fast-path=no
/ip firewall connection tracking
set udp-timeout=10s
/interface bridge vlan
add bridge=bridge comment="Management VLAN" tagged=\
    sfp-sfpplus1,sfp-sfpplus2,bridge untagged=ether8 vlan-ids=25
add bridge=bridge comment="WLAN VLAN" tagged=sfp-sfpplus1,sfp-sfpplus2,bridge \
    vlan-ids=50
add bridge=bridge comment="LAN VLAN" tagged=sfp-sfpplus1,sfp-sfpplus2,bridge \
    untagged=ether3,ether4,ether5,ether6,ether1,ether2,ether7 vlan-ids=60
add bridge=bridge comment="WAN VLAN" tagged=bridge,sfp-sfpplus1,sfp-sfpplus2 \
    vlan-ids=70
/interface list member
add interface=ether3 list=Wired
add interface=ether4 list=Wired
add interface=ether5 list=Wired
add interface=ether8 list=MANAGEMENT
add interface=sfp-sfpplus1 list=TRUNK
add interface=sfp-sfpplus2 list=TRUNK
add interface=ether1 list=Wired
add interface=ether2 list=Wired
add interface=ether6 list=Wired
add interface=ether7 list=Wired
/ip address
add address=172.16.25.1/28 interface=VLAN25 network=172.16.25.0
add address=192.168.50.1/24 interface=VLAN50 network=192.168.50.0
add address=192.168.60.1/24 interface=VLAN60 network=192.168.60.0
add address=192.168.70.1/24 interface=VLAN70 network=192.168.70.0
/ip dhcp-server network
add address=192.168.50.0/24 dns-server=192.168.70.2,8.8.8.8 gateway=\
    192.168.50.1
add address=192.168.60.0/24 dns-server=192.168.70.2,8.8.8.8 gateway=\
    192.168.60.1
/ip firewall filter
add action=accept chain=forward comment="For intervlan troubleshooting, PFSens\
    e is my actual firewall for internet" dst-address=0.0.0.0 \
    in-interface-list=all out-interface-list=all src-address=0.0.0.0
/ip route
add comment="All non-LAN traffic sent to PFSense router" disabled=no \
    distance=1 dst-address=0.0.0.0/0 gateway=192.168.70.2 pref-src="" \
    routing-table=main scope=30 suppress-hw-offload=no target-scope=10
/system clock
set time-zone-name=US/Eastern
/system identity
set name=SW-GR
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=192.168.70.2
add address=time-b-g.nist.gov
add address=time-a-g.nist.gov
/system routerboard settings
set boot-os=router-os

VLAN configuration:
VLAN 25: Management (172.16.25.0/28)
VLAN 50: Wireless (192.168.50.0/24)
VLAN 60: Wired (192.168.60.0/24)
VLAN 70: WAN (192.168.70.0/24 – changed from /30 to /24 for troubleshooting purposes)

My major problem is that InterVLAN routing is not functioning properly. As far as I can tell, Layer 2 connectivity and INTRAsubnet communication works (devices on same VLAN/subnet connected to different switches/wireless APs are pinging each other, and smart devices are all working on the wireless network). Any Layer 3 routing seems nonexistent. Many traceroutes would show that when a device in one VLAN would try and send a ping to another, it would get as far as the appropriate default gateway on SW-GR and go no further. Interestingly, I can still connect to the internet (well enough to connect to Mikrotik Forum and make this post

) which tells me that my default route is working, but any routes that are learned dynamically via my InterVLAN routes are no good. This is as of yet unsolved.

Topology pic as well as all the switch config files are attached as well. They are all the same CRS310-8G+2S+IN model.

I appreciate help from any willing participants and am looking forward to talking about it!

Thanks everyone,
cjseagraves

anav · Thu Apr 04, 2024 4:40 am

Nice setup.
Clearly the problem is the pfsense. Its butt ugly and capricious

Assuming you have
a. a management vlan where all smart devices get their IP address from.
b. trunk port carrying all vlans (tagged) to first 310 switch, this is a piece of torte.
c. the below example applies to all three switches !!!

Note: Not sure why you have a WAN 70 in the mix........ really means nothing for the switches.

1. removed frame types from bridge itself, put them on bridge ports.
2. only the management vlan need be defined in vlans, the rest are coming in and going out ports, not interacting with the bridge.
3. not routing, only one management interface is required.
4. added ingress-filtering on bridge ports.
5. Bridge ports are for defining PORT, both ethernet ports, and wifi ports, you are incorrectly defining interface list entries and even calling them incorrectly interfaces at that...........
6. Your diagram does not match your config, you are using sfp1+1 and sfp+2 as trunk ports but your diagram clearly shows ether7 is the trunk port from pfsense and spf+1 goes to the SW-GR switch.
7. Assumption made that EeRO mesh is a dumb AP, not able to read tags.
8. Will assume ether8 is being used on the switch even though bridge ports failed to show that.
9. Same with ether 1,2,3,4,5
10. Only for the management vlan is the bridge tagged in /bridge vlans
11. Only one address, and all things point to management vlan gateway
12. Added neighbours discovery and winbox mac server as well.

LR

# model = CRS310-8G+2S+
/interface bridge
add admin-mac= auto-mac=no name=bridge vlan-filtering=yes
/interface vlan
add comment="Management SVI" interface=bridge name=VLAN25 vlan-id=25
/interface list
add name=MANAGEMENT
/interface bridge port
add bridge=bridge ingress-filtering=yes  frame-types=admit-priority-and-untagged interface=ether1 pvid=60 comment="home-lan"
add bridge=bridge ingress-filtering=yes  frame-types=admit-priority-and-untagged interface=ether2 pvid=60 comment="home-lan"
add bridge=bridge ingress-filtering=yes  frame-types=admit-priority-and-untagged interface=ether3 pvid=60 comment="home-lan"
add bridge=bridge ingress-filtering=yes  frame-types=admit-priority-and-untagged interface=ether4 pvid=60 comment="home-lan"
add bridge=bridge ingress-filtering=yes  frame-types=admit-priority-and-untagged interface=ether5 pvid=60 comment="home-lan"
add bridge=bridge ingress-filtering=yes  frame-types=admit-priority-and-untagged interface=ether6 pvid=50  comment="WLAN to AP"
add bridge=bridge ingress-filtering=yes frame-types=admit-only-vlan-tagged interface=ether7 comment="trunk from pfsense"
add bridge=bridge ingress-filtering=yes  frame-types=admit-priority-and-untagged interface=ether8 pvid=25  comment="Mgmt port"
add bridge=bridge ingress-filtering=yes frame-types=admit-only-vlan-tagged interface=sfpplus1 comment="trunk to SW-GR"
/ip neighbor discovery-settings
set discover-interface-list=MANAGEMENT
/interface bridge vlan
add bridge=bridge tagged=bridge,ether7,sfp-sfpplus1  untagged=ether8 vlan-ids=25
add bridge=bridge tagged=ether7,sfp-sfpplus1  untagged=ether6  vlan-ids=50
add bridge=bridge tagged=ether7,sfp-sfpplus1  untagged=ether1,ether2,ether3,ether4,ether5 vlan-ids=60
/interface list member
add interface=VLAN25 list=MANAGEMENT
/ip address
add address=172.16.25.2/28 interface=VLAN25 network=172.16.25.0
/ip dns
set allow-remote-requests=yes servers=172.16.25.1
/ip route
add  dst-address=0.0.0.0/0  gateway=172.16.25.1
/system identity
set name=SW-LR
/system ntp client
set enabled=yes
/system ntp client servers
add address=172.16.25.1
/tool mac-server
set allowed-interface-list=NONE
/tool mac-server mac-winbox
set allowed-interface-list=MANAGEMENT

cjseagraves · Thu Apr 04, 2024 4:56 am

This is fantastic! It’s night in my time zone, but I will absolutely take a look at this in the morning.

I appreciate your help so much man!

anav · Thu Apr 04, 2024 5:02 am

Two good references for some background.
DRY: viewtopic.php?t=143620
VID: https://www.youtube.com/watch?v=YLtGQAQ8iS0&t=414s

cjseagraves · Thu Apr 04, 2024 7:10 pm

Alright Anav, I just got around to taking a look at everything you sent me. Really great stuff, and I have a bunch of questions that I wanna ask you about. However, I think some more explanation of my topology ideas is in order.

For now, my design goal is simple: A VLAN for wired devices, a VLAN for wireless devices, and a VLAN for management. The main goal of this was to use the Core switch (SW-GR) as the core of my network that can route between my internal networks at full 10Gb bandwidth, while anything that needed to go to the internet can get routed to my PFsense and handled from there with firewall rules at slower speeds (my ISP speeds are very slow anyway). This setup is critically dependent on taking full advantage of the L3 hardware offloading capabilities of the CS310-8G-2S+IN for local subnets only.

Due to physical constraints, my “core” (SW-GR) switch is not physically close to my modem (which cannot be moved) and my PFsense (firewall/router). Because of this, I created this additional WAN VLAN as an attempt to isolate the PFsense from other devices. Since I have to connect it to an “access switch” (SW-LR) that has other devices on it, I wanted to ensure that anything that was sent from the core switch to the PFsense was separated from other devices connected to SW-LR at Layer 3 AND 2. Since PFsense is so Ugly and Capricious (lol), I tried to do it this way instead of configuring the trunking on the PFsense, so SW-LR’s ether7 stays as access port and not trunk port. This part is working as expected. PFsense thinks that it's just connected to one device, and that communication does not exist in the same broadcast domain as other devices, however this is definitely the sketchiest part of the setup.

Check out the additional "logical" topology that I added. It might give a better idea of the design I'm trying to emulate while keeping in mind the layout of my house.

Logical Topology.png

To comment on some of your other points:

1. "removed frame types from bridge itself, put them on bridge ports." --- I was trying to follow this advice from the RouterOS guide (see picture "

Point 1 Frame types on bridge.png

") to avoid leaving the Native VLAN open as a means of communicating with the bridge.

4 and 5 : "Bridge ports are for defining PORT..." --- the thought here was that I could add the interface list to the bridge, which means that all of the list members are dynamically added to the bridge with the same settings that I applied to the list itself. From what I can see from /interface/bridge/port print detail, all of the ports are added as I expect with the pvid, frame type, and ingress fitlering applied to the list itself (added a picture "

Point 4 and 5 Interface list dynamic add.png

," notice the "D" for dynamic in the status section). Upon reviewing the config I exported the first time, it doesn’t seem to actually show it, which seems a little problematic… Maybe you’re meant to use these lists for other things like firewalls? If that’s not best practice, please let me know—I don’t want to cause unintended consequences such as my VLANs not working!

6: "ether7 is trunk port..." --- I mentioned some of this in my topology explanation. I can change this to trunk if I need, but it seems to be working correctly as is—all subnets are routed from the Core SW-GR to the internet without VLAN configuration on PFsense. PFsense only has SW-GR in its ARP table, and I made firewall rules on PFsense to allow Wireless and Wired networks to the internet for now (SW-GR is not using NAT)

10: "Only for the management vlan is the bridge tagged in /bridge vlans" --- Just so I'm understanding this, I don't actually have to have the bridge on the access switches act as a trunk port, and things will still work properly? Thinking about this, this would be because these switches would not have a VLAN interface that would receive the tagged traffic, and sfp-sfpplus1/2 are already configured to receive tagged traffic. Is that the correct understanding?

11: "Only one address, and all things point to management vlan gateway" --- I did remove that default route on SW-LR to the PFsense, as that was completely unnecessary as you pointed out. Based on point 10, I can also remove any other unnecessary IP addresses on the access switches as well, right?

12: winbox mac server: I will definitely look into this as well! I can see in the RouterOS documentation what it's supposed to do, but currently I can winbox into all of my switches without it set, so I have some learning to do there as to where the winbox mac server comes into play.

llamajaja · Thu Apr 04, 2024 8:04 pm

1. I am not big fan of the documentation myself. I rely on other sources which I referenced and common sense.

2. You have to clearly decide if the PFSENSE is going to the router with DHCP firewall rules, etc or Just termination point for WAN, no rules
and forward WAN vlan to Switch GR and have Switch GR do everything. The other switches wont change.

3. It sounds like you would like to use the GR switch as a router but its not in the same location as the modem.
However its clear you have connectivity between the Pfsense and the switch so why not just use that wiring and ditch the Pfsense if not replace it with an MT router like RB5009

++++++++++++++++++++

4/5 -- Incorrect, solely to identify etherorts/wlan ports that will handle the vlans, either as trunk ports, access ports, or hybrid ports.

InterVLAN routing not working as expected

InterVLAN routing not working as expected

Re: InterVLAN routing not working as expected

Re: InterVLAN routing not working as expected

Re: InterVLAN routing not working as expected

Re: InterVLAN routing not working as expected

Re: InterVLAN routing not working as expected

Who is online