Hello.
I am not a switch/network professional.
I do have the P1 temporary license and plan to purchase.
I am trying to setup a lab using RouterOS CHR within Proxmox 8.1.4
The host machine has on NIC (eno2 <--> vmbr0)
I have setup four(4) proxmox bridges for MikroTik, vmbr0, vmbr1, vmbr2, vmbr3
I did update the etherX 'default-name' within MikroTik, in hopes of making this easier to read.
I have setup IP addresses
I have setup ethernet
I have not setup bridges, dhcp, nat, fw,...etc.
I am simply trying to setup a virtual router.

What works:
Connectivity to/from any container or VM attached to vmbr0
Outbound connectivity from vmbr1, vmbr2, vmbr3, to update/install packages.
I can ping FROM vmbr0, any resources attached to vmbr1, vmbr2, vmbr3 resources, and vice-versa
I can ssh/http ether0 to manage Mikrotik
From 'within' virtualization, I can successfully connect to VMs on different interfaces
PC02 <--> vmbr1 <--> vmbr2 <--> ws01
PC02 <--> vmbr1 <--> vmbr3 <--> ws02

NOT working:
I cannot ssh/https/rdp from pc01 through vmbr0 --> any of the resources attached to MikroTik ether1, ether2, ether3.

Although I have spent hours trying to resolve (understand).. I'm sure this boils down to lack of understanding and am hoping for some help.


[admin@MikroTik] > export
# 2024-03-18 11:24:29 by RouterOS 7.14
# software id =
#
/disk
set slot1 slot=slot1
set slot2 slot=slot2
set slot3 slot=slot3
set slot4 slot=slot4
set slot5 slot=slot5

/interface ethernet
set [ find default-name=ether1 ] disable-running-check=no name=ether0
set [ find default-name=ether2 ] disable-running-check=no name=ether1
set [ find default-name=ether3 ] disable-running-check=no name=ether2
set [ find default-name=ether4 ] disable-running-check=no name=ether3

/ip address
add address=192.168.1.10/24 comment=vmbr0 interface=ether0 network=192.168.1.0
add address=10.0.10.1/24 comment=vmbr1 interface=ether1 network=10.0.10.0
add address=10.0.20.1/24 comment=vmbr2 interface=ether2 network=10.0.20.0
add address=10.0.30.1/24 comment=vmbr3 interface=ether3 network=10.0.30.0

/ip dhcp-client
add disabled=yes interface=ether0

/ip dns
set servers=192.168.1.1

/ip route
add disabled=no distance=10 dst-address=0.0.0.0/0 gateway=192.168.1.1 pref-src="" routing-table=main suppress-hw-offload=no

/system clock
set time-zone-name=America/New_York

/system note
set show-at-login=no

/tool sniffer
set filter-interface=all


root@proxmox:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback
iface eno2 inet manual
iface enp1s0f0 inet manual #not used
iface enp1s0f1 inet manual #not used

auto vmbr0
iface vmbr0 inet static
address 192.168.1.60/24
gateway 192.168.1.1
bridge-ports eno2
bridge-stp off
bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
bridge-ports none
bridge-stp off
bridge-fd 0
#CHR ether1 - F5

auto vmbr2
iface vmbr2 inet manual
bridge-ports none
bridge-stp off
bridge-fd 0
#CHR ether2 LAN

auto vmbr3
iface vmbr3 inet manual
bridge-ports none
bridge-stp off
bridge-fd 0
#CHR ether3 LAN

BTW... if there is an easier way to configure a LAB, to achieve the end goal, I am eager to learn as well.

[Solution] albeit a head-scratcher.

After many Youtubes and Many forum threads, I was not able to make this work.... as it should have.
At the suggestion of a https://www.youtube.com/@apalrdsadventures, I added a static route on my laptop (within the 192.168.1.0/24 subnet) to the virtualized router = success.
Which I can only surmise that although my FIOS router has static routes that work for ping/traceroute, that POS does not route L7 properly ???
I can now connect at L7 to my VMs on vmbr1, vmbr2, vmbr3.

If anyone has a more technical explanation, and thus a more sustainable solution, please comment.

Which I can only surmise that although my FIOS router has static routes that work for ping/traceroute, that POS does not route L7 properly ???

Without seeing config of FIOS router (and understanding what it does) it's impossible to say why using default route path doesn't work.

But if FIOS router was ROS device, here's what may happen if one doesn't configure it carefully: routing triangle.
When end device (e.g. PC) decides to communicate with a VM, it sees that it's outside own subnet and hence uses default gateway as ethernet destination for egress frames (i.e.e IP packet contains the right dst-address, but ethernet header contains gateway's MAC address). Default router receives such frame, inspects contents, notes the dst-address and, according to routing tables, determines it needs to use another gateway (in same LAN subnet) as next hop. (FIOS router may also send ICMP type 5 (redirect) message to PC informig it of a better next hop ... but this may not happen and PC may not accept this ICMP message). Frame is then received by your VM router and passed towards VM. The return packet is then received by VM router which determines, that dst-address is in one of connected IP subnets (your LAN subnet) and passes return packet directly to end client (the PC). So far so good from PC's point of view. If PC doesn't act on (optional) ICMP redirect message and keeps using default router as next hop for packets, directed towards VMs, things may get complicated if FIOS router runs stateful firewall. Since return packets bypass this router (because VM router can deliver them directly to PC), state of connection never gets to proper "connected" state and further forward packets are considered invalid by statefull firewall on FIOS router ... and are thus dropped.

In ROS, this can be avoided to either add exemption from firewall rules for traffic between LAN and VM network. Or these packets can be marked as untracked (which will then be accepted by default accept rule). Or something else, possibilities are numerous. But other "SoHo" router softwares may not be as flexible as ROS and solution using FIOS router may prove to be impossible. In such cases it's usually possible to create a "routing segment" between FIOS router and VM router, cutting VM router from direct access towards LAN subnet (and forcing return traffic via FIOS router). Or perhaps use VM router as default gateway for LAN devices ... if VM router doesn't run firewall, then such routing triangle would not block traffic. If VM router does run firewall, then flexibility of ROS allows to construct rules which won't block traffic either. In either case FIOS router would have to run firewall between internet and LAN.