VRF Shenanigans (need help with VRF Lite)

troy · Wed Jun 14, 2023 10:33 pm

I'm trying to rebuild a router that currently uses PBR with a TON of firewall filter and mangle rules to use VRF instead.

The situation I'm working with, is an open access network. The network operator gets it's Internet access from the participating providers. Each provider needs to be in it's own VRF, leaving the default VRF for the network operator.

Here's my initial configuration, which is working very well for the providers and their customers:

/interface ethernet
set [ find default-name=ether1 ] comment="Access Network"
set [ find default-name=ether7 ] comment="ISP.200 WAN"
set [ find default-name=ether8 ] comment="ISP.100 WAN"

/interface vlan
add interface=ether1 name=ether1.100 vlan-id=100 comment="ISP.100 Customers"
add interface=ether1 name=ether1.200 vlan-id=200 comment="ISP.200 Customers"

/ip vrf
add interfaces=ether8,ether1.100 name=VRF.100
add interfaces=ether7,ether1.200 name=VRF.200

/ip address
add address=172.23.251.2/30 interface=ether8     comment="ISP.100 Upstream"
add address=172.23.252.2/30 interface=ether7     comment="ISP.200 Upstream"
add address=100.99.100.1/22 interface=ether1.100 comment="ISP.100 Downstream"
add address=100.99.200.1/22 interface=ether1.200 comment="ISP.200 Downstream"

/ip route
add gateway=172.23.251.1@VRF.100 routing-table=VRF.100 vrf-interface=ether8
add gateway=172.23.252.1@VRF.200 routing-table=VRF.200 vrf-interface=ether7

I've tried several ways to pick off a single IP from each provider for local use, but nothing I try seems to work exactly right.

The workaround I've found is to physically loop back to the router on another interface.

VRF Shenanigans.png

/interface ethernet
set [ find default-name=ether2 ] comment="Loopback for Internet Access"

/interface vlan
add interface=ether2 name=ether2.100 vlan-id=100
add interface=ether2 name=ether2.200 vlan-id=200

/ip address
add address=100.99.100.2/22 interface=ether2.100
add address=100.99.200.2/22 interface=ether2.200

/ip route
add gateway=100.99.100.1
add gateway=100.99.200.1 distance=10

This works, but I'm not happy with it. There's got to be a way to do this in ROS without looping back to a new interface.

The absolute MUST is that the network operator have internet access via ALL participating providers.

wiseroute · Thu Jun 15, 2023 12:12 pm

hello troy,

it's nice to meet you. I'm Athens. could you be kind enough to give our helen back?

no no. just kidding

ok. @ troy, what is your core? layer 2 network access provider or something?

as we have no idea - what are you trying to achieve (vrf) related to your environment.

troy · Thu Jun 15, 2023 5:54 pm

Not sure what you're asking about the 'core.' It's just a router (CCR1036, soon to be replaced with a CCR2004) that serves as a L3 gateway for both fiber and wireless subscribers.

Currently, we're not using VRF at all, just basic policy based routing, which works, but requires a bit of work in the firewall to keep things relatively secure.

With the VRF configuration I posted, the routing is straight, the subscribers and providers are 100% isolated from each other. The problem is getting internet access to/for the management network, which exists in the default (main) routing table.

Using various routing and/or mangle rules to hop VRFs, I can get some basic connectivity (ping), but can not do so much as a DNS lookup, much less establish a TCP connection to/from anything.

wiseroute · Fri Jun 16, 2023 3:26 am

hello troy,

maybe this wiki can help you?

https://wiki.mikrotik.com/wiki/Manual:I ... s_from_VRF

troy · Fri Jun 16, 2023 4:07 pm

hello troy,

maybe this wiki can help you?

https://wiki.mikrotik.com/wiki/Manual:I ... s_from_VRF

That's where I started. Of course ROS7 is a bit different, but here's where I'm at:

[admin@Core] > /ip route pr where active=yes 
Flags: D - DYNAMIC; A - ACTIVE; c, s, y - BGP-MPLS-VPN
Columns: DST-ADDRESS, GATEWAY, DISTANCE
#     DST-ADDRESS      GATEWAY               DISTANCE
  DAc 192.168.99.0/24  ether1.99                    0
  DAc 100.99.200.0/22  ether1.200@VRF.200           0
0  As 0.0.0.0/0        172.23.251.1@VRF.100         1
  DAc 100.99.100.0/22  ether1.100@VRF.100           0
  DAc 172.23.251.0/30  ether8@VRF.100               0
[admin@Core] > /int dis ether1.200
[admin@Core] > /ip route pr where active=yes
Flags: D - DYNAMIC; A - ACTIVE; c, s, y - BGP-MPLS-VPN
Columns: DST-ADDRESS, GATEWAY, DISTANCE
#     DST-ADDRESS      GATEWAY               DISTANCE
  DAc 192.168.99.0/24  ether1.99                    0
0  As 0.0.0.0/0        172.23.251.1@VRF.100         1
  DAc 100.99.100.0/22  ether1.100@VRF.100           0
  DAc 172.23.251.0/30  ether8@VRF.100               0
[admin@Core] > 
[admin@Core] > ping 8.8.8.8
  SEQ HOST                                     SIZE TTL TIME       STATUS                                                                                                                  
    0                                                              no route to host                                                                                                        
    1                                                              no route to host                                                                                                        
    sent=2 received=0 packet-loss=100% 

[admin@Core] > ping 8.8.8.8 vrf=VRF.100
  SEQ HOST                                     SIZE TTL TIME       STATUS                                                                                                                  
    0 8.8.8.8                                    56 118 11ms750us 
    0 8.8.8.8                                    56 118 11ms770us 
    1 8.8.8.8                                    56 118 12ms483us 
    1 8.8.8.8                                    56 118 12ms683us 
    sent=2 received=4 packet-loss=-100% min-rtt=11ms750us avg-rtt=12ms171us max-rtt=12ms683us 

[admin@Core] > /ip route add routing-table=main  gateway=172.23.251.1@VRF.100
[admin@Core] > ping 8.8.8.8
  SEQ HOST                                     SIZE TTL TIME       STATUS                                                                                                                  
    0 8.8.8.8                                    56 118 8ms445us  
    1 8.8.8.8                                    56 118 15ms896us 
    sent=2 received=2 packet-loss=0% min-rtt=8ms445us avg-rtt=12ms170us max-rtt=15ms896us 

[admin@Core] > ping google.com
invalid value for argument address:
    invalid value of mac-address, mac address required
    invalid value for argument ipv6-address
    while resolving ip-address: could not get answer from dns server
[admin@Core] >

As you can see, after adding a default route for main to jump to VRF.100, I can ping the name server, but cannot actually resolve DNS. Also cannot establish any TCP connections.

Also, there's a NAT rule in place, but this fails:

[admin@Core] > /ip fire nat pr                      
Flags: X - disabled, I - invalid; D - dynamic 
 0    chain=srcnat action=masquerade src-address-list=Management out-interface-list=Outside 
[admin@Core] > ping 8.8.8.8 src-address=192.168.99.1
  SEQ HOST                                     SIZE TTL TIME       STATUS                                                                                                                  
    0 8.8.8.8                                                      timeout                                                                                                                 
    1 8.8.8.8                                                      timeout                                                                                                                 
    2 8.8.8.8                                                      timeout                                                                                                                 
    sent=3 received=0 packet-loss=100%

If, however, I loop back to another physical interface, the main routing table/VRF behaves 100%.

Railander · Sat Jun 24, 2023 2:52 pm

in your routes:

/ip route
add gateway=172.23.251.1@VRF.100 routing-table=VRF.100 vrf-interface=ether8
add gateway=172.23.252.1@VRF.200 routing-table=VRF.200 vrf-interface=ether7

you probably don't want to use vrf-interface at all, unset it:

/ip route
add gateway=172.23.251.1@VRF.100 routing-table=VRF.100
add gateway=172.23.252.1@VRF.200 routing-table=VRF.200

i'm not even sure what that paremeter is for exactly... the table is set by routing-table parameter, and which VRF the gateway can be found in is set by @vrfname in gateway parameter.
even after googling, this is all i've found about it:

vrf-interface () Internal use only parameter which allows identifying to which VRF route should be added. Used by services that add routes dynamically, for example, DHCP client. Shown for debugging purposes.

VRF Shenanigans (need help with VRF Lite)

VRF Shenanigans (need help with VRF Lite)

Re: VRF Shenanigans (need help with VRF Lite)

Re: VRF Shenanigans (need help with VRF Lite)

Re: VRF Shenanigans (need help with VRF Lite)

Re: VRF Shenanigans (need help with VRF Lite)

Re: VRF Shenanigans (need help with VRF Lite)

Who is online