ROS 7.11.2 CHR BGP not Multithreaded and V. Slow

alex_rhys-hurn · Fri Oct 13, 2023 10:42 pm

Hi,

I have CHR running on ESXI with 4 x CPU and 4gb RAM. Single BGP Full table form 1 ISP.

This router is VERY VERY slow to perform any sort of route recalculation, and only One CPU shows any real load. I was under the impression that BGP is multithreded in v7.

Screen shot showsthe CPU load after I changed a route filter to permit all routes in, and the rate at which the routes populate so far is ~20 minutes to do about 150k Routes. Traffic is about 25mbps.

Have I misunderstood the multithreaded aspect of BGP in Ros v7?

BGP Screenshot_2023-10-13_223526.png

Sat Oct 14, 2023 1:48 am

check if your scenario can benefit from this

Advanced BGP tips: affinity
https://www.youtube.com/watch?v=py4up-lO8zY

pe1chl · Sat Oct 14, 2023 3:31 pm

BGP can run multithreaded (see posting above), but when you have only 1 peer there is nothing to gain that way.
Is this only a test? Or else, why would you run full-table BGP with only 1 peer?
Ask the ISP to send you only a default route...

DarkNate · Sat Oct 14, 2023 3:49 pm

BGP can run multithreaded (see posting above), but when you have only 1 peer there is nothing to gain that way.
Is this only a test? Or else, why would you run full-table BGP with only 1 peer?
Ask the ISP to send you only a default route...

Full table with one peer (or more) ensures this network can do BGP RPKI validation. How the hell can do you RPKI validation and filtering with default routes?

https://isbgpsafeyet.com/

pe1chl · Sat Oct 14, 2023 10:27 pm

I don't see how any of that would be an advantage when having only one peer.

DarkNate · Sun Oct 15, 2023 1:22 am

I don't see how any of that would be an advantage when having only one peer.

It's abundantly clear you don't understand how RPKI validation/filtering works and why we should all implemented it. You are under the impression that you need to be multi-homed for RPKI validation to work.

Start here:
https://www.manrs.org/

pe1chl · Mon Oct 16, 2023 3:44 pm

I don't think you can/should do RPKI validation on a single-peer endpoint. Leave that to your upstream ISP. They can do all the route selection for you and send you only a default route.

DarkNate · Mon Oct 16, 2023 4:52 pm

I don't think you can/should do RPKI validation on a single-peer endpoint. Leave that to your upstream ISP. They can do all the route selection for you and send you only a default route.

Are you new to network operations and NOG forums? Do you even know what MANRS is? Very few Tier 1s, Tier 2s and Tier 3s in the world do proper end-to-end, back-to-back RPKI validation/filtering. It is why we take matters to our own hands and do RPKI validation/filtering on our end.

Glad people like you aren't working with me, I'd fire people like you.

psannz · Mon Oct 16, 2023 5:55 pm

I have a few questions regarding the ESXi host and the CHR VM:
What CPU is used? Intel Xeon E5-2620? Intel Xeon Gold 5415+? AMD EPYC 7302?
Are the vCPU on the same Socket? (think NUMA and accessing RAM from different CPU socket)
What setting are you using for the CPU/MMU virtualization?
Which physical NICs are in use? Intel i350? Mellanox Connectx3?
What adapter type is selected for the virtual NIC in the VM? E1000E or VMXNET If latter: is Direct I/O enabled?

Lastly: How far are you overprovisioned on the ESXi? What's the average load accross all CPU cores (Monitor -> Performance)?

Mon Oct 16, 2023 6:01 pm

I don't think you can/should do RPKI validation on a single-peer endpoint. Leave that to your upstream ISP. They can do all the route selection for you and send you only a default route.
Are you new to network operations and NOG forums? Do you even know what MANRS is? Very few Tier 1s, Tier 2s and Tier 3s in the world do proper end-to-end, back-to-back RPKI validation/filtering. It is why we take matters to our own hands and do RPKI validation/filtering on our end.

Glad people like you aren't working with me, I'd fire people like you.

i am pretty sure there is a way to give the technical message without going into personal affairs

DarkNate · Mon Oct 16, 2023 6:07 pm

i am pretty sure there is a way to give the technical message without going into personal affairs

Nothing personal, at all, from my POV. Strictly business here. And as far as “technical message”, that's what MANRS is for.

DarkNate · Mon Oct 16, 2023 6:09 pm

I have a few questions regarding the ESXi host and the CHR VM:
What CPU is used? Intel Xeon E5-2620? Intel Xeon Gold 5415+? AMD EPYC 7302?
Are the vCPU on the same Socket? (think NUMA and accessing RAM from different CPU socket)
What setting are you using for the CPU/MMU virtualization?
Which physical NICs are in use? Intel i350? Mellanox Connectx3?
What adapter type is selected for the virtual NIC in the VM? E1000E or VMXNET If latter: is Direct I/O enabled?

Lastly: How far are you overprovisioned on the ESXi? What's the average load accross all CPU cores (Monitor -> Performance)?

I think you should create separate thread for this. Personally, I would use x64 boxes only for few things:
1. BNGs
2. CGNAT Boxes
3. Stateful firewalling (in enterprise, or large-scale OOB network etc)

So specs, depends on your use-case, target traffic ratio etc. For anything else, I would use dedicated boxes with ASICs, edge routing, layer 3 switching, aggregation, MPLS, last-mile PEs etc.

loloski · Mon Oct 16, 2023 6:45 pm

@DarkNate

As far as I know at least in our region (Asia), ROA record is a _must_ now a days if you are advertising your prefix to upstream that's why pe1chl is suggesting that let the upstream handle this RPKI validation, I personally has this mentality too are we really out of touch on reality? Almost on a weekly basis I have this ticket on our NOC that our downstream has a routing issue only to find out that they don't have ROA record tagged to their AS in APNIC, this has made me to believed that it was really mandatory (RPKI Validation) in every part of the world (t1,t2 & t3 ISP). now you are saying that it is not the case?

Can you enlighten us more please? honestly I don't get it and now i'm fairly confused

could you restrain yourself please don't want to argue with you :p

DarkNate · Mon Oct 16, 2023 8:13 pm

@DarkNate

As far as I know at least in our region (Asia), ROA record is a _must_ now a days if you are advertising your prefix to upstream that's why pe1chl is suggesting that let the upstream handle this RPKI validation, I personally has this mentality too are we really out of touch on reality? Almost on a weekly basis I have this ticket on our NOC that our downstream has a routing issue only to find out that they don't have ROA record tagged to their AS in APNIC, this has made me to believed that it was really mandatory (RPKI Validation) in every part of the world (t1,t2 & t3 ISP). now you are saying that it is not the case?

Can you enlighten us more please? honestly I don't get it and now i'm fairly confused could you restrain yourself please don't want to argue with you :p

I don't remember specific tooling or web dashboards that show real-time RPKI validation data. But you can derive the conclusion using various public tooling:
Cloudflare Radar
Kentik Monitoring
bgp.tools RPKI tagging
Code BGP
Cisco's ThousandEyes
APNIC Dash IIRC
APNIC labs: https://stats.labs.apnic.net/rpki

As per APNIC labs, global RPKI ROV is less than 20% or so. That right there is sufficient data.

Get full tables, reject default routes, perform RPKI validation locally.

Test your RPKI validation using this:
https://isbgpsafeyet.com/

Mon Oct 16, 2023 8:15 pm

there is journey remaining towards full ROA compliance

roa-2023-10.jpg

DarkNate · Mon Oct 16, 2023 8:27 pm

there is journey remaining towards full ROA compliance

Don't confuse ROA data and ROV data. Even if ROA data is 100% compliance, it is useless if there's no ROV implementation.

alex_rhys-hurn · Mon Oct 30, 2023 9:17 am

Before this thread got hijacked and taken off topic, I did learn some things about managing the processes. It seems to me that the various processes are all camping on 1 CPU rather than spreading over whatever CPUs are in the system.

Many thanks to @chechito for pointing me towards Affinity, and the helpful Mikrotik Youtube on the subject.

Alex

ROS 7.11.2 CHR BGP not Multithreaded and V. Slow

ROS 7.11.2 CHR BGP not Multithreaded and V. Slow

Re: ROS 7.11.2 CHR BGP not Multithreaded and V. Slow

Re: ROS 7.11.2 CHR BGP not Multithreaded and V. Slow

Re: ROS 7.11.2 CHR BGP not Multithreaded and V. Slow

Re: ROS 7.11.2 CHR BGP not Multithreaded and V. Slow

Re: ROS 7.11.2 CHR BGP not Multithreaded and V. Slow

Re: ROS 7.11.2 CHR BGP not Multithreaded and V. Slow

Re: ROS 7.11.2 CHR BGP not Multithreaded and V. Slow

Re: ROS 7.11.2 CHR BGP not Multithreaded and V. Slow

Re: ROS 7.11.2 CHR BGP not Multithreaded and V. Slow

Re: ROS 7.11.2 CHR BGP not Multithreaded and V. Slow

Re: ROS 7.11.2 CHR BGP not Multithreaded and V. Slow

Re: ROS 7.11.2 CHR BGP not Multithreaded and V. Slow

Re: ROS 7.11.2 CHR BGP not Multithreaded and V. Slow

Re: ROS 7.11.2 CHR BGP not Multithreaded and V. Slow

Re: ROS 7.11.2 CHR BGP not Multithreaded and V. Slow

Re: ROS 7.11.2 CHR BGP not Multithreaded and V. Slow

Who is online