This content is for Patreon subscribers of the j2 blog. Please consider becoming a Patreon subscriber for as little as $1 a month. This helps to provide higher quality content, more podcasts, and other goodies on this blog.
There always is a lot of talk about Mikrotik RouterOS CPU usage. I wanted to take a few minutes and go over a real-world example and explain some of the ins and outs when discussing Mikrotik CPU usage.
Let’s talk about the router in question. This is a CCR1016-12s-1S+. This is a 16 core 1.2GHz per core and 2GB RAM tilex based router. It is currently pulling in 1,764,849 IPv4 routes. There are two transit provider BGP feeds, multiple direct peers, an Internet Exchange peer to dual-route servers. The router handles a little over 3 gigs of routed traffic at peak times. Most of the traffic is on VLANs coming from a Cisco switch to the SFPPlus port.
One of the first things people turn on is the overall CPU usage within winbox. I like to think of this as an overall view of the CPUs on this router. Keep in mind there are 16.
Th next thing to investigate when it comes to CPU is to open up System..resources. Once there clock on CPU.
It will then bring up a screen that looks like the following.
Oh My we have 100% CPU! Must replace this router ASAP! Calm down, remember you have 16 cores. So, why is this CPU at 100% and what ramifications does this have?
Remember earlier when we talked about BGP? In Mikrotik, BGP is not a multi-core aware process. This means BGP is limited to just one core to do it’s work. Since there are always routes being withdrawn and re-added to the routing table it is a busy process. Lots of math calculations going on. The key thing is this is expected behavior on a router running multiple BGP peers such as this one. This is not a bad thing, but not ideal. Throwing more cores at BGP is not the answer. Optimizing the process, as it has been done in V7 is the way to go.
If we expand the CPU window we will notice other processes are multi-core aware and.or are spreading their load among different cores.
As you can see we are in pretty good shape. We have a few CPUs above 50% utilization but, only a few. I will keep reminding you of the fact we have 16 of them.
Diagnosing CPU issues can get a little complicated because routers like the 3011 have some have the majority of their ports shared with a single CPU bus. https://wiki.mikrotik.com/images/f/f3/Switch_chip_block_diagram.png. As you can tell in the diagram there are 5 ports which share 1 Gig to the CPU. The fact that an actual switch chip with hardware offloading is in the middle helps, but the bus is still oversold. This is one reason consolidating routers to an actual switch will make a difference.
Janis Megis from Mikrotik had presentation at MUM, which is a little older now, still sheds a lot of light on how Mikrotik CPU works. https://mum.mikrotik.com/presentations/US10/Megis.pdf There is some pretty interesting stuff starting on page 14
With Mikrotik switching to ARM processors we will see huge differences with them and RotuerOS7. We will see less cores, but better utilization of those cores. The new 2004 with all SFP and 2 25 gig ports only has 4 CPU.
So the next time you look at a router, take a few moments to see how utilized the entire CPU architecture is instead of just one CPU.
This morning I had a Mikrotik CCR1016 where I had to change the router ID, which caused all the sessions to reset. The following is a screenshot of the time it took to re-learn all of the peers. Obviously, the smaller prefixes were learned pretty quickly. It took about 10 minutes to learn two full IPv4 route tables and about 5 minutes to learn the IPv6 routing tables.
This is why I always get full routes plus a default from the upstream when it warrants full routes. This way I can have slow convergence time like this and still have traffic flowing.
What’s new in 7.0beta7 (2020-Jun-3 16:31):
!) added Layer3 hardware offloading support for CRS317-1G-16S+RM more info here: https://wiki.mikrotik.com/wiki/Manual:CRS3xx_series_switches#L3_Hardware_Offloading
!) enabled BGP support with multicore peer processing (CLI only);
!) enabled RPKI support (CLI only);
!) ported features and fixes introduced in v6.47;
!) routing updates, complete status report: https://help.mikrotik.com/docs/display/ROS/v7+Routing+Protocol+Status
!) system kernel has been updated to version 5.6.3;
*) other minor fixes and improvements;
Mikrotik RBSXTR running RouterOS 6.46.6 on an AT&T sim card. Upload stats were in the 10-15 meg range.
What’s new in 6.45.9 (2020-Apr-30 10:25):
*) chr – added support for file system quiescing;
*) chr – enabled support for VMBus protocol version 4.1;
*) chr – improved system stability when running CHR on Hyper-V;
*) crs3xx – fixed frame forwarding after disabling/enabling bridge hardware offloading for CRS354-48G-4S+2Q+ device;
*) crs3xx – fixed interface statistics for CRS354-48G-4S+2Q+ and CRS354-48P-4S+2Q+ devices;
*) crs3xx – fixed switch rule “dst-port” parameter for IPv6 traffic on CRS305-1G-4S+, CRS326-24G-2S+, CRS328-24P-4S+, CRS328-4C-20S-4S+, netPower 15FR devices;
*) crs3xx – improved SFP+ DAC cable initialization for CRS326-24S+2Q+ device;
*) defconf – added welcome note with common first steps for new users;
*) discovery – do not send CDP and LLDP packets on interfaces that does not have MAC address;
*) ipsec – improved system stability when handling fragmented packets;
*) lte – added “phy-cellid” value support for LTE-US;
*) lte – fixed IP type selection from APN on RBSXTLTE3-7;
*) lte – improved system stability when performing firmware update on R11e-LTE6;
*) ssh – added support for RSA keys with SHA256 hash (RFC8332);
*) system – correctly handle Generic Receive Offloading (GRO) for MPLS traffic;
*) system – improved system stability when forwarding traffic from switch chip to CPU (introduced in v6.43);
*) system – improved system stability when receiving/sending TCP traffic on multicore devices;
I have been wanting to do some photos and thoughts on the Mikrotik SXTR-LTEs and other Mikrotik LTE products. I recently fired one up using dual sims. One is from Tmobile and one is from At&T. Verizon is pretty nonexistent in my area. I am about 2.5 miles away from a Tmobile tower and about a mile from a fiber-fed AT&T monopole.
As you notice in the following photo I am pretty buried in trees.
Some initial notes. Setup of LTE is a very easy process as far as the mikrotik is concerned. I literally had to put in some information in the APN and that was it as far as LTE goes. I did set up standard Mikrotik stuff (DHCP server, security, etc.).
Adding the second sim card can be a huge pain due to the location of the sim card slot. Luckily I had some tweezers that were angled to be able to slide the card in the slot. These were part of a dental kit I picked up off Amazon for releasing stuck SFPs and the like.
Look for a more in-depth series on Mikrotik LTE coming soon.