MAIL - Meraki array of inexpensive links

[PhilipDAth]

Back in 1987 some lads at Berkeley came up with RAID (redundant array of inexpensive disks).  Mainframe hard drives were very expensive, and they figured out they could combine personal computer hard drives into an array, and make something both cheaper, faster and more reliable.

I did some maths over the weekend, and I've decided to coin the phrase, MAIL (Meraki array of inexpensive links).

Did you know if you combine independent low cost circuits with a low uptime you can end up with with an incredible uptime when combined with Meraki SDN?

For example, domestic fibre in my country typically has an advertised uptime of 99.99%.  A 4G rural circuit (note - the second link needs to be something independent of the primary circuit) has an uptime of 99.9% (this is the worst case for rural users - urban users will observe much better numbers).

So if you combine those two the probability of them both failing at the same time is 0.0000001, which yields an uptime of 99.99999%.  That is 7 nines!  That equates to an access layer failure of just 3.2s per year.

Background maths:

99.99% uptime = downtime of 0.01% = probability of downtime of 0.0001

99.9% uptime = downtime of 0.1% = probability of downtime of 0.001

Probability of downtime = 0.0001 * 0.001 = 0.0000001

Consequently, probability of uptime = 1 - 0.0000001 = 0.9999999 = 99.99999%

Uptime of 99.99999 % = 3.2s per year of downtime

[Adam]

I agree with this 100%.  We have been moving to redundant inexpensive links at a lot of our sites in lieu of older expensive Metro and MPLS connections.  The failover and load balancing is seamless and the sites become far less of a headache for us to manage.  Not to mention the cost savings. 

[bigben386]

We dumped our MPLS 2.5 years ago and switched to dual internet with Velocloud. Most sites have Fiber or Cable + LTE. Our uptime, bandwidth, MOS scores, etc have all increased over single MPLS and we have saved a ton of money!

[kevinl]

Ahh, that's the beauty of SD-WAN and the simple load balancing that Meraki provides 🙂 I do insist on carrier diversity - separate ISPs for primary and secondary WAN, because being ex-telco myself I have seen rather..... large....outages impact service. Carrier diversity has saved the places I'm at so, so many times.

We use cheap and fast GPON/FTTH links, and with our own diversity setup we can save a lot of money by going with the more affordable links without as much SLA protection, which is expensive, and often does not provide much assurance anyway - the link still goes down, the harm is done, and people have a field day arguing anyway, while the Internet is still down.

Interestingly, telcos also have a legal right to do maintenance outside of 'regular' business hours and take your circuit down in my part of the world, for up to 4 hours, as long as they provide adequate notice in advance. Carrier diversity also addresses this issue for 24x7 shops.


And for Philip - I have seen one truly interesting scenario where instead of getting static IPs for the MX warm spares, they [names deliberately omitted] got four cheap links with 1 IP address each, going to the same site. We cabled as follows:
Primary MX Internet 1 - ISP 1
Primary MX Internet 2 - ISP 2
Secondary MX Internet 1 - ISP 3
Secondary MX Internet 2 - ISP 4

(there is a thrilling backstory to why there were so many lines installed before I came into the picture and they couldn't get rid of them so we used them as best as we could)

Interestingly, this configuration actually works. If a primary MX in a warm spare group loses BOTH Internet uplinks, it will also automatically surrender the master role to the spare, provided the spare can still reach the Internet. We found this out the day a backhoe cut three out of four links in the street outside.

The probabilities are insane, and it happened anyway. My MX setup stayed up on the warm spare, and I could not believe my luck both ways (with the backhoe and the probability of having 4 links at one site).

Now you know how to use up to 4 ISP connections with an MX, although the warm spare's uplinks will be completely idle and cannot be load balanced. 

[ErikPeterson]

Just to chime in on this thread... I totally agree that MAIL is more cost effective and fault tolerant than any other single telecom solution (copper, fiber, directional wireless, 4G, etc...).  My only problem with MAIL is that it is difficult to monitor all of the WAN links for a site(s) from a "single pane of glass".  I've started monitoring the external IPs of all my WAN links using SolarWinds NPM as a work around, but I wish that the "view all networks" screen in the Meraki dashboard would color sites orange if there was a down/degraded WAN link at the location.  MAIL is only as good as the weakest link, and if you think all your links are up, you will be in for a rude awakening when your primary links go down and you discover that your backup links have been down for the last three weeks...

Thanks!

-Erik

[kevinl]

Totally agree with Erik. I wish there was a screen where you could see all the uplinks and realtime traffic graphs simultaneously, like the 16-camera MV view.

I am showing my age here but if anybody remembers old-school MRTG that showed all your links and all your traffic on one page - yes, I miss that.