I'd be curious your experience with the mx450 from that, as all of ours are tipping over still after RMAing us from mx600's.  Using them in sdwan with upward of 1100 mx's in autotunnel to the hub(s), they can no longer keep up, particularly since 16.x last year, and support actually suggested they'd upgrade us first (then sales fought it, no commish in free!).  I adopted these for a customer after their last meraki guy ran away, and found out quick that 5000 max vpn count in the data sheet is a joke when things started breaking.  First the number given was "real world some ~1500" tunnels we heard from cisco, and the reality is they can't do much over 1100 now on the mx600/450 with them pooping out at some 85% usage.  Now faced with random outages they're having to shard all our branch networks out across 5 hub locations and soon a half mil $$$ later, the customer is not terribly happy.   Sad part is Cisco doesn't themselves have a good reference guide for sharding networks for very large branch quantities across hubs since the wilette.works site died...  Good thing I remember it to recreate. ... View more

Thank you @spadefist, a lot of this I suspect with our current deployment as mentioned, probably far better ways to approach this.  We do this as hub-mode today, again from way-back before the Cisco Borging, and has only evolved minimally along the way, but we do not even specify dual tunnels for full backup.  I suspect it's time for an entire redesign, but no one is bringing this up from an account perspective short of jumping on calls to help when it blows up to brow-beat tac.  Not ideal, and painful from even my perspective to watch.   I look at this like database sharding, striping of concentrators, or whatever you want to call it, but distributing load among multiple distributed endpoints, whatever the method is ideal per what you're saying - we're just not doing this today.  We just need some help to speak some hard truths to the customer that for being a 6B a year company to stop being cheap of they don't want to lose their millions per minute during these store outages that occur far more frequently than we should lately.   See my prior post, I agree with most everything you say, but I'm just a random contractor, plus my time here is nearing an end.  I'd just like to see these guys get some help if not going to listen to me.  No one on the account teams wants to stop them and say "Hey, you're doing it wrong," and I fear my saying it is starting to become rhetoric.   I would suggest you ping the account team if you can see my email address to the account to ask what's going on, as they do need some help here I think, and a sizable account to keep in good graces.  I have a few weeks left on the payroll, but would like to help this along as I hate to see these sorts of issues persist for anyone. ... View more

@Bruce, large retail chain, much legacy debt of a 60-some year old company, etc etc. This solution was purchased before the Cisco Borging, and I think mostly grown unwieldy now.  We have a lot of active Meraki team involvement, but seems no one there has enough gumption to tell us we're doing it wrong, thus these things are biting us now quite regularly.  I'm now the man in black bringing bad news to the table.   I appreciate you and @spadefist giving input on this, as again a bit of my first rodeo with Meraki.  News to me it's capable of prepending even simplistically, as I've not found anything that says it does any metrics in bgp, automagically or other.  This obviously means ebgp relations which I'm all for, but it's still a mix of legacy eigrp, kludge of ospf atop for Meraki as all they did originally, and bgp with mpls wan provider environments that's entirely discontiguous.  I can't blame you for that, but lack of metric control options is, even if you hide it from the lay folk using these things.   Case in point, I've argued if not doing regional hubs, not concentrator mode, we split the ha and term to each store mx meraki separately with dual tunnels to fail over separately to each hub, assuming we could make routing failover faster.  Ideally we go between sites, but again today we're ospf just for meraki with a squishy eigrp middle.  Normally I'd do ospf external type 1, metric one site significantly higher, and run between them for one desired path or another standard, but needs done as a matter of policy, ideally from the meraki or it gets messy in translation (got tags?).  Further complicate that with eigrp around it in the middle of everything, and it comes apart, but the rest of engineers feel ospf is a better long-term option so maybe feasible to overlay and migrate.   I've pushed for ebgp entirely, as I've built whole service providers and clos fabrics around ebgp, vxlan overlay/underlay, and not afraid, but everyone else is, so a bit of a non-starter.  They might consider ibgp+ospf. but now we'd be talking local-pref injection which it doesn't do to control traffic between sites.  I've been through igp+bgp vs. native ebgp-everywhere and prefer the latter, but it's still a foreign concept to most.  It would be nice if these could support things like BFD as well for bgp fast failover, automagical or manual for downstream peering.   I just don't get a lot of control with the routing protocol for how the meraki's can signal downstream a flip in preferred paths between sites vs. dmvpn, viptela, or even fortinet products I've worked with over the years.  It's annoying to me, thus the fischer price comment.   This is not typically my domain and try to stay out of it, but after enough rude overnight wakeup calls myself about these failing in ugly ways, they need to do something.   Oddly, the biggest issue we see has been with a few firewall failovers that cause hung connections in the meraki tunnels through our firewalls in the event of an outage.  There seems to be no proper DPD sort of checking on the tunnels to restart sessions, or any sort of again BFD-style checking available to ensure tunnel health occurring to even restart the tunnel.  This is another huge annoyance, as at least with DPD and a hung connection, a new inbound establishment should correct this, but instead we still end up with ~1200 retail stores stuck until we forcibly flush connections on the firewall, which causes the Meraki to trip out, break, go dual-active, etc due to load today.  Yada Yada, round and round.   Seems some minor features could alleviate a lot of this, but just your average network monkey here.  It would be nice to see better engagement with the Meraki folks, but not normally my domain, and no one seems to want to give some hard truths to the customer (except me) as a consulting architect.     ... View more

No idea even myself he worked for Meraki, or just a large enough consumer with some common sense.  It be nice if this would go out as a SRND/CVD or some other Meraki-blessed gospel design as it made sense more sense to me than anything I've seen in Meraki docs.   I'm not at all even certain why our account team doesn't push them more in this direction, but we've had several major outages related to either capacity (ie. the upgrade) or peripheral outage that has caused a lack of failover to a single cluster with hung connections (a whole other ball of worms).  We keep pushing out the upgrades now, and otherwise only input we've gotten is to sell them replacement mx450's vs. horizontal scale.  I'm just left scratching my head, but trying to not inject myself in the meraki problems more than I should. ... View more

Hi Phillip,   >> You can't expect Cisco Meraki to halt the development of new features that new hardware can use just to allow someone to hold onto an old device that is operating near its maximum capacity already.  Just doesn't make business sense. It's nothing personal.   I don't disagree with you - I was rather shocked when Meraki TAC recommended it and said they'd be sending us 3x new mx450's.  I just assumed we weren't the first customer to be blown up with this, and was some fallout control, but with that being a major outage, most upper management heard it as I did, and reneging on it later was even more an ugly black eye.  Why I figure that TAC guy is probably looking for a job now, or told to stfu at least.   Because routing has been a mess no one wants to tackle, even with us possessing regional hubs, they haven't been put into use.  I suspect it'll probably be this way after my time here consulting.   Funny you mention Aaron's link about that, it's as close to something describing this I can find even over official docs to show this.  I've sent this out numerous times to my team, but since it doesn't have a cisco logo on it, it hasn't been taken as any officially sanctioned design.   Again, I've somewhat taken to playing dumb around the Meraki issues as a whole, but considering when we affect other things and it inadvertently breaks Meraki in ugly ways resulting in wake-up calls, I'll give it one last stab to try to lead a horse to water here. ... View more

Thanks Bruce,   Yes, have seen the sizing docs, but these have been deployed far longer than I've been here, and just sort of the corner they've painted themselves into now.  Unfortunately we don't get a lot of account support that says "here's what we *should* be doing so far, I think mostly as no one wants to admit there are limitations we've been lead into overrunning.   Most of our stores are dual ISP, however not actually using dual tunnels even due to current limits, however they continue to move more stores from mpls to sdwan, and not accounting for a proper redundant, scalable design.  Any time we lose a single dc where our hub cluster lives, it's getting more and more painful.   While we're at some ~1500 stores total including converted sdwan and tbd mpls-to-sdwan conversions as capable, the goal is to keep growing more branch stores as the business deems suitable, so this number will definitely go up for spoke sites.   On routing, these were deployed using OSPF with ugly redistribution into the core EIGRP (eww), remote sites across BGP all (poorly) meshed, as that's all that was supported at the time of implementation many moons ago.  I've pushed for moving to BGP entirely, as well as recommended by the account team to do so long term, but BGP scares folks here.  I've looked at and designed some logistics to do so, biggest problem I run into is lack of metric control in Meraki, such as prepending/local-pref/etc to metric redundant hubs as primary/secondary within the internal network without external mangling via route-maps necessary on real routers.    It would be real nice to do some prepends or local pref on secondary hubs/tunnels, or flipping at the hubs when necessary to control this, but this doesn't fall into your "simplicity" model.  It's like dealing with the fisher-price of networking here, my first sdwan!  I've worked some with viptela for other customers, at least there I tend to get a full feature set of ios with as much or little control as I need, but seems we're not the only "big" customer to need this sort of thing growing up with Meraki too.   Thanks for the validation on the multiple hubs, when I have brought this up, I get looked at like I have 3 heads so far internally.  I usually just play dumb anymore when it comes to Meraki here. ... View more