Meraki

Matt-Ignite · ‎Jan 30 2022

Hey @DM21 , To be honest, I'm not sure what ended up resolving this one. I know that's absolutely no use to you! There was tickets logged with both Meraki and the ISP, with both claiming that the other side was the one that was dropping the connection. After a few weeks of this, it settled down and simply started working. I suspect that someone changed something, but when asked both sides were adamant that they had made no changes. So I'll take them both at their words. I suspect it may have been something like Robin outlined above - that there was a firmware upgrade done on some carrier equipment, or something odd like that (that would have been handled by an internal engineering team with no line-of-site to active Support tickets) that settled things down. Maybe the ISP's monitoring turned up a dodgy carrier board somewhere that got replaced and fixed us? I don't know. In short, the site has been running perfectly since shortly after my last reply. It fails over to the 4G link as needed, and comes back onto the primary. But the primary stays rock solid - I couldn't tell you the last time it flicked over. I don't think my experience has been typical though - so I certainly wouldn't be overly concerned that what I went through will happen to you. Both the ISP and Meraki seemed quite surprised that it was doing what it was doing, which suggests to me it's not a common fault. If you're really concerned, you can set up the Meraki to email you when it does a connection fail-over and fail-back, so you're at least notified when it happens (only works of course if the Meraki is controlling the failover - if the 4G is on the upstream mode/router, you may have a different problem). I have my alerts going directly into my AutoTask queue so we keep abreast of such things (and get a feel for whether they're frequent or not). But obviously that only helps when someone's watching the queues - if it starts doing it on a Friday evening and the main VM backups all run over the 4G link on Saturday, then you're screwed either way. But even still, I'd rather have the backups protected offsite and need to top-up the 4G than go unprotected, so it's not completely bad. As far as how well it detects loss of internet and how quickly it fails back, there's good information here: Connection Monitoring for WAN Failover - Cisco Meraki . You've probably already read that, but it saves me re-typing out a complex set of measures. In short, if you have a soft-failure (link still up, but internet not actually working) you're looking at up to 5 mins of outage before the MX will fail over to the secondary link. Yes, that does really suck. No, I don't know how it's considered acceptable in 2022. Good luck! Matt

Matt-Ignite · ‎Sep 8 2020

That's what I thought/feared. I really wish there was more control over the flows in general. I'm considering going real low-tech, with a $9 manual timer on the power socket of the 4G modem. Set it to drop out for 10 mins every 24 hrs at night time, forcing the Meraki to fail everything back to the primary link. https://www.jaycar.com.au/24-hour-mechanical-timer-a-n-switch/p/MS6113 Don't get me wrong - I don't *LIKE* it, but it might be the cheapest and most reliable way out of the problem. Cheers, Matt

Matt-Ignite · ‎Sep 7 2020

Hi team Meraki, I come before you again with another WAN failover problem. Although, to be fair, I think this is really a continuation of the same problem I've had for years, just with a new client. The question: Is there any way (either from the front end GUI or via a support team modification through the back end) to configure a timeout for active flows to reset routing preference? The backstory: Meraki MX64 WAN1 - wired NBN internet link. WAN2 - 4G modem with static IP Meraki is configured with WAN1 as the primary uplink, with failover to WAN2 as needed when WAN1 drops. Problem: Client has an onprem VoIP phone system. Phone system maintains a long-running SIP connection for signaling back to the provider's environment Phone system processes the actual voice calls on a different IP/port socket. This bounces up and down as calls are made, etc. Client WAN1 has been unreliable, and dropping out. Aside - I suspect these dropouts are false-positives and are caused by link saturation which causes the Meraki DNS probes to time out. Either way, it doesn't affect the actual problem When WAN1 fails, all traffic is mapped over to WAN2. Phones continue to work, etc. WAN1 comes back up. Web browsing, etc, is mapped back to WAN1. Easy peasy - these flows start and stop all the time, so it's easy for them to time out (~5 mins from memory) in the Meraki and then be established next time on the primary WAN. But the phones, specifically the Signaling connection, stays on WAN2. Because this is a long running connection with keepalives, it never drops. If it does drop (by restarting the VoIP server, for instance), it attempts to reconnect to the upstream WELL within the 5min flow timeouts. Because of this, it NEVER fails back from WAN2. It just stays on WAN2, chewing up expensive data and having all the unreliability of a 4G/cellular connection. I understand that the software is designed like this so that it doesn't induce a second outage on the failback. For most things, this is fine. But in this scenario it means that some things stay on the secondary link forever. What I'd ideally love is a way to keep the default behaviour, but mark specific flows as needing to be reevaluated every so often (30 mins, say?). If after 30 mins the flow is reevaluated and better matches a different route, the flow preference should be hard-switched to the new interface/route. If that causes the connection to drop and need to be reestablished, so be it. Am I dreaming here, or has someone else already solved this problem with Meraki devices, long-running connections and WAN failover? Cheers, Matt

Matt-Ignite · ‎Jan 27 2020

Thanks for the detailed response Robin. I've run the ping tests, and determined that the max packet size is 1492 (right as you expected). A ping of 1464 works, but 1465 fails. So that's bang on the money for a PPPoE connection. Do you think it's worth me asking Support to set a lower MTU on the Meraki (IE, much lower, like 1400)? I would expect that since the Meraki knows it's doing the PPPoE encapsulation, it would have automatically adjusted itself down to suit? re: the cabling works - it was a simple replacement of cabling from the server rack through to the ISP pit on the footpath. Same length of cable, and same ISP run back to the exchange. That said, I wouldn't be surprised to find if there may be a problem on the new cable terminations (at either end). IT's not likely, but I have had that before where the tech simply had a bad day. I'll have a look at the SnR next time I can arrange to get someone onsite. Because the modem is bridged, it's a pain to get onto the local management interface. Cheers, Matt

Matt-Ignite · ‎Jan 27 2020

I probably should give that a try. I'll have to double-check all the port-forwards that I need, but it shouldn't be too hard to switch it over. You're right in that it's probably the next step to try.

Matt-Ignite · ‎Jan 22 2020

Hi team, I've got a problem that I'd like to pick your brains about. I've got a Meraki MX64 deployed at a client site. Connected to the Internet port is a VDSL modem (carriage is via Australian NBN - FTT). VDSL modem is in bridge mode, and PPPoE Auth is set up on the Meraki. This setup has been in place for a few years now, and over this time has been completely rock solid. Meraki gets the public static IP terminated onto it, and everything works swimmingly. Late Oct/early Nov last year, there were some cabling "problems" at the site, caused by a concrete saw and a 10t excavator. New twisted pair was pulled in new ducting all the way in from the street, so the cabling should be brand new (that was the only upside). Site was back working faultlessly for 4-6 weeks following this - back to it's normal rock-solid reliable state. Early Jan, the service started to experience frequent (and prolonged) failures of the primary link. This would occur multiples times during the business day, and some of the outages would be for hours at a time. The site is configured with a backup 4G link, and the Meraki would fail over connection to the backup in the event of a primary failure. However the backup link is slow and expensive. Also, the failovers are disruptive with the timeouts that Meraki specifies. Obviously we want it to stop and go back to it's nice reliable state. Suspecting the cheap ISP-provided VDSL modem had failed, we replaced it with a brand new one ASAP. No good - same dropouts. Nothing else at the site has changed. All equipment is installed in a locked 19" full-height rack, in a 24/7 air-conditioned room, and all cabling is secured behind the rack. No obvious damage to cabling from rats, no one has mucked with anything, etc. I've logged tickets with both the ISP and Meraki support. From the ISP's side: ISP says that they can see the PPPoE connection dropping, and then being reestablished when the Meraki next asks for it. ISP can see that throughout all of this, the VDSL modem has maintained sync with the network, so they don't think it's a carriage issue. They believe that the Meraki simply isn't connecting the PPPoE tunnel. ISP has run a line quality diagnosis test however that is showing abnormal attenuation on the line, so they suspect the internal cabling (the brand new cabling) may (suddenly) be faulty. From Meraki's side: Meraki have looked at the logs, and can see nothing wrong with the MX64. From their side it looks like the ISP is dropping the PPPoE tunnel and then refusing connection requests for a period of time until it successfully reestablishes. One thing that the Meraki tech did note is that the MX64 is not seeing a PADO response from the ISP on a lot of occasions. They believe that the ISP side may be overloaded, and it's simply not responding to PPPoE connection requests. The MX64 is running ver 14.40 firmware. It was on the previous version when the problem began occurring, and upgrading to the current fw was one of the first things I tried. @Bri84 seems to have had a similar problem in October 2019 (albeit a different ISP): https://community.meraki.com/t5/Security-SD-WAN/PPPOE-Issue/m-p/63557/highlight/true#M16134 The only thing I can think of is maybe it's a MTU problem? I have no idea why it would suddenly crop up, but maybe the ISP replaced a piece of equipment in their network that's got a lower MTU setting, or something? adam2104's comment in this thread: https://www.reddit.com/r/meraki/comments/cax4cf/obscure_meraki_firewall_problems/ is kinda what I'm thinking of trying next. Not that it should affect the PPPoE tunnel setup at all, but maybe? Does anyone here have any thoughts on what might be causing this, or what I need to try next? Meraki Support have asked for a packet capture, but obviously catching one in the act is difficult. Other than that - anything that I should be looking for/at? Cheers, Matt

Matt-Ignite · ‎Jan 22 2020

Hi @CaptainDan and @MikeG1 , Apologies for the late response here - for some reason I never received an notification about your posts. I never resolved this. In fact, shortly after posting this the client was so upset with the unreliability of the phones that they pulled all contracts with me and moved to a new provider. Honestly I can't blame them - I spent months trying to get this to work, and it simply wouldn't. 3CX blamed the Meraki device and how it managed it's failover. Meraki blamed 3CX and said their device was working perfectly. Each party was able to produce logs that backed up their position, but at the end of the day the system as a whole simply didn't work. I suspect Azure IaaS might have had a hand in it, specifically how it handles it's networking internally. I found traces of different forum threads across the internet that suggested it was doing something very strange that normally no one cared about, but this exact combination tripped up on. I know people who are running AWS-hosted 3CX with Merakis, and it's fine. I know people who are running Azure-hosted 3CX with other routers/firewalls/UTMs, and they're fine. I know people who run 3CX on-prem on a NUC behind a Meraki, and it works fine. But there was something with this combo that simply did not work reliably. From memory, I was also looking at a potential problem to do with the SIP INVITE Header field from the Yealink phone getting too long. I can't remember specifics, but I remember eventually finding one forum thread somewhere where a handful of people were having the same problem. I can't find it now, of course. My next port of call was to deploy a Pi-based SBC to the site, to see if that made the problem any better. 3CX at the time didn't manage the SBC's well (the SBC's themselves worked well, but there was no visibility from the PBX into the SBC status or any remote manageability. This has now got a lot better), which is what had held me off. I had the SBC built and couriered to the site - the morning it arrived was the morning that I received the phone call about losing the client. Again - completely understand where they were coming from. NB - in my case the SBC would have only affected the single Yealink IP phone. The softphone apps all create their own tunnels back to the PBX. Given that the softphones were also giving me a lot of grief, deploying an SBC would have only, at best, helped shed some light on the problem rather than resolve it completely. My original thread in the 3CX forums is here: https://www.3cx.com/community/threads/yealink-t58-407-proxy-authentication-required.58528/ I've got a feeling that wasn't the end of it though - it wasn't as simple as "the Meraki was dropping inbound packets on the secondary when the primary came back up", otherwise I would have simply replaced the Meraki. Again, from memory (which really isn't good), I think packet captures on the inside of the network showed that wasn't the case. I'd be very interested to know if either of you gents were able to resolve this, or even if you managed to make any headway in troubleshooting it. I spent months and months on it, and came out even more confused than when I went in. I'd consider myself pretty adept at networking and IT in general, but I came up absolutely empty on this one. Sorry! Cheers, Matt

Matt-Ignite · ‎Oct 3 2018

Hi team, I've got a problem with a site of mine that's running an MX64. I'm hoping you can help shed some light on what might be going on. The primary internet link is a FTTP NBN service (for non-Aussies, this is presented as an ethernet connection). I'm using a NetGear LB2120 device on the second WAN port (LAN-4) for a 4G connection as a failover link. This has worked well, and been a lot more reliable than using a USB modem on the Meraki. When the Primary drops, the Meraki detects the failure in around 60 seconds, and fails over to the secondary WAN. When the Primary comes back up, the device fails back. Around 3 months ago, we deployed a cloud-hosted VoIP phone system to the business - a 3CX server running in Azure. There is one physical Yealink IP phone, and all the staff have the softphone app on their iPhones (which are all on the local wifi, connecting out through the Meraki. All phones are STUN provisioned - there's no SBC on site or tunnel. Again, this works really well. The problem comes in when the Meraki fails back from the Secondary to the Primary. Failover to the Secondary is OK, because the Primary interface goes down. All traffic is routed to the secondary Failback, however, induces problems. From what I can tell, when the Meraki fails back, it only fails back NEW "flows" to the Primary. In an effort not to cause dropped calls, etc, it leaves any existing flows operating over the secondary. A flow persists for up to an hour with no traffic passed before it expires. The onsite IP Phone keep chatting to the PBX though, which keeps that flow alive and keeps it operating off the Secondary WAN. This wouldn't be a problem, except that I believe the Meraki is potentially dropping traffic that's coming back inbound onto the secondary interface. The end result is that the IP Phone is stuck whereby it can talk out to the PBX (make calls), but it can't receive inbound calls/connections. It also continues to chew bandwidth on the 4G, eating up the data cap on that device. The fix for this seems to be a reboot of the Netgear LB2120 device, which drops the secondary WAN interface and forces the Meraki to map all flows back to the Primary. Today though, that's not been enough and a bunch of the iPhone softphone apps are taking ~35 sec to start ringing - I suspect their PUSH is still registering out the 4G secondary. The 3CX support are saying that the PBX config is fine, and they're not seeing the attempt from the phone to connect. I've got the Wireshark traces, and from what I can see, they're right. They suspect that the Meraki is doing somethign fancy and it's making it all go south. Has anyone here experienced a similar thing? I really don't want to have to reboot this firewall after any failover/failback event - seems to sort of defeat the purpose of automatic failover/failback if I've got to DOS myself. Thoughts? Cheers, Matt

Matt-Ignite · ‎Oct 3 2018

These look really good! As someone who's had no end of trouble trying to get a USB modem working with the MX64 devices (and ended up giving up because it just won't work here), having an integrated LTE modem is a real benefit. The downside is obviously the cost - the MX67C is twice the price of the MX64 by the looks of things. I'd pay a bit more for integrated LTE, but an extra $1000 is a tough sell to a customer when a USB dongle is only $50. On the official Documentation page for the new devices (https://documentation.meraki.com/MX/MX_Overviews_and_Specifications/MX67_and_MX68_Overview_and_Specifications), there's mention of being able to set a custom APN via the dashboard UI. I don't support there's any chance of getting this added to the MX64 devices as well? That would be a huge help when dealing with 4G failover. Cheers, Matt

Meraki

Community

About Matt-Ignite

Matt-Ignite

Community Record

Badges

Re: PPPoE through VDSL Modem - connection dropouts. Missing PADO?

Re: WAN failover - flow timeout?

WAN failover - flow timeout?

Re: PPPoE through VDSL Modem - connection dropouts. Missing PADO?

Re: PPPoE through VDSL Modem - connection dropouts. Missing PADO?

PPPoE through VDSL Modem - connection dropouts. Missing PADO?

Re: MX64 - 4G Failover and STUN-provisioned IP phones

MX64 - 4G Failover and STUN-provisioned IP phones

Re: What are you most excited about with the new MX and Z-Series?

WAN failover - flow timeout?