MX consuming GB of bandwidth with management traffic

[lpopejoy]

We have one site where they use a Verizon 4g connect (and pay per GB for overage.).

They contacted us last week and said they had received a bill for $6,000 in overage charges.

We worked with their provider last week who said that their 4G modem was compromised with malware.  We RMA'd that, and figured the problem was resolved.   ...Until they reported that 50GB of traffic was used the following day. 

This is a small site - a couple computers.  Traffic analytics shows nothing - like a few GB over a month.  It is extremely inactive. 

We worked with the ISP to get a packet capture from their layer 2 device...  This revealed around 15 MB of traffic over a 5 min period - all going to one of Meraki's management IP's.

This is extremely abnormal, and very sketchy.  Has anyone else seen anything like this?

I do have a case open that was sent over to engineering for me.  I have a bad feeling about cases that get sent to the black hole of engineering. 

[PhilipDAth]

I have not seen that - but I think I would factory reset the MX unit and let it pull a fresh config from the cloud.

https://documentation.meraki.com/zGeneral_Administration/Support/Resetting_Cisco_Meraki_Devices_to_F...

Is it running stable firmware?

[lpopejoy]

@PhilipDAthIt was running stable (12.24) until Friday.  I updated it then to 13.24. 

I can reset to defaults, but I wanted to give engineering a chance to dive into it before doing so.  I would love to get a straight answer on what's going on with this. 

[lpopejoy]

Oh, btw, did you know the packet capture excludes traffic to Meraki management IP's? 

Unfortunately, I did not...  Therefore it took me a week to find this problem. 

The only way to see management traffic is to use a filter and filter for management IP's directly.

[Tat0rt0t]

Good to know! Thats for bringing that up! 

Unfortunately I haven't experienced this, but I don"t have data limit issues currently. Im curious what you find out though, sorry I cant help 😞 

[PhilipDAth]

---

@lpopejoy wrote:

Oh, btw, did you know the packet capture excludes traffic to Meraki management IP's? 

I didn't realise that.  It must only do traffic passing through the appliance.

[lpopejoy]

Well, if that were the case, you wouldn't be able to see the traffic when using a filter "host x.x.x.x." 

I think the more likely scenario is that there is a default filter behind the scenes that is filtering out all management traffic, but when you explicitly specify a filter, it overrides this.  It kind of makes sense, I just wish there were more visibility here.

In this case, the Verizon engineer had to do a packet capture on his side, which gave me the offending IP, that I could then use to filter for in the Meraki packet capture.  It made me look quite stupid when I was saying the problem isn't on our side, and then come to find out I couldn't see all the traffic.

[lpopejoy]

Here is what support just sent me on this issue:

So it looks like the MX did not successfully upgrade to 13.24. It's currently running on a firmware build that has since been deprecated. This is most likely the reason why we are seeing the device continue to reach out to the Meraki Cloud. It's attempting pull down the firmware you configured in dashboard. But for one reason or another it seems to be failing. When is a good time for me to call you back so we can see if we can get it to successfully upgrade it's firmware? It would be best to do this during a maintenance window in order to minimize the impact to network users as we may need to reboot and reset the device more than once. If we can do this during my shift, I'll go ahead and plan to give you a call. If it's after my shift, then you can just call into our Support Hotline and someone will work with you to get the device running on 13.24.

[Tat0rt0t]

Wow Ok, thats wonderful, does the device show it is running on 13.24 under firmware?

[lpopejoy]

It says "Firmware Up to date" on Appliance Status, yes.  However, that just reflects the firmware it is configured to run, not what it is ACTUALLY running.

AFAIK, there isn't a way to view what a device is ACTUALLY running. 

Under Organization -> Firmware Upgrades, it shows it is running 13.24.  It also shows this:

MX 12.22 → MX 13.24

Completed Oct 2, 9:29 AM

However, I did the update last Friday...  (though apparently this doesn't show in the audit log so I can't fact check that)

[Tat0rt0t]

@lpopejoy Thanks for the update, This makes me want to verify with support what version I am in fact actually running since from your point a view, you did update and the dashboard claims this. 

[lpopejoy]

Ok, so apparently if you want to see what version of firmware you are actually running, it will, in fact, require support to verify this information for you. 

On this particular network, the appliance was running v10.  That's right.  Yes, v10.  Yes, deprecated firmware.  No, nothing was telling me this.  The dashboard was saying 12.24 "firmware up to date" AND allowed me to update to 13.24 and showed that it was also up to date.

The reason for the excessive bandwidth use is that the MX has been trying to download the latest firmware update since last week when I pushed 13.24.  Before that, I can't say what was going on, unfortunately. 

[Tat0rt0t]

---

@lpopejoy wrote:

Ok, so apparently if you want to see what version of firmware you are actually running, it will, in fact, require support to verify this information for you. 

On this particular network, the appliance was running v10.  That's right.  Yes, v10.  Yes, deprecated firmware.  No, nothing was telling me this.  The dashboard was saying 12.24 "firmware up to date" AND allowed me to update to 13.24 and showed that it was also up to date.

The reason for the excessive bandwidth use is that the MX has been trying to download the latest firmware update since last week when I pushed 13.24.  Before that, I can't say what was going on, unfortunately. 

---

Well thank you for clearing that up - Guess i'm going to contact Meraki support and have them verify firmware versions as well as going forward after I perform them. 

Im going to bring this up with my Account rep and i suggest you do to, get this sorted so we can actually know firmware has been applied. 

[lpopejoy]

Support did an audit all of my MX's (around 60 or so) and all of them were >= 12.24, except the one that I'm having this problem with.  So that's good news.

Additionally, engineering is saying this MX has a hardware issue and they are replacing it.  I guess that's good news too.