Cellular only connection failing due to "NAT: Unfriendly"

[MT]

We have several devices in our network running on cell only, either Verizon or ATT, creating VPN tunnels.  We have one device running ATT service on a compatible modem according to the list on the Meraki site (AT&T Velocity USB Stick).  That unit is not successfully building a VPN due to "NAT: Unfriendly"    We have not run into this with any of other other cell onlys.... usually the only time we see that error is settings within the site firewall, usually SonicWalls.  But that not the case with cell only internet of course.

Anyone had cell only VPNs fail due to unfriendly NAT?

[SoCalRacer]

This sounds like an MX device or Z device. This is a WLAN topic, so might be moved.

It sounds like the Carrier or Device is doing NAT. If on the modem you can forward the ports that should work. Another option is to try using a different APN. If you can provide some details about the devices, like firmware, model, etc.

If this is a MX make sure it has been connected via WAN cable to connect to the dashboard to update firmware before troubleshooting any of these issues.

Another item to check is with the USB plugged into a computer do you get internet access?

Verify the USB ID =  19d2:1225

This USB stick is not supported on a Z series

[MT]

The Meraki we're using at this site is a Z3 with latest firmware.  This is a remote site, but they gave us the ATT USB modem info was given to us as SW-VER:MF985V1.7 Model: MF985.

These are almost all remote sites, sometimes far away (it is Montana after all) and in rural counties where the Cell only possibility of these Merakis is very tempting for them.  But unfortunately we don't often have much access to the actual devices, or even to the PCs being used, etc.  We are simply responsible for building the VPN tunnel to get back to secured info on our servers.  So we troubleshoot with the folk on the site and they do best they can at getting us the info we ask for, etc.  We have a lot of Z series running USB modems...strangely we see the older ones seem to be a lot more solid, while many of the newer modems cause nothing but problems and network drops (the 720L modem with Verizon, for example)...even though it is listed as a compatible modem on Meraki site.  I worked with Meraki techs for weeks troubleshooting before they just gave up.  We just don't use 720L now, which is a bit of a drag because it is the main one Verizon will offer.  

Anyway, yes as you mention I figure the device must be doing some sort of NAT'ing of its own (I have not seen this in any of the Verizon ones I've dealt with)...  so possibly going into the carrier software and setting up some kind of port forward may work, though again then we are trying to walk some remote site employee through the process.  It does make me question the modem compatibility list of Meraki though, as we are finding more and more models that are not playing nice without configurations, forwarding, if anything....

[SoCalRacer]

Since this modem is not supported on the Z series YMMV.

Do you have the firmware version? Was it recently deployed? Did it update firmware via the WAN cable before switching to cell?

[MT]

Where do you read that the Z doesn't support USB modems?  I would be interested to see that, since last i knew they did:

https://documentation.meraki.com/MX/Cellular/3G_-_4G_Cellular_Failover

Under the supported modems they mention it for MX and Z series.  And by in large, we've had decent success with verizon cell connections anyway, with the old Pantechs....the new 730Ls do NOT work for us, consistently.  I know they are not recommended for cell only but rather to backup loss of wired connection, but in our case it has had to suffice in many sites.

[SoCalRacer]

Check the support matrix this specific modem (ATT w/ Velocity USB Stick) isn't supported on Z series.

Also note "* Requires connecting the MX to the cloud to download the latest cellular modem definitions"

[MT]

I see what you're saying; thanks for the clarification.  Interestingly enough, the Meraki shows a stable internet connection with that Velocity Stick modem, much better than it did on their old Sierra Wireless AirCard® 313U  which was on the supported list (now discontinued).....but that 313U didn't work at all.  

We just are getting that unfriendly NAT killing the VPN tunnel, so as folk have pointed out, either something acting as a NAT with the modem device connection, or needing some sort of forced port NAT in the config to make it happy, which takes a bit of wrangling on the client's side and isn't exactly plug and run.  I'm offering up a few things to try for the folk there and have asked they engage their AT&T rep as well in case we need to bring them into the picture.

[Sameh_Sackla]

there is nothing to do with internet access. you will have a stable internet connection regardless of being hit by CG-NAT or not. CG-NAT breaks the VPN tunnel. try following the instructions I mentioned in my previous comment and see if this helps.
We have the same issue in Europe on multiple vendors with multiple providers and this workaround on the MX seems to have solved it. Let me know if you need further help.

[MT]

Right, I hear you re: the NAT.   

I was meaning we've had modem devices on the Meraki compatibility chart that have not even allowed stable internet connections, so without that reaching the cloud we have nada right off the bat...the 730Ls were an example of that for Verizon until Meraki yanked it from the compatibility list after we had already had a number of sites putting them into use.  We don't want some production solution going out to dozens of locations that take too much finagling every time.  So to me it was interesting this Velocity is not on the compatibility list yet is showing steady internet.  We never got steady internet on the last modem they tried, and it WAS on the compatibility list.

In the future we want to give the Z3C unit a fair look and skip this whole USB modem stuff, but as of yet Verizon is not supported on those so that's a deal breaker, and of course they are WAY more expensive than a standard Z3 and sites are looking at the cheapest modems they can that are on the list.

yep, I will try the steps you mentioned..in fact i had already set up the manual NAT on our side via the portal, but having the customer on their side work out the port forwarding is a task, as these are often just small offices and dispatch employees, etc., not I.T. staff or someone designated for this stuff.  At this site she is already fed up and is just packing the whole thing up and bringing it to At&t to see what they can do.  Which may be a good thing. We can hopefully have them set up the forwarding on their side and give that a go.  Or, alternatively, try some other modems or different APNs, etc., that don't cause a firewall block.

At ethernet/switch sites, we encounter that Unfriendly NAT a lot, but having them change the persistent NAT setting on their own firewall (usually SonicWall) always does it.  But i have not seen a modem connection have a firewall/NAT problem until this one.

Thanks again..

[PhilipDAth]

This is frequently an APN issue.  Find out what other APNs are available that provide "direct" Internet access without going through a firewall, and then program that APN into the device using a notebook.

[PhilipDAth]

One more thought - have you plugged this moden into a notebook and confirmed it works?

I have seen this happen before when a SIM is not activated properly or has run out of credit.

[MT]

Thanks, I will offer up some of the recommendations to the site...walking some folk through some of that, or having them work with their carrier, etc., can be challenging.  

[Sameh_Sackla]

That's due to a CG-NAT issue which breaks client-server communication. There is a possible workaround to punch VPN tunnels from a "spoke" site behind Cellular to a "Hub" site sitting on a fixed internet connection.

Follow these steps:
1-Make sure the cellular site is a spoke
2-Make sure the hub is a fixed site (NOT cellular)
3-For the hub site, configure Manual NAT traversal by choosing a UDP port of choice (e.g. UDP 1234)
4-For the hub site, in case of ANY firewalls upstream your MX facing the internet (e.g. Internet -> Firewall -> MX) please add a port forwarding rule on the firewall (NOT the MX) to allow ANY traffic towards the MX on the UDP port configured in step 3 (For example: Port forwarding source-ip-any source-port-any MX-WAN-ip UDP-1234)

That should fix the tunnel issue and you will have bi-directional traffic between spoke and hub. You might still have NAT Unfriendly on the VPN status page, but indeed you can do site to site ping and tunnels will be stable.