Portal connectivity from 2 Africa sites

[Richard-Tapp]

We have over 100 Meraki AP sites currrently running, so we know our standard config works ok in the majority of cases.

We have 2 sites in Africa, different sides of the continent and next to countries that have working APs.

These 2 sites nearly always show down in the portal, but connect for about 5 minutes ever few hours. So they get up dated and pull details like pre shared keys etc and locally they are working as expected.

A network capture shows them sending and receiving UDP data to the Meraki portal IPs.

We have opened full IP outbound to test, but still nothing.

Has anyone else had this type of issue ?

[Rimccart]

Hi Richard-Tapp,

From the issues that are described, it may not be specific to Africa that would cause this constant disconnection behavior for the networks. To help narrow the scope a bit further is this issue only observed to impact the device management traffic to dashboard, so clients do not see an issue themselves?

When the APs are disconnecting is it a short flap or will be be prolonged periods?

Is there any means to recover the APs manually or do we wait for a natural recovery?

What is the current AP firmware? Newer firmware should be using TCP 443 for dashboard communication. It may also help to have APs on the latest firmware as well (MR30.5 currently). 

What IPs and ports were observed in the packet captures?

[BlakeRichardson]

1. are the access points using the correct country setting?

2. Have you opened a support ticket. I think in this instance having support being able to see more in depth logs is going to help you resolve this. 

3. Could also be a stuck firmware update. 

[PhilipDAth]

I think this is going to need a support case to figure out ... someone is going to have to look at what the backend is seeing.

[Rimccart]

Both messages from @PhilipDAth and @BlakeRichardson are helpful. A situation like this is going to need a more in-depth review of the device-to-cloud communication and Meraki Support should be able to help with this. 

If we can, we should try to avoid rebooting the devices to recover them. Rebooting an offline device before it has an opportunity to reconnect will clear device logs and then Support would not be able to get much data on any outage event. 

[cmr]

Which countries are they?  Many countries now have countrywide firewalls that can cause this sort of behaviour.  For example in Egypt you have to get the ISP to open up ports and destinations for anything that looks like a secured tunnel if you want a reliable connection.

[Richard-Tapp]

Thanks all, I will try to answer as many of these questions first.

Case with Meraki was opened and they could not help.

The AP was replaced, MR18 to MR36, same issue. So dont think it is firmware related. Also not sure how we could do if not getting online.

The outages on the portal are prolonged, with only 5 minutes of connectivity every 2 hours or so.

The site is powered off every night, this is a local restriction.

I am sure we checked if there could be government firewalls, sure there were not. Funny our Eygpt sites dont have any issues.

This is a capture for about 5 minutes today. At the end you can see 2 way communication to a Meraki address

864 105 3.824014 10.86.30.10 -> 255.255.255.255 0 BE UDP
866 105 3.825006 10.86.30.10 -> 255.255.255.255 0 BE UDP
867 105 3.825006 10.86.30.10 -> 255.255.255.255 0 BE UDP
4010 105 17.142998 10.86.30.10 -> 255.255.255.255 0 BE UDP
4035 105 17.237039 10.86.30.10 -> 255.255.255.255 0 BE UDP
4036 105 17.239038 10.86.30.10 -> 255.255.255.255 0 BE UDP
4037 105 17.239038 10.86.30.10 -> 255.255.255.255 0 BE UDP
5372 159 22.704025 10.86.30.10 -> 255.255.255.255 0 BE UDP
5373 159 22.705017 10.86.30.10 -> 255.255.255.255 0 BE UDP
5376 159 22.706024 10.86.30.10 -> 255.255.255.255 0 BE UDP
5377 159 22.707016 10.86.30.10 -> 255.255.255.255 0 BE UDP
5378 159 22.709014 10.86.30.10 -> 255.255.255.255 0 BE UDP
5379 159 22.711013 10.86.30.10 -> 255.255.255.255 0 BE UDP
5380 159 22.713012 10.86.30.10 -> 255.255.255.255 0 BE UDP
5381 159 22.714019 10.86.30.10 -> 255.255.255.255 0 BE UDP
5390 159 22.759015 10.86.30.10 -> 255.255.255.255 0 BE UDP
5444 146 23.042997 10.86.30.10 -> 209.206.56.46 0 BE UDP
5445 146 23.042997 10.86.30.10 -> 216.157.140.13 0 BE UDP
5446 154 23.042997 10.86.30.10 -> 209.206.52.7 0 BE UDP
5447 146 23.042997 10.86.30.10 -> 209.206.59.34 0 BE UDP
5483 88 23.245049 209.206.52.7 -> 10.86.30.10 0 BE UDP
5500 105 23.347034 10.86.30.10 -> 255.255.255.255 0 BE UDP
5502 105 23.381028 10.86.30.10 -> 255.255.255.255 0 BE UDP
5503 105 23.382035 10.86.30.10 -> 255.255.255.255 0 BE UDP
5504 105 23.382035 10.86.30.10 -> 255.255.255.255 0 BE UDP
5767 159 24.618031 10.86.30.10 -> 255.255.255.255 0 BE UDP
5768 159 24.618031 10.86.30.10 -> 255.255.255.255 0 BE UDP
5849 159 24.954012 10.86.30.10 -> 255.255.255.255 0 BE UDP
5850 159 24.954012 10.86.30.10 -> 255.255.255.255 0 BE UDP
5851 159 24.954012 10.86.30.10 -> 255.255.255.255 0 BE UDP
5852 159 24.954012 10.86.30.10 -> 255.255.255.255 0 BE UDP
5853 159 24.954012 10.86.30.10 -> 255.255.255.255 0 BE UDP
5854 159 24.955004 10.86.30.10 -> 255.255.255.255 0 BE UDP
5855 159 24.957003 10.86.30.10 -> 255.255.255.255 0 BE UDP
5856 159 24.957003 10.86.30.10 -> 255.255.255.255 0 BE UDP
5857 159 24.957003 10.86.30.10 -> 255.255.255.255 0 BE UDP
5858 159 24.957003 10.86.30.10 -> 255.255.255.255 0 BE UDP
5859 159 24.958010 10.86.30.10 -> 255.255.255.255 0 BE UDP
5860 159 24.958010 10.86.30.10 -> 255.255.255.255 0 BE UDP
6739 159 28.724013 10.86.30.10 -> 255.255.255.255 0 BE UDP
6740 159 28.724013 10.86.30.10 -> 255.255.255.255 0 BE UDP
6741 159 28.726012 10.86.30.10 -> 255.255.255.255 0 BE UDP
6745 159 28.738020 10.86.30.10 -> 255.255.255.255 0 BE UDP
6746 159 28.739012 10.86.30.10 -> 255.255.255.255 0 BE UDP
6747 159 28.740019 10.86.30.10 -> 255.255.255.255 0 BE UDP
6748 159 28.741010 10.86.30.10 -> 255.255.255.255 0 BE UDP
6768 159 28.789012 10.86.30.10 -> 255.255.255.255 0 BE UDP
7196 159 30.472027 10.86.30.10 -> 255.255.255.255 0 BE UDP
7197 159 30.472027 10.86.30.10 -> 255.255.255.255 0 BE UDP
7198 159 30.472027 10.86.30.10 -> 255.255.255.255 0 BE UDP
7199 159 30.472027 10.86.30.10 -> 255.255.255.255 0 BE UDP
7200 159 30.472027 10.86.30.10 -> 255.255.255.255 0 BE UDP
7204 159 30.473034 10.86.30.10 -> 255.255.255.255 0 BE UDP
7205 159 30.473034 10.86.30.10 -> 255.255.255.255 0 BE UDP
7206 159 30.473034 10.86.30.10 -> 255.255.255.255 0 BE UDP
7207 159 30.473034 10.86.30.10 -> 255.255.255.255 0 BE UDP
7208 159 30.474026 10.86.30.10 -> 255.255.255.255 0 BE UDP
7209 159 30.474026 10.86.30.10 -> 255.255.255.255 0 BE UDP
7210 159 30.476025 10.86.30.10 -> 255.255.255.255 0 BE UDP
11189 146 48.042997 10.86.30.10 -> 209.206.56.46 0 BE UDP
11190 146 48.042997 10.86.30.10 -> 216.157.140.13 0 BE UDP
11191 154 48.042997 10.86.30.10 -> 209.206.52.7 0 BE UDP
11192 146 48.042997 10.86.30.10 -> 209.206.59.34 0 BE UDP
11235 88 48.246041 209.206.52.7 -> 10.86.30.10 0 BE UDP

[Purroy]

Please, make sure you are running MR software version 29 or above.  MR30.5 is current stable.

There is a dashboard connectivity method change in these new versions.  With the newer versions Dashboard connectivity uses TCP port 443, prior to this was UDP port 7351 which could improve your connectivity issue.

[Richard-Tapp]

Can we request Meraki to push this version to just the one device ? We have 24 others which it is saying would be done if I do the auto update and I would rather do one first if possible

[Rimccart]

Hi Richard-Tapp, 

It is advised to upgrade all APs for a network at once as it is against best practice to have a mixed firmware environment. The request to test firmware on one device can be made but it is not guaranteed to be accepted.
The complication with running deprecated firmware is that it becomes more difficult to support especially if we are suspecting an issue related to a bug. We additionally would have the option to roll back firmware if we found it to be necessary. 
I would still encourage re-opening your last Support case with Meraki Support to resume troubleshooting the issue and updating to a current stable firmware to help serve as an additional data point that this issue is spanning multiple firmware versions. 

As you clarified the APs are offline for extended periods It would be good to access their local status page and obtain the support data file for support to review.