Cisco Phones not registering from Riyadh, Saudi Arabia !

[ahmadtat]

Hi,

We're facing a very strange issue with multiple organizations, different customers, different MX models, 

starting from Friday 25/5/2018 all customers who have Cisco Call Manager (different versions) which have the Call Manager in a city Not Riyadh, and use Cisco Meraki MX for VPN connectivity suddenly lost the ability for the Cisco IP Phones to register to the call Manager from Riyadh City in Saudi Arabia.

the ISP's are different, different connections methods (fiber/DSL), different MX models , same firmware of MX with other branches that are located in Saudi Arabia (other branches are working perfectly fine).

The only two common things are:

1) all clients (organizations) have Meraki MX to connect their branches together as VPN.

2) this issue is happening only in Riyadh City in Saudi Arabia

(again, different ISPs and different connection methods)

Have anyone faced this issue? can anyone guess what could be the reason?

[PhilipDAth]

Is this all to the same Call Manager?

Is it using the same back haul provider? For example, in my country me had an issue with users web browsing in half of one of our islands last week.  Turns out one of the back haul providers had upgraded a piece of infrastructure, and changed from one vendor to another.

The net result was the MTU was reduced - and this affected every ISP and service provider using this back haul provider.

[ahmadtat]

Actually, no.. it's separate call managers each!
and I contacted the Meraki support and we tried reducing the MTU of the MX's to 1400 and 1300 but no change in the results.

What else should we look for?

That's realy critical as it does not affect customers who has MPLS connectivity ! only customers who has MX with VPN over internet connections.

[PhilipDAth]

Do a packet capture and see what is going wrong.

[ahmadtat]

Here is a packet capture yesterday:

574 May 28, 2018 08:41:00.678529000 PDT 37.237312 1 10.50.2.96 192.168.60.250 TCP 51236 → 5060 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=44284081 TSecr=0 WS=16 78
575 May 28, 2018 08:41:00.715240000 PDT 37.274023 1 192.168.60.250 10.50.2.96 TCP 5060 → 51236 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1192 SACK_PERM=1 TSval=3380312915 TSecr=44284081 WS=128 78
576 May 28, 2018 08:41:00.716054000 PDT 37.274837 1 10.50.2.96 192.168.60.250 TCP 51236 → 5060 [ACK] Seq=1 Ack=1 Win=29200 Len=0 TSval=44284085 TSecr=3380312915 70
577 May 28, 2018 08:41:00.788916000 PDT 37.347699 1 10.50.2.96 192.168.60.250 SIP Request: REGISTER sip:192.168.60.250 (remove 2 bindings) | 925
578 May 28, 2018 08:41:01.019341000 PDT 37.578124 1 10.50.2.96 192.168.60.250 TCP [TCP Retransmission] 51236 → 5060 [PSH, ACK] Seq=1 Ack=1 Win=29200 Len=855 TSval=44284116 TSecr=3380312915 925
579 May 28, 2018 08:41:01.259307000 PDT 37.818090 1 10.50.2.96 192.168.60.250 TCP [TCP Retransmission] 51236 → 5060 [PSH, ACK] Seq=1 Ack=1 Win=29200 Len=855 TSval=44284140 TSecr=3380312915 925
579 May 28, 2018 08:41:01.259307000 PDT 37.818090 1 10.50.2.96 192.168.60.250 TCP [TCP Retransmission] 51236 → 5060 [PSH, ACK] Seq=1 Ack=1 Win=29200 Len=855 TSval=44284140 TSecr=3380312915 925
744 May 28, 2018 19:18:27.803948000 PDT 35.778795 10.50.2.96 192.168.60.250 TCP 49336 → 5060 [RST] Seq=1 Win=0 Len=0 40

where 192.168.60.250 is the CUCM in Jeddah, KSA
and 10.50.2.96 is one IP phone in Riyadh , KSA

what can you get out of this ?

[PhilipDAth]

Have you got it in wireshark format?

[ahmadtat]

this is a link for cap from the CUCM:

https://www.dropbox.com/s/qzbdylai4h16ods/CUCM.zip?dl=0

[PhilipDAth]

Traffic to tcp/5060 is being blocked - probably by a firewall.

The phone sends a SYN.  Call manager sends a SYN ACK back.  The phone should transmit an ACK back - but it never does. This suggests that the phone never got the SYN ACK - or the ACK response was filtered.

My guess is the phone never got the SYN ACK - because something between the packet capture point and the phone blocked it.

From here you could try changing the configuration to not use tcp/5060, change to using a 100% encrypted connection, or do more packet captures to isolate the point where the SYN ACK goes "missing".

[PhilipDAth]

When I Google "Riyadh blocking SIP" I see Saudi Arabia has a habit of blocking SIP traffic.

I would look at converting everything to run over encrypted channels (such as Meraki AutoVPN) - or stop using port 5060.

[ahmadtat]

BTW, from Meraki dashboard, the SIP traffic (5060) seems to reach from Riyadh to HQ in Jeddah as per below ..

am I missing something?

[ahmadtat]

Just wanted to update you that phones started registering now !

that's the weirdest thing.

[PhilipDAth]

I think the Meraki portal is showing the SYN part of the conversation - but the traffic capture definitely shows tcp/5060 was not completing its setup.  Something was preventing the TCP connection from forming (probably a firewall).

[Zia]

Hi Ahmadtat

We are experiencing the same issue with STC 5G internet and meraki at branches used to form auto vpn tunnel with main site meraki, where cisco SIP phones are not registering. Whereas if we use Mobily or Zain 5G the phones get registered. Could you please detail the resolution in your case what was causing the block.

Thanks