Cisco 8811 Phones are not registering with Call Manager at remote branch office

[earl_m]

Hello,

I am new here to the Meraki forums. We are having issues with Cisco 8811 phones reaching our Call Manager at a remote branch office. More specifically, it is getting a "udp port 69 unreachable" when trying to contact our Voice Gateway that is housed at a District office in Jacksonville. Because of this, the phones cannot download their firmware/configuration from the TFTP/Voice Gateway server. We have a Meraki MX68 VPN appliance at a remote branch office in Ponte Vedra Beach, FL. We also have a MS225-48LP and a MS225-24P switch that are plugged into the MX68 appliance. The MX68 appliance creates a LAN-2-LAN VPN tunnel to our Distrcit office in Jacksonville, where there it terminates with a router then to a MX250 appliance where it enters our "intranet" network. The MX250 appliance is in the Jacksonville office and so is our Voice Gateway that we are using for this VoIP phone system. The Voice Gateway is actually a Cisco 2911 router hosting voice gateway services. On the MX68 appliance at the Ponte Vedra Beach branch office, I have Option 150 set in our DHCP server pointing to the Voice Gateway, but I have also tried pointing that Voice VLAN with Option 150 to our Call Manager as well. I was told that using the Voice Gateway for option 150 is what it should be set at, assuming this is where the TFTP server is hosted on. Moreover, the phones can be pinged from the Voice Gateway in Jacksonville and the MX68 appliance can ping our Call Managers in Tallahassee. The phones are getting IP addresses handed from our local DHCP server that we have running in the MX68 appliance; however, we are not seeing these phones register with our Call Manager. Provided below are some packet capture examples that I have got. Any help or advice would be greatly appreciated.

16:29:21.777557 IP (tos 0xb8, ttl 253, id 34922, offset 0, flags [none], proto ICMP (1), length 56)
172.20.203.2 > 172.23.14.80: ICMP 172.20.203.2 udp port 69 unreachable, length 36
IP (tos 0xb8, ttl 61, id 41516, offset 0, flags [none], proto UDP (17), length 59)
172.23.14.80.50944 > 172.20.203.2.69: 31 [|tftp]
16:29:25.730359 IP (tos 0x60, ttl 64, id 41564, offset 0, flags [none], proto UDP (17), length 59)
172.23.14.80.50944 > 172.20.203.2.69: [udp sum ok] 31 RRQ "CTLSEPCC7F75055883.tlv" octet
16:29:25.781978 IP (tos 0xb8, ttl 253, id 34926, offset 0, flags [none], proto ICMP (1), length 56)
172.20.203.2 > 172.23.14.80: ICMP 172.20.203.2 udp port 69 unreachable, length 36
IP (tos 0xb8, ttl 61, id 41564, offset 0, flags [none], proto UDP (17), length 59)
172.23.14.80.50944 > 172.20.203.2.69: 31 [|tftp]
16:29:31.015116 IP (tos 0x0, ttl 64, id 26706, offset 0, flags [none], proto TCP (6), length 60)
172.23.14.80.51348 > 172.20.203.2.6970: Flags [S], cksum 0xa13b (correct), seq 1873281886, win 29200, options [mss 1460,sackOK,TS val 6859770 ecr 0,nop,wscale 5], length 0
16:29:31.079141 IP (tos 0xb8, ttl 253, id 36511, offset 0, flags [none], proto TCP (6), length 40)
172.20.203.2.6970 > 172.23.14.80.51348: Flags [R.], cksum 0x277c (correct), seq 0, ack 1873281887, win 0, length 0
16:29:31.621927 IP (tos 0x60, ttl 64, id 41634, offset 0, flags [none], proto UDP (17), length 59)
172.23.14.80.49188 > 172.20.203.2.69: [udp sum ok] 31 RRQ "ITLSEPCC7F75055883.tlv" octet
16:29:31.672766 IP (tos 0xb8, ttl 253, id 35345, offset 0, flags [none], proto ICMP (1), length 56)
172.20.203.2 > 172.23.14.80: ICMP 172.20.203.2 udp port 69 unreachable, length 36
IP (tos 0xb8, ttl 61, id 41634, offset 0, flags [none], proto UDP (17), length 59)
172.23.14.80.49188 > 172.20.203.2.69: 31 [|tftp]
16:29:32.629043 IP (tos 0x60, ttl 64, id 41638, offset 0, flags [none], proto UDP (17), length 59)
172.23.14.80.49188 > 172.20.203.2.69: [udp sum ok] 31 RRQ "ITLSEPCC7F75055883.tlv" octet
16:29:32.679123 IP (tos 0xb8, ttl 253, id 35547, offset 0, flags [none], proto ICMP (1), length 56)
172.20.203.2 > 172.23.14.80: ICMP 172.20.203.2 udp port 69 unreachable, length 36
IP (tos 0xb8, ttl 61, id 41638, offset 0, flags [none], proto UDP (17), length 59)
172.23.14.80.49188 > 172.20.203.2.69: 31 [|tftp]
16:29:34.631282 IP (tos 0x60, ttl 64, id 41785, offset 0, flags [none], proto UDP (17), length 59)
172.23.14.80.49188 > 172.20.203.2.69: [udp sum ok] 31 RRQ "ITLSEPCC7F75055883.tlv" octet
16:29:34.682712 IP (tos 0xb8, ttl 253, id 35946, offset 0, flags [none], proto ICMP (1), length 56)
172.20.203.2 > 172.23.14.80: ICMP 172.20.203.2 udp port 69 unreachable, length 36
IP (tos 0xb8, ttl 61, id 41785, offset 0, flags [none], proto UDP (17), length 59)
172.23.14.80.49188 > 172.20.203.2.69: 31 [|tftp]
16:29:37.455711 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 172.23.14.80 tell 172.23.14.65, length 42
16:29:37.456270 ARP, Ethernet (len 6), IPv4 (len 4), Reply 172.23.14.80 is-at cc:7f:75:05:58:83, length 42
16:29:38.631855 IP (tos 0x60, ttl 64, id 41988, offset 0, flags [none], proto UDP (17), length 59)
172.23.14.80.49188 > 172.20.203.2.69: [udp sum ok] 31 RRQ "ITLSEPCC7F75055883.tlv" octet
16:29:38.682398 IP (tos 0xb8, ttl 253, id 35983, offset 0, flags [none], proto ICMP (1), length 56)
172.20.203.2 > 172.23.14.80: ICMP 172.20.203.2 udp port 69 unreachable, length 36
IP (tos 0xb8, ttl 61, id 41988, offset 0, flags [none], proto UDP (17), length 59)
172.23.14.80.49188 > 172.20.203.2.69: 31 [|tftp]

Thank you.

[cmr]

Are you using IPSEC VPN from the MX68 to the DC, or Meraki Auto-VPN?  If not could you change to use Auto-VPN?

We have 8 sites with Auto-VPN and Cisco 88xx handsets talking to a central CUCM server.  We also have three home users with Z3/small MX that have 88xx handsets too.

[cmr]

The only custom DHCP options we have are:

The first is our internal domain, the second is the CUCM TFTP servers. 

[earl_m]

I am using IPSec VPN tunnels. The MX68 is a spoke and the MX250 is a hub. The Voice Gateway is on the side of the MX250 hub. The DHCP Option 150 is pointing to the JAX Voice Gateway (172.20.203.2) that you see there in that packet capture. We have several Cisco IP phones throughout the state that actually traverse an MPLS network to reach off-site remote voice gateways, and they register just fine. This IPSec VPN "Hub and Spoke" topology is the first one we have tried using.

[DarrenOC]

Jumping in late here but why is your option 150 pointing at your VG?  This should be pointing at your CUCMs.

Do your phones pick up an IP address?  If they do is the tftp server they’re picking up that of the CUCMs?

From your MX do you have IP reachability to your CUCM cluster?

[earl_m]

I have tried pointing Option 150 to the VG and the CUCM cluster, neither of which have worked. I wasn't sure which one it should be pointing to. The phones do pick up an IP address from the local voice subnet that I have on the MX68 because the DHCP server is hosted on the MX68. I have been told by my co-workers that the TFTP server should be on the VG and not on the CUCM servers. The MX does have IP connectivity to the CUCM cluster and VG.

[DarrenOC]

Hi @earl_m  if the tftp server is on your VG I assume all the required phone firmwares are located in the device flash?

Is your Vg configured correctly to serve those files and permit the new voice vlan subnet from the spoke site?

[Bruce]

@earl_m, like @DarrenOC said, I’d be very surprised if your TFTP server was running on the VG - it’s a lot of effort to setup and maintain - although it’s not impossible. However, the VG is responding with ICMP that UDP port 69 is up reachable, which suggests to me that the TFTP server isn’t there.

it’s possible there are other servers in the CUCM cluster running the TFTP service, which may not be the call processing servers. Do you have access to a working phone that you can look the TFTP details up on? Or a subnet that already has the correct DHCP options configured?

[Greenberet]

CCIE Collaboration here.

as @Bruce & @DarrenOC already mentioned. The TFTP has to be set to the CUCM servers which are having the tftp service enabled. The quickiest way would be to check the configuration of a working phone.

The only reason to set it to the VoiceGW is, when you are running a callmanger express (NOT SRST) there.

Please also check your firewall settings. The tftp protocol is not really used anymore. The phones will get the correct servers via DHCP option 150 and then they will download the configuration & firmware files over HTTP/S over the Ports 6970/6972. If these are blocked, then the phones won't be able to download the correct settings

[earl_m]

Hi Greenberet, that is good to know. We have several VoIP phone systems within our enterprise (across the state) that have their Option 150 set to the nearest Voice Gateway, and it has always been successful. We do not have ports 69/6970/6972 filtered in our central firewall. Now that I have the Option 150 in the Voice VLAN set to our call manager cluster, I am still not seeing the phones register with Call Manager. Perhaps I should reboot the MX68 appliance for changes to take effect?

[Greenberet]

a reboot of the MX shouldn't be needed.

I have an MX68W in my home office and I'm using a non meraki site2site VPN without any problems. My hardphone gets registered without any issues.

The CUCM is probably configured via FQDN. What are the DNS settings of the subnet? Do they point to the correct DNS servers to resolve the FQDN?

I would check the status messages on the ip phone (Settings -> Admin Settings -> Status -> Status messages)
You should see the issue there.

It could also be some kind of trust list issue, if the phones were registered on a different cucm in the past.

[earl_m]

Hi Greenberet. We only use 2 DNS server addresses throughout our enterprise, so I don't think it's anything DNS related. The Voice VLAN has the same DNS server settings as every other phone in our enterprise uses. I will get with someone at the branch office to read me the status messages on the phones. Thank you.

[earl_m]

Okay so I got onsite personnel to send me the phone status messages we are seeing. It is getting a DNS timeout to our Call Managers. The DNS server information assigned to the Voice VLAN is correct. We only use 2 DNS servers throughout the whole enterprise and have never had any issues with DNS with our phone system. I am stumped here.

[cmr]

As your servers are set by name, have you set option 15 for domain suffix?

[earl_m]

Hi cmr, I have tried that and still no luck.

[Greenberet]

I think I found the problem now. (re read your status messages)

You have a timeout with the cucm hostnames.

There should be two options two fix this.

Either set the dns search domain on the DNS server or specify the FQDN on the CUCM Nodes (CUCM Admin: System -> Server)

See https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-cal...for further details.

I would go for the second option. Since version 10.X it's the prefered way to use the FQDN for the servers as it is required for jabber, SSO, ...

[Greenberet]

I would check the different firewalls (don't forget the firewall on the dns server itself, if there is one), if there is really nothing blocking.

You can also try to traceroute from the DNS server to the ip phone. Maybe you are missing a route somewhere in the setup

[earl_m]

Hi Greenberet, I put a PC on the voice VLAN at the site and was able to reach our DNS servers so IP communication looks good. I do not believe there is a firewall on the DNS servers, but I will find out. I do not have access into our DNS servers, but I can try and get someone in our Systems group to try that. It's looking more like it is something within our Call Managers that is missing.

[earl_m]

Hi Bruce, my apologies for the delay in response. I have now set the Option 150 to our CUCM cluster, and still the phones are not pulling their config/firmware. I asked another co-worker within our department to check the Call Manager settings and he can't seem to find anything out of the ordinary. The 7942 model phone in my office has it's TFTP set to Call Manager 2. Of course, that is within our HQ office. This phone system over a Site-to-Site IPSec configuration (remote branch office) is the first of it's kind that we have within our enterprise.

[RPidcock]

Wondering if anyone ever found resolution to this topic.  I have a very similar scenario.  I'm setting up a remote branch via and Meraki MX67C.  Using Meraki AutoVPN and tunnel establishment is good.  I have confirmed bidirectional communication between CUCM nodes and the phone device on the remote side.  I'm totally stumped at this point.  I've engaged with Cisco TAC and Meraki support but no luck.  Any help or thoughts/ideas appreciated.

[OVERKILL]

As long as you have your option 150 set properly in the DHCP server at the remote site, it should just "work", at least mine do. 

[cmr]

Same here, option 150 is all we set on the remote MXs, as long as the devices can see the CUCM servers.

[RPidcock]

Man I wish that was the case.  This thing is killing me.  I've grabbed pcaps off of both ends of the connection and can confirm the bidirectional communication.  I'm running DHCP on my remote MX67 and have option 150 set with the IP of one of my subscriber nodes.  I see the expected DHCP settings on the phone.  Cisco TAC has suggested a network issue due to missing a keepalive timeout.    Are you using a wired or cellular connection on the far end (remote side device).

[OVERKILL]

A number of different connection types, one is wireless (cellular) but was previously on wireless P2P, others are cable and DSL. 

My 150 option is set to the tftp server I have running at the HO in this system which has the handset configs and phone firmware loads on it. 

One thing you might want to check is to ensure that there's no firewall rules blocking access from the remote subnets to your TFTP server. 

[cmr]

Our option150 is set to publisher (primary tftp) and subscriber (secondary tftp)