Allen-Bradley PLC's not available over Client VPN

[GraphiteJess]

Hello all, 

I'm a technician for an MSP provider, and we recently inherited a client with all Meraki gear. Their former MSP handed over control of the network, and by and large, it has been bulletproof. However, the client's industrial automation guy installed some new PLC's for their microwave controllers, and we're unable to work with them over the VPN. When you're on the default VLAN, in the building, there are no issues. As soon as you use a remote connection and are connected via the VPN, nothing. No web interfaces, lost pings, etc. I've even gone as far as opening up huge holes in the firewall. 

I've just settled in to review Cisco's CIP information, but this all seems relegated to their switches, rather than any Meraki gear, so I worry that these devices aren't ever going to work correctly on the network. 

I would appreciate any info about Meraki and Common Industrial Protocol, or Allen-Bradley/Rockweel PLC's in particular. Given my google results on these items, it feels like a long shot. 

[Nash]

You're on a different subnet when you're connected in via the client VPN.

Is that subnet allowed to talk to your PLCs?

Do your PLCs have an actual gateway set on them?

[GraphiteJess]

It is. I have no problems hitting the rest of the resources on the default VLAN. The gateway on the PLC is set to the IP of the MX on the Default VLAN. I've expressly permitted all traffic between the VLANS via the Firewall interface, but I can't set static routes on either, as it appears to not be an option; the MX complains about overlap. 

[SoCalRacer]

When connected to the client VPN can you ping the MX and other devices on the default VLAN?

[GraphiteJess]

Yes. In desperation I connected to a local printer and sent a help note over the VPN; printed no problem. 

Based on what I'm seeing from Wireshark, it looks like the PLCs want to run things layer 2 with ARP requests. When I attempt to ping the PLC, the MX ARPs for the PLC IP, and the PLC arps a response. When the PLC ARPs for my computer's IP on the VPN, it does NOT get a response from the MX. 

[Nash]

Right, the MX will automatically route between the client VPN and other subnets it knows how to get to.

Is your PC at the office on the same vlan and same subnet as the PLC, or is it on different?

[GraphiteJess]

When the PLC programmer is in the office, he IS on the same VLAN, and can do what he needs to do. When he's remote and on the VPN, he can't. 

[Nash]

Dang. So definitely works within the same broadcast domain, but breaks when routed.

So you're sure the PLCs have the correct gateway and subnet mask, and valid IP addresses? How are IP addresses assigned to them?

[SoCalRacer]

Printer and PLC are on the same VLAN?

[SoCalRacer]

Most of the time these PLC devices are setup static or at least that is how the setup instructions usually go. I bet the automation guys set it up and didn't know exactly what to put in the device, and went with best guess

[GraphiteJess]

We've been over it a few times, though I'm not sure the mask is set correctly; they have reserved IP's if they're getting it via DHCP, though I'm pretty sure we configured them manually. It sucks because they have to be restarted when IP configs change, and they're currently running normal operations, so I can't just reboot them without causing issues with the business. 

[jdsilva]

---

@GraphiteJess wrote:

We've been over it a few times, though I'm not sure the mask is set correctly; they have reserved IP's if they're getting it via DHCP, though I'm pretty sure we configured them manually.

---

Are you able to compare what is configured on the PLC vs what is set up on the VLAN on the MX?

[GraphiteJess]

Yup. In addition to a number of security devices, other PC's, etc, and I have no issue hitting any of them. 

[SoCalRacer]

Might take a look at your Route table.

Not that it is the cause, but is Dynamic ARP Inspection on?

https://documentation.meraki.com/MS/Other_Topics/Dynamic_ARP_Inspection

While on a PC or device on the default VLAN can you ping the Client VPN device?

[GraphiteJess]

Dynamic ARP appears to be a switch setting, not the MX, and the PLC is plugged directly into the MX.

I can ping the Client VPN machine from a computer on the default VLAN.

[jdsilva]

---

@GraphiteJess wrote:

When the PLC ARPs for my computer's IP on the VPN, it does NOT get a response from the MX. 

---

Nor should it, unless Proxy ARP is enabled, which I'm not sure the MX supports. 

So what this sounds like to me is that the PLC either doesn't respect the default gateway setting or the subnet mask is configured incorrectly. The only way the PLC _should_ ARP for a device not on the subnet is if it doesn't have a default gateway (solved by Proxy ARP) or if it thinks the IP is on its local subnet when it's really not (incorrect mask). 

If the mask is correct and the PLC refuses to use its gateway to get to another subnet then it could be worth a try calling support to see if they can enable proxy ARP.

[SoCalRacer]

I would take a look at the PLC (on-site) and verify the settings (gateway/mask/etc.) yourself, not that you don't trust the automation guy, but best to see if there is anything that stands out to you. I would say if you can schedule that after production hours that is best in case you want to make a change and need a rebeoot. If not then go during production hours and just document all the settings/options. Then I would start reaching out to the vendor's support to see if this is common in their product line.

[GraphiteJess]

I just got a restart on the PLC, and it getting the wrong mask from the DHCP server. The VLAN is supposed to be 10.24.0.0/23, but the mask it's getting from DHCP is 255.255.252.0, when it should be 255.255.254.0, right? 

[SoCalRacer]

Correct 23 bit is 255.255.254.0

[GraphiteJess]

Whelp, back to static and waiting for another restart. 

Why is the MX giving out the wrong mask? If this fixes the problem, imma be pissy about it. 

[SoCalRacer]

This makes it seem like the 24 bit mask is hard coded into the PLC.

http://forums.mrplc.com/index.php?/topic/18124-clx-ethernetip-subnet-mask/

[GraphiteJess]

You'd think so, but it was at 14 before, because whoever intiially set up this network decided that fourteen computers and half a dozen phones needed 64k IP addresses. It is possible it can only do some mask bits? That seems pretty.... odd. 

[Nash]

---

@GraphiteJess wrote:

You'd think so, but it was at 14 before, because whoever intiially set up this network decided that fourteen computers and half a dozen phones needed 64k IP addresses. It is possible it can only do some mask bits? That seems pretty.... odd. 

---

Like fourteen fourteen? I could see a manufacturer restricting to 8/16/24. But if they allow fourteen, I'd find it strange if they didn't allow something like a 23 or whatever.

[GraphiteJess]

Well, it didn't work before, either. If this all fails, I might try rolling a plain old 24 and see if it plays nice. 

Cisco support just asked for the same pcap I just sent them, so... not holding my breath there. 

[Nash]

Honestly, if these PLCs aren't already on their own vlan, segregating them is better hygiene.

Then you can create a subnet that's fully supported by their network requirements, and add an interface to your router/L3 switch/whatever.

[SoCalRacer]

That would be my recommendation is create a 24 bit masked VLAN that is just PLC stuff. Honestly I have seen these kind of devices be on DHCP/Auto and then have a manual mask or gw defined. I believe this is why almost all PLC vendors request to have their equipment set as static and not DHCP reservation.

[PhilipDAth]

>it looks like the PLCs want to run things layer 2 with ARP requests.

This is typical when the subnet mask is configured wrong.

[Priesty]

Hi, we used to see similar behaviour with other non-Cisco meraki equipment and PLCs, almost everytime it was the result of the VLANs not traversing over the network.

Is the industrial network segmented from the admin network? Are you coming in on the admin network side and is there routes available to the industrial network?

If so, are the switches, firewall or something else blocking the VPN client protocol? Are there VLANs in place? can these VLANs route between the admin network and industrial network?

Essentially the PLCs should be on their own network assigned via VLAN, but untagged as in connected to an access port on the switch, work your way back from there. Is the client VPN subnet on a range that is conflicting? Do the PLCs have the correct IP address information including their gateway addresses. 

Try working backwards from the PLC, or find out if there are any restrictions on the client VPN - 

to me, it sounds like a route issue, and perhaps your microwaves do not have the ability to route VLANs across them, in which case you will need a L3 router on each end.