MX95 TLS “Client Hello” Failure

[DaleHuitt]

We have just deployed a pair of MX95's in a failover configuration with firmware MX 17.10.2.  When connecting to an Internet web site, browsers are displaying "establishing secure connection" or "Performing TLS handshake".  Web sites can take 30 seconds to 2 minutes to load.  The only thing that changed is moving our Internet connection that was working from a Cisco ASA to the Meraki MX95 and the default route from the ASA to the MX95.  This is with Threat Protection disabled and Content Filtering disabled.  There is no Active Directory or other authentication.

A Wireshark packet capture shows the client sending the TLS "Client Hello" packet and then no response is coming back from the server.  The client then resets the connection and tries again.  After multiple resets the TLS negotiation is successful.  This happens on every client in our environment.

Again, this was working without issue prior to the change.  The only thing that changed was replacing the Cisco ASA with a Meraki MX95.  Right now, the Internet is nearly unusable for our enterprise due to this issue.  Please help.

[RaphaelL]

I would suggest opening a case with Meraki first. 

You should probably try to downgrade to MX16.16.X 

Many users are reporting issues with web browsing with the MX 17 code.

[DaleHuitt]

Thank you.  I have already done that and waiting on a response from support.  I posted the question since so many people have had the same exact issue I hoped someone would have a resolution.  Downgrading the firmware isn't an available option.  The MX95's came with v17 firmware.  Firmware v16.9 is the minimum firmware that can be run on the appliance.

[NFL0NR]

it might be YOUR only option, but Meraki might be able to downgrade it further on their end.

[alemabrahao]

I agree with @RaphaelL.

[DaleHuitt · Accepted solution]

After much packet capture analysis and research, I have finally found the issue.  The issue is with Meraki and Client Tracking implementation.  A firmware downgrade does not resolve this issue and isn't much of an option.  Meraki, by default, tracks clients based on MAC address.  This function breaks TCP traffic outbound, and some inside, if you have a non-Meraki Layer 3 device behind the Meraki MX95.  In order to use the Meraki MX95 with non-Meraki Layer 3 devices, you must section those off in the dashboard in separate networks.  They cannot be combined in the same network.  One network for Meraki MX95 and any other Meraki devices using MAC Address client tracking or the beta Unique Client Identifier.  Then, non-Meraki Layer 3 devices must be placed in their own separate network using IP Address client tracking.  Then another network for non-Meraki Layer 2 devices.  Non-Meraki WAP;s should also go in their own network.

So, the bottom line is that the Meraki client tracking breaks TCP traffic if you use MAC Address Client Tracking and have non-Meraki Layer 3 devices behind that Meraki MX.  A firmware downgrade as everyone suggests, will not fix this issue and problem.  It only introduces vulnerabilities in to your network security.

[alemabrahao]

Ok, the L3 switch was not mentioned. But if it was solved is the most important.

[DaleHuitt]

Yep.  A Meraki Layer 3 device isn't an issue.  But a non-Meraki Layer 3 device downstream from the MX95 and using the default Client Tracking of MAC Address breaks everything.  If you are going to use non-Meraki Layer 3 devices in your network, they have to be in their own separate network and use IP Address.  Even though my entire environment is Cisco, it isn't compatible with Cisco Meraki.  Non-Meraki Layer 3 devices have to be in their own Network with Client Tracking set to IP Address.  Meraki devices go in their own separate Network with Client Tracking set to MAC Address.

[CptnCrnch]

I believe you're the first person in this forum experiencing something like this...

[DaleHuitt]

No.  The Forum is full of several people experiencing the same issue with other MX versions and no resolution has been provided.  Just do a quick search for TLS “Client Hello” Failure.  This is WELL known issue with Meraki MX95 security appliances.  The Cisco TAC engineers immediately said this was the issue before we even started doing PCAP's and other investigation of the issue.  This is a well known issue with Cisco Meraki support.  You cannot mix non-Meraki Layer 3 devices with MX security appliances in the same Network configuration while using MAC Address Client Tracking.

[RaphaelL]

Isn't the Client-Tracking Options only for reporting usage ? I don't see how it impacts flows. Is this documented somewhere ?

[DaleHuitt]

Nope.  May have been in previous firmware releases.  But, now it is a core component and how Meraki moves TCP packets.  Another component of what it does id report usage, but it has gone far beyond that.  Not sure why Meraki chose to use this method of data packets.  I spent an entire afternoon on a call with an engineer from Cisco TAC and that was the first thing he said.  Issue was related to Client Tracking using MAC addresses and non-Meraki Layer 3 devices behind it.  Layer 2 non-Meraki devices aren't an issue.  Just Layer 3.

If you go in to the dashboard an look under Security & SD-WAN -> Addressing & VLANs and look at the heading for Client Tracking.  It shows the default is by MAC Address.  If you read through the description, there is a hint that this might cause issues with non-Meraki Layer 3 devices.  The last sentence says, "Clients behind a layer 3 routing device downstream from this security appliance will not be identified."  So, usage of those clients won't be able to be captured, but it also breaks TCP.

[CptnCrnch]

Could you provide us with the PCAPs that let you conclude that tracking breaks TCP? 

[DaleHuitt]

Unfortunately, I no longer have them and recreating that environment to do another PCAP is not an option.  I do have a screenshot from that time period.  Also, this wasn't a "conclusion" hunch or guess on my part.  When I saw the behavior that was happening with TCP, I started a support call with Cisco TAC.  The engineer who assisted said that what was happening was being caused by Client Tracking.

Again, it isn't Client Tracking specifically.  It is the Meraki implementation while using MAC Address for Client Tracking and having non-Meraki Later 3 devices behind the MX95.  If you are going to use the default Meraki Client Tracking of MAC Address and you have non-Meraki Layer 3 devices, you MUST use sperate Networks.  Meraki MX security appliances cannot be used in the same Network configuration as non-Meraki Layer 3 devices while using MAC Address Client Tracking.

[James_NFX]

Thanks for sharing this, we encountered this issue last week and came across your post. We had been holding back the firmware for an MX95 also for some time on the 16.6.6 build as the 17.x firmware and 18.x firmware right up to the latest 18.107.2 build was causing issues. Web Dev testing the issue was running 'time curl -v <webpage>' would either timeout or take 20-30 seconds on the newer firmware, rollback to 16.6.6 and instantly load. Changed the the client tracking mode from MAC address to IP address per your post and the issue was resolved.