MX HTTPS Inspection Coming ...

[PhilipDAth]

Starting with 15.11 a closed beta of HTTPS inspection has been released.  Their is now public documentation about this feature.

https://documentation.meraki.com/MX/Content_Filtering_and_Threat_Protection/HTTPS_Inspection

[BrechtSchamp]

Interesting! Thanks for the share Philip.

Have you heard anything about the performance impact of enabling it?

[PhilipDAth]

No I don't know the performance impact.  I'm trying to get myself onto the beta program.  Being a close beta I'm not sure, but I asked if I could talk about this in public and was told yes and the documentation is publically posted.

I'm generally not a fan of SSL inspection - because it is a lot of work to deploy and breaks things.

You need to load a certificate onto the MX (or any device that does TLS inspection), and then load that certificate as a trusted root certificate onto every device sending traffic via that MX.  For some mobile devices that is a real pig of a job.

You end up creating more VLANs, so you can inspect some types of devices and avoid others because they are too much work - or you create huge bypass/whitelist rules.

Then you get sites (like www.google.com) that specifically check that the certificate returned is Google's, and will report a security problem.  So you end up telling the browser to ignore this, and the user has worse protection when they are out of your office, or you start whitelisting things all over the place.

Then you get issues with new versions of server software offering newer encryption and TLS versions faster than the SSL inspection engine is updated, breaking things.  For example, this TLS inspection feature only supports TLSv1.2.  So if you went to a web site that only offered TLSv1.3 (not likely at this stage) it would break.

With Cisco Firepower you can say only TLS inspect sites with a "rank" below a certain value.  Then it ignores high ranking sites like Google, Office 365, etc - and only pays attention to low ranked sites (far more likely to be used for malware).  This also relieves a lot of load off the device, as the bulk of your traffic tends to be to high ranking sites (the very defination of high rank).

[BrandonS]

---

@PhilipDAth wrote:

No I don't know the performance impact.  I'm trying to get myself onto the beta program.  Being a close beta I'm not sure, but I asked if I could talk about this in public and was told yes and the documentation is publically posted.

I'm generally not a fan of SSL inspection - because it is a lot of work to deploy and breaks things.

You need to load a certificate onto the MX (or any device that does TLS inspection), and then load that certificate as a trusted root certificate onto every device sending traffic via that MX.  For some mobile devices that is a real pig of a job.

You end up creating more VLANs, so you can inspect some types of devices and avoid others because they are too much work - or you create huge bypass/whitelist rules.

Then you get sites (like www.google.com) that specifically check that the certificate returned is Google's, and will report a security problem.  So you end up telling the browser to ignore this, and the user has worse protection when they are out of your office, or you start whitelisting things all over the place.

Then you get issues with new versions of server software offering newer encryption and TLS versions faster than the SSL inspection engine is updated, breaking things.  For example, this TLS inspection feature only supports TLSv1.2.  So if you went to a web site that only offered TLSv1.3 (not likely at this stage) it would break.

With Cisco Firepower you can say only TLS inspect sites with a "rank" below a certain value.  Then it ignores high ranking sites like Google, Office 365, etc - and only pays attention to low ranked sites (far more likely to be used for malware).  This also relieves a lot of load off the device, as the bulk of your traffic tends to be to high ranking sites (the very defination of high rank).

---

That's a good breakdown, @PhilipDAth As a reseller, I just look forward to it so it checks the box as something competitors currently do, but Meraki does not..

[Adam2104]

doc says 85-90% throughput drop, yikes, no thanks.

[MarcP]

---

@Adam2104 wrote:

doc says 85-90% throughput drop, yikes, no thanks.

---

OMG...

Throughput

The additional overhead of decrypting and inspecting client traffic significantly reduces the security appliance’s throughput capabilities. A reduction of 85-90% vs stateful firewall throughput spec may be seen. For example, an MX250 capable of 4 Gbps stateful firewall throughput may achieve 600 Mbps with HTTPS inspection enabled.

[jdsilva]

Endpoint security, people! The network is the wrong place to do this.

[peto]

Has anybody tested it with MX67? thank you

[WillSparing]

That throughput...

I'm just going to nope out of this... ty.

[RYN0]

The Doc does note that you can whitelist via L3 and L7 rules to exempt them from inspections. Though what you say Firepower can do is smarter though.

[KrisWithaK]

There is no question that SSL Inspection is both incredibly resource intensive and that its implementation comes with a painful initial transition and tweaking period where you work out exceptions and such. With that said, when was an important network security function implemented that wasn't a complete pain to put into place? 

The bottom line is that if you are not inspecting SSL traffic, and it contains malware, that malware will make it to your environment.  Then you have to ask yourself if you have invested in an endpoint AV strategy that has a high enough success rate to keep you from allowing a big malware problem into your network.  Are you willing to let your users see whatever embedded videos they want to see on their favorite sites?  Without full inspection, that's what you're going to allow unless you're running a straight whitelisting strategy.

As far as mobile devices go, if you're running those on the same VLAN as your desktop and laptop environment, you've got bigger problems.  The idea here is not to run deep packet inspection for every bit of your traffic, but just the devices that carry your critical information (desktops, laptops, servers, etc.).

In summary, no one is a fan of implementing deep packet inspection, but sticking your head into the mud won't keep your business from suffering from the lack of a proper security strategy for your entire attack surface.

[PhilipDAth]

>With that said, when was an important network security function implemented that wasn't a complete pain to put into place? 

Cisco Umbrella is usually trivial and easy to implement.  It uses DNS to block known bad domains, and use a transparent proxy to scan the content of "grey" domains.

https://umbrella.cisco.com/products/features/intelligent-proxy

I've learnt over many years that hard difficult to implement security technology usually lets you down due to human failures.  Simplicity is king.  I've seen this a million times - "x" isn't working - can you bypass "x" from the security.

>Are you willing to let your users see whatever embedded videos they want to see on their favorite sites?

We use content filtering only to block known malware and general "bad" sites.  We don't have any other filters enabled such as blocking porn or embedded videos at work.  Everyone is expected to be a professional and strive for professional standards.  Everyone also knows we have detailed analytics enabled in the Meraki Dashboard.  It would stand out.

We've never had a problem.  To be honest, it would probably be more of an embarassment for someone at my work being caught out like this and an disiciplinary issue.

>if you have invested in an endpoint AV strategy that has a high enough success rate to keep you from allowing a big malware problem into your network

I'm feeling pretty confident in this area to be honest.  I wouldn't use an AV platform unless I was confident.  Nearly everyone at my work uses a notebook - because they are expected to be mobile workers.  Having TLS inspection at just the office on its own doesn't adequatly address the overall risk.

It's probably one of the reasons we are not so worried about content filtering at work.  If someone wanted to use something dubious they could just do it when they were working at a clients office, at home, while on 4G (all our notebooks havew 4G built in and enabled), or in a cafe (where I am right now).

Our AV platform also has similar security content filtering categories as configured in our Meraki network.  Users get a similar experience everywhere they are working.

We also have our systems patched right up to date.  All our platforms use a supported and modern OS.

>Without full inspection, that's what you're going to allow unless you're running a straight whitelisting strategy.

That simply is not true.  The initial HTTPS handshake includes the DNS name you are trying to connect to in clear text.  This is how content filtering can block access to categories, even though they are using HTTPS.

>In summary, no one is a fan of implementing deep packet inspection, but sticking your head into the mud won't keep your business from suffering from the lack of a proper security strategy for your entire attack surface.

Touche.  I believe continuing to push the line of using deep packet inspection for TLS, a technology from the 00's, is an "old school" way of thinking.  Security strategy has moved beyond this technology using methods I have already given - security monitoring, endpoint protection, DNS inspection (such as using Umbrella and MX), end user education, using modern supported OSs and regular patching of known security issues.

If you are a bit stuck - you can use the last sentence above for your new security policy.  I promise you - you'll be much safer with this than using TLS inspection.  🙂

[SteveMcKee]

Great discussion - SSL inspect is getting more difficult. TLS1.3 introduces some additional challenges and having the client downgrade to TLS1.2 for the MIM/Inspection Device which will then negotiate 1.3 to the target service seems to be the workaround thinking. Major issue is that for any of this to work, you have to have every endpoint under management in order to deploy (and have trust) the MIM/Inspection Device certificate. This, in itself, becomes more challenging when we allow BYOD or IoT devices (I use the term IoT as a catch all that would be non-PC/User Device related........even printers, network enabled displays, etc).

A layered defense model is as valid today as it has ever been. Endpoint, Network & Cloud as macro categories to look at layers of protection seem reasonable and, then, to have orchestration, threat management across the macros would also seem reasonable. This you'll see from Cisco around things like Cisco Threat Response (CTR) top level with services such as Umbrella and AMP for Endpoints feeding in to it (amongst others - such as Cloud Email Security (CES), another entry point for sneaky malware :)).

Umbrella in combination with AMP for Endpoints (AMP4E) is a great Endpoint Protection combination. Umbrella Roaming for those DNS/Intelligent Proxy protections when off-net and AMP4E to protect against everything else (and with CTR to give you a fighting chance to respond to the gnarly stuff it (probably when) it ever happens). For Umbrella Intelligent Proxy, here SSL (if endpoint is managed) can be inspected but is done selectively and based on Umbrella ranking of sites (not configurable as to what is and isn't inspected, I believe).

Encrypted Traffic Analytics is another great area to look in to. Here, the idea is to correlate a number of elements/attributes of network traffic and to provide a high-degree of probability score against malware (encrypted) without having to decrypt it. It might be worth looking in to upcoming ETA support in Stealthwatch Cloud (great add-one for Meraki implementations) and, perhaps, ETA capable Netflow in Meraki devices (I believe there's a form of netflow already in MX units) going forward.

As an extra bonus, if your endpoints happen to use AnyConnect VPN clients, there's a great wee module in there called the Network Visibility Module. So whether an endpoint is on-net or off-net, you can be collecting network telemetry (including processes that kicked off the network traffic, etc) to be consumed by the likes of Stealthwatch (or other systems, such as Splunk) to get some super-valuable insight in to the behaviour of the endpoints.

Lastly, you might also be interested in Umbrella SIG (Secure Internet Gateway) in combination with the MX units. Here, it would be possible to use the Full Proxy service of SIG (relatively new) where, again, managed endpoints would have their SSL Proxy as the Umbrella SIG and, therefore, protected their (Umbrella service also runs AMP, for example, as a service).

Great discussion and kind regards,

Steve

[BrechtSchamp]

Thanks for that response @SteveMcKee !

[dalmiroy2k]

Finally! I can't wait to try the beta.

[GIdenJoe]

This is normally only used to get a decent blocking page out there for https traffic.

However I think Umbrella is a better way to go about this.

Maybe if you go to a public IP instead of an allowed DNS could the inspection be used to check that?

[peto]

after a month of waiting the support finally enabled the ssl inspection. now my question is what kind of certificate I need to upload? I created the root certificate with the private key using openssl uploaded it but it doesnt work - no https page loads.

[PhilipDAth]

I had the same problem.  I think it is broken in the current 15.x release.

[peto]

after upgrading to the newest Beta it works somehow. The problem is that the webpage doesnt load completely. 

[MyHomeNWLab]

For your information.

I contacted Meraki Technical Support because I could not see the documentation.

The HTTPS Inspection document has been intentionally de-published.

There are the following reasons.

HTTPS Inspection is BETA version and the specifications are subject to change at any time.

HTTP Inspection could lead to a significant impact on existing functionality and network stability.

In addition, the public release of the document was cancelled due to the high risk involved in operating the system for general users.

I contacted support in Japanese, so the nuances may be slightly different when translated into English.

In my opinion, I don't consider it a serious matter because there are ways to offload to Cisco Umbrella.