Our Onboarding Journey so far....

[Jeff_Longley]

Thought I'd share some of our experiences with SASE so far.

1. Secure connect breaks dashboard. 
Not long after implementing secure connect tunnels, our switches went dark - Literally dark. We couldn't see any activity, whether stuff was connected or not. Support initially thought it was a failed switch, but as it was happening in every single site with a secure connect, we disagreed. The solution was to add the dashboard ranges to the VPN exception rules, which brought everything back to life. Have raised with the onboarding team, as it should surely either NOT break dashboard traffic by going through the tunnel, or be an exception by default.
The dashboard rules also fixed a problem with Access points complaining about their regulatory domains being incorrect (because they're routing through SecureConnect DC's and appear to have changed country).

2. Hubs not supported
We've a fairly classic hub and spoke auto VPN setup at the moment, with our hub site having a VPN concentrator. SASE currently doesn't support hubs. Its apparently coming, but right now, this has stopped our roll out.

3. MTU on VPN tunnels

We have some internal security boxes that VPN out of the network to a DC. These broke. The support discovered that the MTU size was set to 1280, which was halting communication. Once again, the fix was to add these to "bypass VPN tunnel" list.

The other frustration is that VPN exception rules ONLY (and despite what the GUI says) allows CIDR ranges, not Hostnames. 

Trying to remain positive, but there seems to be a lack of joined up thinking - The dashboard rules is an absolute own goal. I'm sure I can't be the only company that would like to be able to route traffic over internal VPN's to our DC, rather than through secure connect VPN's to our DC?

[Bucket]

Great information! My only experience so far with secure connect is the dcloud lab, so it's nice with some feedback from the real world.

Number 1 should definitely be an exception by default.

Regarding number 2. What do you mean by it isn't supported? What is preventing you from using the Umbrella hubs as exit hub on the hub in the DC? And why can't traffic from spokes to the DC go directly and not via Umbrella?

The fact that you can't do local breakout based on domains and wildcard domains is a big problem. That being said, when you go down this path it is expected to put in quite some work on bypassing umbrella completely or doing selective decryption for different destinations/applications. That's the same for all vendors.

[Jeff_Longley]

So my DC is a MX hub. Hubs do not appear as selections when you create the SecureConnect Tunnels.

So all my sites have tunnels to SecureConnect, and tunnels to my DC.
However, if I wanted to VPN into my DC, SecureConnect cannot see it because it doesn't have a connection to it.

[Gary_Geihsler1]

1. We have found that not every deployment has caused issues with communicating to Meraki dashboard over Secure Connect tunnels, hence there is not default exemption of Meraki dashboard destinations in VPN exclusions. We are working on root cause analysis. 

2. Hubs will be supported soon, no public timeline to share beyond that. Please engage your Meraki team for more detail if desired. 

3. The MTU size requirement is documented, we are looking to provide the option to increase MTU and support fragmentation, work ongoing in these areas. 

4. VPN exclusions are only by CIDR for TCP/UDP/ICMP and for DNS hostnames are supported. The UI is a bit confusing on this front, feedback submitted. 

[PhilipDAth]

Thanks for the feedback.  I'm waiting to wait this one out another year for it to mature more.  🙂

[PhilipDAth]

>The dashboard rules also fixed a problem with Access points complaining about their regulatory domains being incorrect (because they're routing through SecureConnect DC's and appear to have changed country).

I would have that for sure in my case.  I'm based in New Zealand, but our nearest node would be Australia.

Another problem with using Umbrella SIG (which this is based on) is it breaks Google search and maps locality searches because it sees you searching for a different country (Australia) rather than the country I am located in.

[Gary_Geihsler1]

Umbrella does support XFF headers so the location would be preserved. AnyConnect, proxy chaining, PAC file, and IPsec are supported connectivity methods. the Meraki Cloud On-Ramp and Secure Connect are not currently supported. 

[PhilipDAth]

I can tell you for a certainty @Gary_Geihsler1 , it doesn't work.  I have had Umbrella support engineers look at this case.  It is easy to demonstrate.

Turn on Umbrella SIG and search for a location on Google Maps, and turn Umbrella off again and do the same search and get results from different countries.

Google shows you news articles from different countries.

Umbrella SIG breaks anything Google related that is location-based.

They gave up trying to solve it.

[Jeff_Longley]

That's going to be fun with our users then.....  We had to move from a centralised MPLS solution several years ago for similar reasons!

[Gary_Geihsler1]

I will try to find some public documentation on the support for XFF header. Hard to prove out with the way Umbrella works. Best I can provide is screenshot of my device on SIG with the XFF header of my actual public IP which will be respected by websites that honor the XFF, as google does. This is using Secure Client as the connectivity method, Secure Connect and Meraki Cloud On-ramp are not currently supported with XFF. the XFF became the standard way SWG operates about a year ago. 

[Jeff_Longley]

Just updating this:
Currently we're stalled on deployment. Hubs are the stumbling block at the moment. 
Surely I'm not the only customer who wants to be able to route to their datacentres over their Auto-VPN tunnels when clients are in a Meraki site, and VPN into that datacentre when remote?

We've also started hitting occasional problems with websites not liking the exit points (XFF header). We've been offered a paid for option where you can reserve IP's in certain regions so that you appear to exit from them. One option would be a local breakout exception, but it only appears to accept CIDR/IP rather than a hostname.

[Gary_Geihsler1]

We have a private preview of Meraki hub to Secure Connect as a supported topology, as we progress we will continue to work with your organization to implement when appropriate. 

We can not control which sites/services respect XFF headers and which do not. 

VPN exclusion is available for major applications, DNS based on hostname, and L3 via CIDR. 

[Jeff_Longley]

Yes, we were told that over a month ago.

Latest update is "There were some stability issues with hub connectivity feature last week, so engineering is trying to stabilize it before connecting new customers"

[Jeff_Longley]

One Step forward, two steps back.

So, we've now got our hub enabled for secure connect. Sadly, we haven't got around to doing any testing as we're now battling our spoke sites blocking DNS traffic - ALL DNS traffic, including Umbrella. 

We notice it first when all the switches/AP's in the site suddenly report that "DNS is misconfigured" - it's not, it just can't talk to the DNS server its configured for. Then the flood of user calls come in. Support confirmed with a packet capture that they can see DNS requests going out, but nothing ever coming back.

Quickest fix? Remove secure connect tunnel and everything is restored.

[Bucket]

I wonder if it has anything to do with the service DNSCRYPT not running on the MX. Meraki has a bug where the service stop running so DNS traffic tunneled to Umbrella stops working. A restart to of the MX solves the issue, but it might or might not come again. They don't know why it happens atm.

[Jeff_Longley]

But if dnscrypt failed, traffic wouldn't leave the MX and it would be dropped?
We had no issue resolving DNS via the autoVPN tunnel, and packet captures showed DNS request going to secure connect.

[Gary_Geihsler1]

The support team and PM team are working to help resolve the issues during the private preview of hub to hub connectivity. 

[PhilipDAth]

This has been a terrible experience!

I'm going to buy a licence so I can play with this.  It seems you might be on the bleeding edge, as no one else seems to have experience to be able to help you.

[Jeff_Longley]

So, coming full circle, we're fully onboarded and everything is working as it should.
We actually had a second onboarding session (after our first one had ended) and it was a much more thorough experience.

Is the solution perfect? No
Does it work? Yes

Things we're still hoping for improvements on.

Always on / Auto Sensing VPN. We have endless problems with using connecting the VPN when they're on a MX site. I'd have thought it relatively straight forward to have a either a whitelisted domain suffix or IP range, which tells the client NOT to connect.

Bypass rules are still CIDR only and can't be logically grouped (so for example, to Bypass Meraki Dashboard, you can't create a single entity called Meraki Dashboard and add all the IP's in). You add comments either, so looking at the Bypass list to figure what's been done and why is "fun".

No auto client update - we have 3rd party solution for updating software, but I'd much prefer if the client can look after itself!

[PhilipDAth]

I am playing with this feature myself at the moment, and your feedback is good.

The Cisco Secure Client management portal lets you cloud manage Cisco Secure Client clients.  For Asia Pacific the URL is:
https://secure-client.apjc.security.cisco.com/

We use this to keep all our clients up to date.  You can choose specific versions, or a simple option like "Latest" or "Recommended".

It also allows you to create multiple deployment groups.  We have "Production" and "Pilot" to make testing changes easy.  From the cloud portal, you can move users and groups of users between the profiles, and they gets the updated settings automatically.

It also lets you configure the "AlwaysOn" feature (called Trusted Network Detection).  This can trigger connect/diconnect on things like DNS servers given out by DHCP, domain name, etc.

Have you seen the new client based ZTNA coming out August 15, 2024?  This lets you define applications that are proxies through Umbrella without the user having to touch the VPN client.
https://documentation.meraki.com/CiscoPlusSecureConnect/Cisco_Secure_Connect_-_ZTNA_Architecture_Sta...

[Gary_Geihsler1]

Jeff, the comment on having issues connecting to the VPN when onsite. Is the scenario the user tries to initiate the VPN while onsite and something is not working as expected? In most cases the VPN should not be needed while onsite. 

The ability to use groupings and/or objects in VPN exclusions is good feedback. I will submit to the MX team for consideration. 

As Philip mentioned the Secure Client Cloud Management can allow you to pre-package a single executable with the appropriate profiles (VPN, Umbrella Roaming Security) loaded. You can also manage upgrades from the Cloud Management. Not all options shown in cloud management work for Secure Connect. Always-on VPN si not supported in Secure Connect currently. 

[Jeff_Longley]

Its more a user problem - If we don't auto launch the client, users don't connect to the VPN and log a ticket. If we DO auto launch, they'll connect VPN when they're on a site that's already connected to our DC, and log a ticket saying everything's really slow....

Groupings and Objects would be great - would be EVEN better if they could be set globally too (much like radius servers).

[Gary_Geihsler1]

I see, this challenge is one we have seen before. User education is the first answer-although one with multiple challenges of course. As was mentioned earlier, the new client ZTNA may alleviate some of this. The client ZTNA supports all ports and protocols and feels like an always-on VPN but is a different secure access method. The user experience is an initial enrollment process and then the user does not have to login when accessing private applications/resources. 

I created a short video showing the user experience.
https://app.vidcast.io/share/50f01122-21c9-454e-ba90-7a347c8f3dd3