Meraki

Davide_C · ‎Feb 9 2021

Hi guys,

A customer wants to implement Meraki SD-WAN to connect several Headquarters/DCs of different countries, each with its own WAN topology.

There are countries with MPLS, Internet, or both between HQ/DC and branches, the Design should be versatile.

From a High-Level standpoint, I would consider any country as a hub-and-spoke topology, all the spokes should communicate with their hub through Internet, MPLS, or both, and all the hubs communicate through Internet.

First question: would it be ok to implement a VPN Mesh inside countries if there are only a few spokes?

Suppose to have HQ/DC and Branches connected through both MPLS and Internet connections.

In the HQ/DC, we have dedicated firewalls at the edge and the MXes deployed in a One-Arm VPN Concentrator fashion, as Cisco suggests for Hubs.
Let's suppose that in the Branch the MXes should provide both the VPN and the Security/Gateway functions.

1 - Would be the MXes in the HQ able to bring up multiple VPNs through the single-arm toward the internal transit? Should they be placed to the edge, in front of Firewalls, in a Dual-Arm fashion (The first WAN connected to the MPLS transit and the second WAN connected to the Internet transit)?

2 - Would be possible to deploy MXes in the Branch in a Dual-Arm fashion (as supposed before)?

3 - Suppose we have edge firewalls in the Branch, with the same logical topology as the HQ, would be right to use a One-Arm deployment also there (If it works as supposed in the first point)?

Thanks all in advance for the support,

Davide

Davide_C · ‎Feb 18 2021

Here it is:

Cisco Meraki Best Practice Design - Cisco Meraki

Under the "VPN Registry" chapter:

The process for adding a new MX into an infrastructure is as follows:

A new MX reports its uplink IP address(es) and shared subnets to the registry
The information is propagated to the other MXs in the infrastructure
The MX establishes the proper VPN tunnels
1. The MX will try the registry-reported private uplink IP of the peer first
2. If a connection to the private uplink IP of the peer fails, the MX will try the public uplink IP of its peer

I didn't spot this document, it is basically the root of all the other design documents 😆

I'm gonna read it all!

View solution in original post

Bruce · ‎Feb 10 2021

1 - Yes, the MXs will be able to form multiple tunnels, each tunnel uses a different UDP port. Depending on the firewall you may need some ‘fine-tuning’, but fundamentally it will work. You will need to ensure there is a path from the MPLS network via the firewalls to the Meraki cloud (specifically the VPN registry, as well as others).

2 - Yes, the MXs in routed mode, with two WAN connections (one MPLS, one internet) is fine, you just need access to the Meraki cloud from the MPLS network; see the mention around internet access via the firewalls above.

3 - If you do single arm in the branch you’ll only get one VPN tunnel via one path - so no SD-WAN and no load balancing/performance routing of traffic, and no security features, so you lose most of the benefit of the MX solution. Probably not a great idea.

Davide_C · ‎Feb 10 2021

Hi Bruce and thanks a lot for your reply.

1 - How can you tell the MXes that there are two different paths (Internet and MPLS) when they are placed with a single-arm behind the L3 Core? Do they need also to access the Meraki cloud directly from the MPLS (so with a different public IP than the Internet connection - I suppose the Internet path is enough)? Is there any public document that describes that behavior?

2 - If there is already an Internet connection in the branch shouldn't the MXes use that to access the Meraki cloud? Do they need to do it going through the HQ?

3 - So there are limitations when you deploy MXes as spokes, are they documented somewhere? If I have dedicated firewalls in the branches and I want to keep them at the edge of the network and place the MXes behind, how can I do it? If I lose SD-WAN with a single-arm deployment in the branch, the only way to implement it is by configuring the MXes in Routed Mode and therefore with NAT enabled? The No-NAT feature is still in Beta?

Actually, the third point has a lot of questions 🙂

Thank you very much,

Davide

Bruce · ‎Feb 10 2021

1 - The MXs register to the Meraki VPN registry with their given IP addresses (whether this is private or public) from all the WAN ports and the VPN registry also learns the public IP address that the registration is coming from. From this it will instruct the MXs between which IP addresses to form the VPN tunnels. Its through this 'magic' that the VPN concentrator gets to learn about the two paths to the remote MX. (There was a good Meraki document on the the operation of the VPN registry, I just can't find it at the moment).

2 - They need connectivity to the Meraki VPN registry over both the paths, that's how it learns about the two paths. Yes, the majority of the management traffic will go over the primary uplink, but there needs to be a path to the Meraki cloud on the secondary link too.

3 - I wouldn't say there are limitations with deploying the MXs as spokes, its about getting your design right so that all the devices in the network function as they are meant to. I've mostly seen the spoke MXs deployed as both the edge of the network (i.e. the firewall) and the SD-WAN device. You could for instance connect your internet circuit to the firewall, then connect that back to one WAN port on the MX; the other WAN port on the MX can be connected to the MPLS circuit, the outcome is the same.

With regards to no-NAT, its definitely a feature, whether its Beta or not I don't know. You'll need to get support to enable it for you, and they can answer that question too. You need to consider how you would use the no-NAT feature too. It wouldn't be needed for the traffic being tunneled to the hub as the IP addresses are maintained inside the tunnel anyway, and in general the NAT on the MX and the outer firewall won't impact traffic. Where I could see you may want to do it is between the MX and the firewall on the internet if you are using some form of user to IP address mapping to influence the firewall rules; but then you could also be doing that on the MX if you wanted to. It all comes down to what you are trying to achieve, and how you want to achieve it.

Davide_C · ‎Feb 11 2021

Bruce thanks for your explanation, but it brings me to other doubts 😆

In a one-arm deployment, we have the MXes that work as hubs connected to the L3 Core with only one WAN port (correct me if I'm wrong), configured with a private IP address of the internal transit subnet.

Internet Tunnel:
The MXes register to the Meraki Cloud through the Internet cloud (because the Firewalls have a default route toward the Internet CPEs), so the Cloud learns the private IP, the public IP, and the public port used by the MXes on that path.

MPLS Tunnel (This is tricky for me to understand):
If the MXes need to reach the Meraki cloud also over the MPLS path we need an "Internet exit" in the MPLS cloud, but MPLS is supposed to be a private L3VPN usually isolated from the Internet (at least in my country ISPs may not support this configuration).

1 - How does the cloud learn about the MPLS path if we don't have an Internet exit in the MPLS cloud?
2 - Suppose that the ISP provides that internet exit (and we load balance the default route between the two paths), How would be the VPN tunnel created over the MPLS network (using private IP addresses)?

MX spokes limitations:
You said: "If you do single arm in the branch you’ll only get one VPN tunnel via one path - so no SD-WAN and no load balancing/performance routing of traffic, and no security features, so you lose most of the benefit of the MX solution".

So I suppose there are limitations based on which deployment mode you choose, because we may have the same topology in the spokes (a firewall couple at the edge that we don't want to touch with an L3 Core behind), but spokes would not be able to form multiple tunnels. Did I understand wrong?

Many thanks Bruce for your support 🙂

Bruce · ‎Feb 11 2021

Yes, you are correct about the MX in VPN concentrator/one-arm mode - single connection, single IP address.

Internet Tunnel: Yep, that how it works, the MX contacts the Meraki VPN registry and registers its private IP address and public IP address, along with the port its going to use for the VPN.

MPLS Tunnel: The MPLS network has to have a path to the internet, whether this from the MPLS network (e.g. provided by a firewall hosted by the service provider) or whether its via the hub site. When any MX contacts the VPN registry via this path the public IP address will be the same for all devices, and thus the MXs are instructed to create the VPN using the private IP address.

1 - You have to have internet access from the MPLS network, this could be directly from the cloud by the service provider, or by a default route being injected into the MPLS routing from the hub site where the VPN concentrators are. The MX management traffic is never encrypted into the VPN tunnel, it is always sent natively on the WAN port, and contact with the VPN registry is required on both WAN ports.

2 - The VPN registration is performed from each WAN port, not from the MX as a whole, and for each WAN port it will use the gateway configured for that interface. Management traffic is not impacted by the MX configuration (load-balancing, etc.) it always uses either the configured primary interface or a specific interface (i.e. for VPN registration).

MX spoke limitations: You've got the gist of it, and if you want to call it a limitation, then that's okay. But if you have one device at each end of a path, and each device has only one IP address, then traffic between the two is always going to take the exact same path since routing is based on the IP addresses. Running in NAT/routed mode provides two IP addresses to a device, which is what then allows SD-WAN, or two different paths.

Davide_C · ‎Feb 12 2021

Thanks again Bruce,

so is it right to say that MXes first try to establish VPN tunnels on a specific WAN port using its public IP address and if it is the same with the destination then they use private IP?

Is there any public document that describes this behavior?

Suppose that the MPLS has its own Internet exit, the public IP would also change on the MPLS path.

In that case how could the Meraki Cloud know that it should use private IPs on that path to form VPN tunnels inside the MPLS?

For the MX spoke limitations, I got it, it's simply routing 😆

Bruce · ‎Feb 12 2021

I don’t believe the MXs try to build the VPN Tunnel with public IP addresses, and then fallback to the private. My understanding is that this decision is made in the Meraki Cloud using the VPN Registry information, then the MXs are instructed what IP addresses to use to form tunnels. There used to be a Meraki document that explained this but I can’t find it now, the closest I can find are these two, https://meraki.cisco.com/lib/pdf/meraki_whitepaper_autovpn.pdf, and https://documentation.meraki.com/Architectures_and_Best_Practices/Cisco_Meraki_Best_Practice_Design/....

If there are multiple exit points from an MPLS network, say one offered by the provider, and then also one via the data centre with a one arm concentrator then you may have a problem. As you say if the VPN registration is from different public IP addresses then you may get sub-optimal or unexpected VPN tunnels being formed. I don’t know the exact logic that is used to determine the IP addresses to establish the VPN tunnels between, but I’d tend to stick with a single internet entry/exit point to the MPLS network - normally the DC where the VPN concentrator is. If the MPLS provider is providing an internet exit point then at the DC you could use the MX in NAT/routed mode instead of a concentrator.

Davide_C · ‎Feb 18 2021