trouble of setting S2S between vMX and Azure VPN Gateway

[IntrovertEngg]

Hello All, 

Good Day. Need your help with below setup... 

my client has 2 separate entities as below in network term.

organization A = 35 spoke Meraki sites with Physical Hub MX250 * 2

organization B = 30 spoke meraki sites with Physical Hub MX250 * 2

client is moving DC services to Azure and has built vWAN hub. Has asked me to spin up 2xvMX per organization in Azure. Requirement is to bring up those vMX, setup peering with vWAN Hub, introduce that as 3rd and 4th Hub in respective organization, spoke should use those newly introduced but less preferred hub in case any azure subnet needs to be accessed, there will be a time when physical hub will be removed once full data is migrated to azure. Both organization (A & B) will connect to same vWAN hub however both organizations are not supposed to talk to each other or share any server/app resources at all.

Right now I am just focussing on Organization A. 

Client team gave me /24 VNET dedicated to SDWAN and subnetted it further to /28 for 2*VMX deployment. reserved further IP for future requirement if may.

I deployed vMX and can see those online in dashboard as well, received public IP (Seems this is Natted IP, correct?) anyways bringing up vMX seemed to be really the easiest part. Now trouble starts from here. Client does not want me to setup eBGP peering between vWAN hub and vMX but deploy S2S VPN (Non-Meraki VPN Peer) between vMX and VPN GW attached to vWAN Hub.

1. I have setup vMX in passthrough mode.

2. I have configured local subnet as /24 VNET dedicated to SDWAN

3. I have configured public IP of VPN GW, then configured private subnets inclusive of test service subnets, vWAN Hub subnet. 

4. I have also configured IPSEC policies to match with Azure VPN GW.

More or less similar things are done from VPN GW side, Client has configured WAN IP of vMX as remote IP and SDWAN /24 subnet as private IP... 

Below IPSEC policies are configured on both sides... 

  Phase 1

Encryption - AES256

Authentication - SHA256

PSUEDO-RANDOM - DEFAULTS TO AUTHENTICATION

DH - 14

LIFETIME - 27000

  Phase 2 

Encryption - AES256

Authentication - SHA256

Pfs Group - Off

Lifetime - 45

Now no matter what I do, VPN fails. It keeps showing red signal in front of vpn peer. I ran packet capture and I was surprised to see no activity related to S2S, no attempt to setup tunnel, any initiation or failure is observed. Its complete radio silence as long as S2S is concerned. Not sure why is that?  I tried to generate ICMP traffic between two vnets by creating test VNET in the hope IPSEC will try to ride on that interesting traffic but no.

One thing on azure side - no route table is configured - per client it is not required. .

What could be going wrong here? Why there is no attempt from Meraki to even initiate IPSEC tunnel ? Any guesses guys ?

1. Is setting up S2S good idea here? I was told BGP is not possible with single vWAN for multiple organizations and soon there will be 2 more organizations may get acquired.

2. Or you think setting up BGP peering is good idea and that way can get rid of S2S and also easy for spokes to learn routes? Do you think vWAN can form neighborship with all different vMX ? Should I assign ASN per vMX or ASP per Org and peer it with vWAN ? 

3. If anyhow S2S setup works, how do I let Meraki branches know about azure service subnets? How can those branches reach out to vMX, go through IPSEC and land on azure. I was reading any subnets learned from Non Meraki Peer wont be advertised via AutoVPN.

4. Regarding IPSEC not coming up - Can there be anything wrong with resource groups ? I have both VMX deployed under different RG ? I was reading somewhere it can be a challenge may be but was unable to understand. 

5. Can VNET to VNET peering  work here and take out the need to have IPSEC tunnel between vMX and VPN GW ? In that case should I ask client to peer vWAN Hub VNET and SDWAN VNET? How about incoming/outgoing route filtering as multiple organizations SDWAN VNET will come in the picture? Can route table/network intent be able to separate traffic ?

6. I see nothing in logs but couple of events like " Non-Meraki VPN negotiation msg: FIPS mode disabled "  what does it mean ? Anything wrong with IPSEC policies here?

Obviously ran into CISCO TAC and not having enough guidance so far. I hope I was able to explain my query... Any help would be really really appreciated.

[alemabrahao]

Did you check this thread?

https://community.meraki.com/t5/Security-SD-WAN/Azure-S2S-VPN/m-p/132160#M32990

[IntrovertEngg]

Thanks, I will have to check this with cloud team. Currently that VPN GW has already made IPSEC with FW and working fine. 

[PhilipDAth]

This approach will crash and burn.

Also note that you need to install the VMX in passthorugh mode in Azure for your use case.  If it was not in passthrough mode when you deployed it you'll need to delete and re-deploy (the mode can not be changed post deployment).

Also make sure you select "none" for availability zones when deploying to get a public IP address that is allowed to accept inbound traffic (if you select an availability zone all inbound traffic is blocked and you can not change this without deleting and re-deploying).

You can make this work without BGP or IPSec by simply using static routes in Azure, and defining local subnets on the VMX (which say what to route to Azure on the Meraki side).

[IntrovertEngg]

Hi Philip,

Thanks.

vMX is in passthrough mode.

Since I had two vMX for each org,  I was told to deploy first in AZ1 and second in AZ2 for high availability. if I select NONE, wouldn't it compromise the HA? Or given additional vMX it should be OK? BTW both are going to be active active. Also is there any documentation link you could refer to which mentions AZ challenge as you suggest?

Static route configured where? Which instance it should be configured on ? Are you referring to UDR or route table? and I can configure supernet covering all physical branches network or subnet per site under local subnet in vMX to advertise towards Azure side and that will anyways reach to vNET gateway and driven further correct? Because given in passthrough mode, I dont think I can configure any static route on vMX anymore. 

[PhilipDAth · Accepted solution]

>Since I had two vMX for each org,  I was told to deploy first in AZ1 and second in AZ2 for high availability. if I select NONE

If you found it unreliable and intermittently had to restart the VMXs to get AutoVPN going again - would you call that compromised HA?  Because that is what you'll experience if you use availability zones.

Actually, since you have two VMX for each org the static route approach is more complicated.  You will have to use this approach:
https://documentation.meraki.com/MX/Other_Topics/Deploying_Highly_Available_vMX_in_Azure 

What this does is monitor each VMX and updates the static routes in Azure should one fail.  Note that the Meraki instructions and the Azure instructions both contain serious grevious errors, and it is quite difficult getting this approach working if you have not done it before (you need to be able to figure out, correct and work through the documentation errors).

You'll need a supernet static route in Azure pointing to a VMX from each org for the branches that sit behind that VMX.

This is where you add the subnets located in Azure in the VMX (you'll need to do this on all VMXs).  You'll also need to add routes here for the branches sitting in the other org.

You are much better off configuring the BGP approach:
https://documentation.meraki.com/MX/Deployment_Guides/vMX_and_Azure_Route_Server 

So be warned.  This is doable, but it is going to be labour-intensive, complex, and it is going to need a good head when running into documentation errors and figuring out what the correct approach is.

[IntrovertEngg]

Hi Philip thanks for this insight.. helps a lot

. So latest development.

We dropped the plan of s2s peering because of need to setup tunnel from every network to  azure peer and not just from hub vMX. 

Now I have two org to connect with azure so for first org i enabled vnet peering between vmx vnet and vwan hub... then I added azure routes under local network.

but when I added vMx in hub preference for remote site.. couple routes were shown red and couple were green on remote site.

I was able to ping those routes from vMX but not from remote site.. i think I need to convert vMX into routed mode then also add static routes for local networks and then it should work, what do you think??

For second org, I enabled bgp peering between vMX and vWAN and received all routes and also got propogated to remote site but there was no filteration from azure vwan so received every route. Azure is going to release route map in production soon, hope we receive it on time.

One thing though.. given azure FW is coming in between vWAN hub and vMX.. its public IP is being masked to both instances of vMX under every organization. Is this right, will it affect later?

[PhilipDAth]

The VMX should be in concentrator mode, not routed mode, and it should be a hub.

It sounds like AutoVPN had not come up between the VMX and some of its spokes.  Is that what you are describing?
Did you definitely select an available zone of none?

Enable manual NAT traversal, it is much more reliable.  Note you MUST have selected an availability zone of NONE for this to work.

https://documentation.meraki.com/MX/Site-to-site_VPN/Automatic_NAT_Traversal_for_Auto_VPN_Tunneling_... 

[IntrovertEngg]

Client is refusing to let me select AZ as NONE. As per him, not selecting AZ for both instances would put them in any single AZ and if that AZ fails - both instance will lost the connectivity. Do you have any CISCO documentation confirming this about selecting NONE AZ or this is your experience based on practical scenarios which is not yet documented in Meraki guide.

I have enabled manual NAT traversal, thanks for that but not sure how remote sites will be able to form Auto VPN tunnel with two vMX who are having same masked IP from Azure FW. I have used different UDP port for each vMX but will it work properly do you think?

[IntrovertEngg]

Now I see all routes in green on remote end which are advertised on local network of vMX however i am not able to ping those from remote sites. I tried to ping SDWAN VNET GW IP and I receive no response found error which means return traffic to be configured from Azure side ? Azure route table should be useful here correct?