VMX 100 communication with multiple AWS account and VPCs

[Blur_nom]

Hi Guys,

I have set up a vMX on a VPC in AWS. The vMX is able to communicate with another VPC through peering. However, if i were to have a site to site VPN configured for the vMX to a physical MX. The clients behind the physical MX were not able to reach the VPC. According to AWS, VPC peering doesn't support L3 routing information which is why i wasn't able to reach the VPC.

Any recommendation? This is my setup. MX -> VMX -> VPC

Also what is the best way to connect up my VMX to multiple VPC across multiple AWS accounts. 

[PhilipDAth]

With the VMX, it will route all subnets listed under:

Security Appliance/Site-to-site-VPN/VPN Settings/Local Networks

Here is a screen shot of one I do serving multiple availability zones.

If you have two VPC's peering with each other you should be able to get to the remote VPC.  You will need the remote VPN routes listed here, and on the remote VPC you will need routes to get back to your remote VPN subnets.

You can not connect a vMX across multiple VPCs across multiple accounts.  You need multiple vMXs.

[Blur_nom]

Hey PhilipDAth,

 That is exactly what i did. So basically in my setup of MX -> VMX -> VPC. My VMX was able to reach both MX and VPC. However, my MX wasn't able to reach the VPC. So according to Meraki support, they are able to see my ping going out. Just that the VPC attached via peering to the VMX does not have a route back.

AWS tech confirmed that the VPC is not supposed to pass routes which it doesn't know about. Thus setting of default route wouldn't work. They are recommending the usage of transit VPC.

[Scobb]

Did you ever get this figured out?  I have a client with 3 VPCs, all in the same AWS account, and I'm wondering if I install a single vMX in one of the VPCs, if I can connect other subnets in the other 2 VPCs back through the SDWAN of the single vMX via VPC Peering.

Has anybody done this?

[PhilipDAth]

The whole point of a vPC is that it provides a virtualised network environment.  Their is no communication at all between vPCs.  Because their is no communication between them, a vMX is one can not possible talk to the others.

If you are instead referring to networks that belong to a single vPC, then yes, you can do that.

[Scobb]

There is actually communication allowed between VPCs.  See https://docs.aws.amazon.com/AmazonVPC/latest/PeeringGuide/Welcome.html.  I'm just wondering whether I can use this type of peering to send traffic from all VPCs destined to the branch network across the Auto-VPN/SDWAN to flow through the one vMX.  It sounds as if that is not the case though, which means this client will have to have a vMX for each VPC.  Bummer!

[Todd-rmn]

Here's our plan that we are about to test:

We'll have our vMX living in a VPC that will connect to branch office Meraki hardware via Auto-VPN

We'll have our vMX connect to other VPCs via IPSec/AWS VPN Gateways in each VPC.

Anyone else doing this type of thing?

[Scobb]

We have not tried this yet, but we did try to connect a branch MX back through a vMX to a peered VPC and that did not work.  So, our case looked like this:

MX ><AUTOVPN><vMX/VPC-A><VPC PEERING><VPC-B

I can ping VPC-B from the vMX and vice versa, but when I ping from the MX to VPC-B, I get no return traffic.  With doing a packet capture, I can see the ping hit the vMX and be sent to VPC-B, but I never get return traffic to the vMX that would be passed to the MX.  Essentially, the VPC peering does not allow routing across the peering connection.

We are getting ready to try exactly what you are proposing.  Let me know how it goes for you!

[Todd-rmn]

Has anyone tried this?

office MX(s)> <AUTOVPN> <vMX in VPC-A> <IPsec Tunnel> <AWS Managed VPN in VPC-B

I can't really find much on this approach, but in theory it should work...Even with tunnels from the vMX to multiple VPCs right?

[Scobb]

We just recently tried this and it does NOT work.  The reason is that the vMX (and any MX for that matter) will not route from a Meraki Auto-VPN connection to a 3rd party site to site connection.  So you can stand it up, but you can't route traffic from the branch MX to the AWS VPC connected by the AWS VPN.  We have modified our architecture so that we have a site-to-site VPN from every branch MX to the other AWS VPCs that we need to access.  Not the best, but the only option.

[Blur_nom]

So this is what i have gathered by far. 

VPC peering does not pass external routes. As such peering other VPC to the vMX is not going to help.

Amazon is proposing a couple of things 

- Transit VPC

- AWS direct connect

- Cisco CSR router.

I believe the above-mentioned solution is the only way out unless somehow xMX supports NAT-ing. 

[Todd-rmn]

"The reason is that the vMX (and any MX for that matter) will not route from a Meraki Auto-VPN connection to a 3rd party site to site connection."

Meraki support has confirmed this, but it's a shame this isn't documented anywhere.

Regardless, our only options seem to be:

Add Meraki vMXs to each VPC that we need to connect to...using Auto-VPN to connect all hardware and virtual MXs.

OR

Link hardware MXs to each other via Auto-VPN and then also create an individual VPN connection between every office and every VPC.

[Anish-Patel]

I am looking to setup something similar.  

LAN -> physical MX -> vMX -> out my vpc.

Does the vMX need two interfaces or will the public subnet interface land the VPN and then decrypt and out the same interface out to the iGW?

[PhilipDAth]

The vMX can only have one interface.

The decrypted traffic goes to the default gateway - which is usually the VPC.

[THE_Daghan]

I have the exact same problem! Ohh... the irony, considering I was the PM who launched MX 8 years ago...lol!

[CarolineS]

Haha hi @THE_Daghan! Nice to see you here 🙂

[THE_Daghan]

Thanks CarolineS. I couldn't stay away. 🙂

[helgaali]

Hi everyone, 

Have you tried AWS transit gateway! it's a new service I tested and it worked for that use case you need. In my lab, I have the following diagram. and I was able to ping "storage" client from the client behind the physical MX.

 

Also, Transit gateway works between different accounts as per this here: 

I hope this helps someone!

[PhilipDAth]

Thanks for that @helgaali .

I've talked to many customers about this, but haven't actually had the opportunity to do one of these yet.

[louie]

Hi Helgaali,

Can you share what VPC configuration you have done or what routing table entries you created. We are trying the same approach in our lab and using your example, we cannot ping storage from behind the MX.

Thanks.

[DTanzmann]

Hi helgaali

(referring your diagram) 

how did you setup routes back to MX from  VPC-A  linked through Transit GW?

I can ping client behind the transit GW in VPC-A from vMX but cannot ping this client from MX

thanks

David

[DTanzmann]

I have figured it out ... it actually work pretty well!!!

[bforst14]

Any chance you can provide guidance on your VPC/TGW routing? I have a vMX vpc with the appliance set up. Cant figure our which mode to use and how to configure routing to enable bidirectional flow. 

[PhilipDAth]

This option - and you have to do it PRIOR to deploying the VMX.

[bforst14]

I've created a VPC with it's own egress path. I have my vMX in Passthrough mode. I have defined local subnets in vMX (AWS Subnets with VPC routes pointed to the TGW) and I have a summarized route in my VPC route table pointed at the vMX ENI. I also added a route for the summarized field MX range to my TGW pointed at the vMX VPC. Experiencing some pretty weird stuff where I can ping my AWS subnets from field office MXs but can't ping Field Segments in VPN from AWS Hosts. Thoughts? Hard to find full guide with VPC/TGW routing included.

[LibertyM01]

Hi helgaali,

I'm working on a similar design. Can you please share the configurations within your VPCs i.e Route Tables, Internet Gateway and Security Groups. Also, do you define the two VPCs as local to the VMX in the Meraki Dashboard.

Thanks!

[TyShawn]

I'm running across this same issue. I have my vMX in a hub network with multiple VPCs connected to the hub VPC. Like others I can ping between the vMX and any device in any other VPC, but the return traffic from AWS is not be generated back to an on site subnet. 

[bforst14]

I can help if you can better explain your topology. I have my vMX with its own Elastic IP assigned to the EC2 and an IGW for outbound. What does the route table look like in the VPC/subnet where your vMX is?

[TyShawn]

So working with AWS when I had the VPC peering in place the routes were good tcpdump showed the traffic hitting the VM and the VM sending the traffic back. What was noticed is the next hop was not identifying the VMX as the next hop causing the packets to drop. The fix for me was to place transit gateway in place and then configure the routes in the TG/ each VPC / and the vMX. As on this moment I have traffic processing through the vMX to my VM.