REPLACE CISCO 5500-X WITH MERAKI MX84 ISSUES

[confuzzled]

Hi Meraki community,

I'm having issues trying to setup my MX84. I'm working with support, but of course that's taking a while, and of course they always want to work on it when its convenient for them, and at the worst possible hours (like 10am on a Monday).

We are a state agency, technically a LAN inside of a LAN. Our current ASA is a Cisco 5512 which I am attempting to clone the settings from, and plug into the MX84. I've been told "this can't work" by other state IT, and that they've attempted to get Meraki ASAs to work in this setup before, but were never successful. Currently, they claim the only way to have this work is to give the Meraki ASA a public IP. I do not think this is correct.
When I simply plug in and allow the MX84 to pick up an IP from a DHCP range, (say, in my office) everything works great. Its connecting to the Meraki cloud and all the lights are green, etc. When I try to plug in the static settings and test it after hours from the config running on the 5512, the power LED stays red and I'm not able to get to it from the dashboard.
So far, we've gone over every single detail I can think of. The "outside" IP settings of my current Cisco ASA match what I am putting in for a static IP on the Meraki (WAN2) port. It's 1st hop is a "router", an old Cisco 2900 series. From there the 2900 dumps into a Cisco 3750 switch, where it makes the transition from eth to fiber, and then hits the fiber box. From the 1st hop on is a different state agency in charge of "our internet service", so I cannot speak to how all of that is setup. I do have a very high level topo map, but that's it.

The "agency in charge of our internet service" says the reason the 1st hop doesn't work for the Meraki is because that IP "isn't rout-able". They're using a 10.x.x.x network schema for their LAN. Our state's public IPs are 164.x.x.x. My befuddlement is that the current Cisco ASA is working quite fine with its 10.x.x.x address, and passing it along to the 2900, then the 3750, etc. When I try after hours to plug the Meraki in, no go.

The latest troubleshooting method was to plug in a static route for the same address as the default gateway (which is also the first hop). The Meraki support guy I'm working with seems as confused as I am with why it isn't working. I'm a little frustrated since Cisco bought Meraki, I sort of thought this should be cake to them. My most recent idea tonight (Have the 2900 issue the static via DHCP to the Meraki) was shot down by the other state agency senior IT who said the 2900 as a layer 3 device could only act in the static role.

Any help would be greatly appreciated. I think this can work, I think I may just be missing something simple / stupid.

[PhilipDAth]

An MX can work with either a public or private IP address on its WAN interface.  The only requirement is that it can talk to the Meraki cloud on the Internet.

You can copy your 5512 "outside" interface settings to the MX, even though it is a 10.x.x.x addressing.  However you also need to specify some reachable DNS servers - otherwise the MX wont be able to do a DNS lookup to get to the cloud.  You might have some specific ones you are allowed to use, or you might be able to use Google's, 8.8.8.8 and 8.8.4.4.  And obviously you need to specify the default gateway which is the same as the default route on the ASA.

If the MX is not coming up connect to the local status page and see what error it is reporting.

https://documentation.meraki.com/zGeneral_Administration/Tools_and_Troubleshooting/Using_the_Cisco_M...

If the ASA has a VLAN tag configured on its outside interface make sure you configure the same tag on the MX WAN interface.

Does your Meraki MX plug directly into your default gateway?  If it doesn't then you'll need who ever is in charge of the default gateway to do a "clear arp" on it. Otherwise the default gateway wont know that the MAC address for the assigned IP address has changed.

[PhilipDAth]

An MX can work with either a public or private IP address on its WAN interface. The only requirement is that it can talk to the Meraki cloud on the Internet.

You can copy your 5512 "outside" interface settings to the MX, even though it is a 10.x.x.x addressing. However you also need to specify some reachable DNS servers - otherwise the MX wont be able to do a DNS lookup to get to the cloud. You might have some specific ones you are allowed to use, or you might be able to use Google's, 8.8.8.8 and 8.8.4.4. And obviously you need to specify the default gateway which is the same as the default route on the ASA.

If the MX is not coming up connect to the local status page and see what error it is reporting.
https://documentation.meraki.com/zGeneral_Administration/Tools_and_Troubleshooting/Using_the_Cisco_M...

If the ASA has a VLAN tag configured on its outside interface make sure you configure the same tag on the MX WAN interface.

Does your Meraki MX plug directly into your default gateway? If it doesn't then you'll need who ever is in charge of the default gateway to do a "clear arp" on it. Otherwise the default gateway wont know that the MAC address for the assigned IP address has changed.

[confuzzled]

Hi Philip,

Thank you for so much good info. Your opening statement matches mine as far as my understanding to how Meraki's work. Now comes all the "fun" parts.

I've brought up the DNS part before - the "state agency that acts as our ISP" refuses to provide those internal DNS settings, citing security concerns. They are now some crazy cloak & dagger linux dns bank that no one is allowed to know what the IPs are except for them, was the summary of what I was told. I've been using google's 8.8.8.8 as my DNS1 when testing the Meraki and one of my internal DNS servers (which, to me seems like it wouldnt help a bit - but thats what you do out of desperation and when the response you get to your question of 'what DNS should I use' is a cryptic 'whatever you want').

To the rest of your excellent points (ty again by the way);

The default gateway on the MX is set to the router's IP. This matches and is the same as what the current ASA has for its "route outside 0.0.0.0 0.0.0.0 10.x.x.x 1" entry / static route, which if my understanding is correct, that is the default route on the current Cisco ASA.

I didn't know the status page reported anything, thats very good to know and I will definitely try that to see whats happening.

As to the last part, yes the MX plugs directly into the "router" which is the evasive way I'm explained how this convoluted monkey they've got running is configured. I'm assuming that since the current ASA is plugged directly into this "router", that is the default gateway. I've also power cycled this "router", which is a Cisco 2900 something I haven't googled yet to see exactly what the heck it is. So yes, when testing the MX also gets plugged directly into it, restarted, to make sure that nice arp table gets flushed as you had mentioned.

No VLAN tag that I am aware of.

[lpopejoy]

As @PhilipDAth brought up, I would verify that the WAN isn't using a VLAN tag.

And, also as he mentions, it could also be ARP.  Have the "agency responsible for internet" check the ARP cache on their router - and check to make sure there isn't a static entry defined. 

Interesting problem - I'm very curious as to what the solution ends up being! 

[confuzzled]

@lpopejoy,

Thanks - I'm going to specifically ask about both arp caching and any other strange config on that Cisco 2900 "router" aka 1st hop. You're the 3rd person to mention it, and its starting to tickle my spidey senses (if more than one person says something - pay attention).

Thank you for the feedback. I will email my contact now. Have a good Monday!

[PhilipDAth]

Here is something else to try.  Can you configure an ordinary computer with the static settings and plug it into the Internet feed directly and get that to work?  If not, you can take that back to your internal support people - as they can't complain that this case wont work.

[confuzzled]

Actually, that's exactly what Meraki support had me try. And here's the part where "everything I ever learned about networking..... doesn't apply today" comes in. And of course, it does not work.

So they said "yeah, its not supposed to, because that 1st hop / the router IP isn't rout-able. They keep saying that, like its supposed to "explain everything" to me. 

So here's the crap-funnel down to the LCD (lowest common denominator) for me.

Does a Cisco ASA have the ability; to "route" past a "non rout-able" IP, where the MX does not? Because apparently, that's exactly what our current ASA is doing, right now. Heck, I'm sending this message through it. Another thing I'm not that sure about - I'm looking at the ASA through ASDM, and it says firewall mode is "routed". Is that a key term for "extra special" meaning normal network rules don't apply?

Other than that, I think I'll wait till after hours and see what the status page error looks like.

Thanks @PhilipDAth

[PhilipDAth]

I think there is some confusion with the term "non routable ip" and "private ip".  These are not the same thing.

Lets get more specific.  Please obfuscate the 2nd and third tuples.  So if you had an IP address of 10.11.12.13, please describe it as 10.x.x.13.

What is the IP address, subnet mask and default gateway you are trying to use?

[confuzzled]

So I was able to get a test done this evening, mainly to see if we could get anything to show up on the local status page. I worked with the other state agency that we hand off to, who also made sure to manually flush the arp table. Unfortunately, no go. We were able to get a DNS error to go away, though. @PhilipDAth, in this case they are both private and non-routable (this is per the other IT guy).

1st, the local status page:

https://1drv.ms/i/s!AnLjK1XRrLhQgbMtdr7NXy-In2cO5A

So for grins, on the MX I'm using 

10.x.x.18

255.255.255.248

Default Gateway 10.x.x.17

(Default Gateway is also the routers IP)

Hope that clears that up.

[PhilipDAth]

This is suggesting the appliance has not got an Ethernet connection.  Have you definitely got a link light?  Have you tried a cross over cable?

[confuzzled]

Good morning @PhilipDAth,

The MX definitely has a link light, traffic too (or at least trying) - green and blinking. I have not tried a crossover cable though. I thought on Meraki's we did away with the whole crossover thing?

[PhilipDAth]

Could you post the outside interface of your ASA, obfuscating the 2nd and 3rd octets?

[confuzzled]

@PhilipDAth

First, thank you for being so helpful on all of this; I know this is entirely optional for you, so I really, really appreciate all the help and enthusiasm you've shown in trying to get this solved.

My current Cisco ASA outside config as you requested:

!
interface GigabitEthernet0/0
description outside
speed 1000
duplex full
nameif outside
security-level 0
ip address 10.x.x.18 255.255.255.248
!

[PhilipDAth]

This is very useful. Did you configure the Meraki WAN port to only use 1000/full (this is on the Ethernet tab of the local status page)?  This speed and duplex configured must exactly match what is configured on the 2900. As the ASA is working, lets assume for the moment it has the correct configuration.

We know the Meraki is reporting a basic Ethernet connectivity issue - so the issue is something close to this area.

I don't think a cross over cable will fix it - but I know we are facing some kind of basic Ethernet connectivity issue so I thought it was worth a try.

[confuzzled]

Hi @PhilipDAth,

I left the config at "auto". I also thought this shouldnt matter (although the thought did cross my mind that it might cause a problem).

I'm still waiting on the external IP information from my cohort over at the other agency for the router. 

[PhilipDAth]

And can you get a copy of the routers interface with the IP address 10.x.x.17 (also with the second and third octet obfuscated)?

[confuzzled]

@PhilipDAth let me see if my contact will provide that and allow me to post it. Standby.

[ww]

Also make sure no (other) firewall upstream blocks the traffic from your IP to the meraki adresses or your specified dns servers "Help -> Firewall".

[Tat0rt0t]

From this whole thread - it sounds like they aren't routing 10.x.x.x /whatever subnet they re using to be allowed internet access and used for internal only. Thats your problem. What is the purpose of the 29xx model router? Can that be replaced and what is its interface configuration to the 3750 switch? 10.x.x.16/29 you are using for the Meraki/ASA is serving as a connection between that device and the router only and advertised out over some routing protocol most likely.

Need to know more about the upstream and what the 29xx router is doing. 

All this is with the acting though of:

Change the Interface facing the MX to a IP space that is allowed internet access and update the 29xx and the MX accordingly, some /29

Advertise that network over X routing protocol.

[confuzzled]

@Tat0rt0t,

Your verbiage is correct. Thats exactly how they describe their 10.x network, internal only and "they aren't routing it".

But if thats the case, then why on earth can the ASA get to the www and the MX can't?!

As for the other items - I'd have to guess, since I don't have access to their devices.

The 3750 has SFP connections,  and the 2900 does not, so I'm guessing they are using it to pass the eth to fiber.

The 2900 cannot be replaced at the moment. Long term, perhaps.

This is also a "beta" setup which will need to be replicated at 6 other sites; so its very important that we figure out the problem and not start replacing equipment to try and force it to work; otherwise it'll get costly mighty quick.

To your last, we can easily request that the .17 be assigned an external facing IP; thereby bypassing the rest of their network. That isnt what I want though, because that opens us up to being completely liable for our connection out to the web, and also adds our IP as an exclusion to their FW. (Ultimately - down the road this means when we get audited, I would have to do all the same paperwork they do).

Thank you for your help!

[Tat0rt0t]

---

@confuzzled wrote:

@Tat0rt0t,

"Your verbiage is correct. Thats exactly how they describe their 10.x network, internal only and "they aren't routing it".

But if thats the case, then why on earth can the ASA get to the www and the MX can't?!"

This is because the ASA doesn't need to access the internet from that network(10.x.x.16/29) where the Meraki does, the unfortunate difference - All the ASA is doing is routing to your devices behind it and that IP space is being advertised out via the 29xx (dont know what your internal space is), I think you said before - The ASA had a 0.0.0.0 0.0.0.0 10.x.x.17(29xx model IP)

Basically the ASA is taking whatever traffic it gets from its specified inside interface and dumping it to the 29xx which is i'm sure advertising out your giving internal IP space. 

[confuzzled]

@Tat0rt0t

Now that I think is exactly what I was looking for. The difference I can't seem to get anyone to tell me "why".

So if that is correct, then I ask this question - "WHY"?

Why? Why does an ASA "pass along" traffic to .17 and the MX says "noooo"? I'm completely not understanding the difference here, besides about two decades worth of technology difference between the Meraki company before Cisco bought it.

The rest of your description sounds almost verbatim what I've heard from the sys admin III type that I'm working with trying to get this to work. So in basic networking terminology; the MX is designed to basically ignore TCP/IP principles, and the ASA isn't?

So the ASA says "aww, screw it" and just throws traffic at .17, and the MX panics and says "I can't see the web! I cant see the web!" and never actually gets over itself?

Btw, I just got the config.

@PhilipDAth

!
interface GigabitEthernet0/1
description Office of the State Engineer f/w outside
ip address 10.x.x.17 255.255.255.252
no ip redirects
no ip proxy-arp
ip verify unicast source reachable-via rx
ip nbar protocol-discovery
duplex full
speed 1000
ntp disable
no cdp enable
no mop enabled
!

[Tat0rt0t]

---

"Now that I think is exactly what I was looking for. The difference I can't seem to get anyone to tell me "why".

So if that is correct, then I ask this question - "WHY"?

Why? Why does an ASA "pass along" traffic to .17 and the MX says "noooo"?"

The answer to your why is, You are giving the Meraki a network 10.x.x.16/29 with IP 10.x.x.18/29 on the WAN port, this is taken as the network the Meraki uses to access the www (Meraki cloud) to get its configuration, confirm it is online and licensed and  once all that is done, it can be on its merry way of working for you. The hang up, 10.x.x..16/29 does not have access to the internet. So your solution is to Request that 10.x.x.16/29 have internet access or use a different subnet that does in fact have access to the internet, something your Sys level 3 admins can help with. Since this is a larger project i think you said, probably a sit down meeting with them do design it out. 

---

"So in basic networking terminology; the MX is designed to basically ignore TCP/IP principles, and the ASA isn't?" - Negative - this is not the case. 

[confuzzled]

@Tat0rt0t -

I hear what you're saying, I'm still not completely clear on it  though. The way they've got their  network setup, I won't be able to request the .16 network to have internet access, it would require too much reconfiguring on their part, plus they're happy with the way things are setup. My options at this point are to figure out if this can work "drop in place", or if I need to have .17 bound to an external IP, which we already know works, but bypasses the other agency firewall.

[Tat0rt0t]

@confuzzled

Im going to draw up a network diagram of what i expect the rest of your network looks like, i think that will help us figure it out but a thought on a quick a dirty work around, hook up a Cellular connection to it with a ethernet converter, something like . a cradle point ( https://cradlepoint.com/products/cor-ibr350 )  - This will allow you to get connected to the Meraki cloud and then just under traffic shaping, send all your network traffic to your other WAN connection.

Standby on the network map. 

[Tat0rt0t]

@confuzzled Ok so this drawing is with alot of assumptions BUT the general jist I want to show is, 10.x.x.0/8 is being used for all internal connectivity only and from a security standpoint - wouldn't be allowed out to the internet, mostly because it wouldn't need to be.

The hang up with the Meraki is, Meraki needs to be able to access the internet or it will not work. Normally with a MX this is never an issue since its usually the device directly connected to the internet, however, in your environment, your "internet" in the eyes of the MX since you are configuring the WAN interface is actually a larger LAN segment and on top of that, that network space isn't allowed to the internet.

Why can your computers in your offices network access the internet, because they are a separate network being advertised  and routed via the 29xx

This is probably achieved via sub interfaces on both the ASA and the 29xx - information here - https://www.networkstraining.com/how-to-configure-vlan-subinterfaces-cisco-asa-5500-firewall/

[confuzzled]

I get what you're saying. but all our traffic for this site is going through that ASA. The 2900, as far as I know is only routing traffic from the ASA "outside" interface, the .18. I'd have to double check the config, but I'm pretty sure thats how its working.

I'm guessing I'll just have to pursue the public IP route. Its probably for the best anyway; since we have outages all the time, and I cant be sure if they are from the rest of the state LAN or the ISP. This will at least clear that up and give us some insight as to who to hold accountable.

Thank you very much for all your help!

[PhilipDAth]

Note the network mask you are using on your ASA is wrong as well.  It should be 255.255.255.252 to match the router.

However, this is not the cause of your issues with the MX not working.

[PhilipDAth]

From the ASA can you do a ping to 8.8.8.8 and does that work?

Does the upstream firewall have any restrictions on what the ASA IP address can access?

[confuzzled]

@PhilipDAth

No restrictions on outbound traffic per the agency.

Ping result to 8.8.8.8 is a negative ghostrider, both from the CLI and ASDM.

[PhilipDAth]

I think I know what is going on. Can you supply your Nat configuration from the ASA please obfuscating the second and third octets.

I think your outbound requests are getting nat'd to address space outside of the link subnet.

[confuzzled]

See, I've always been a little confused on this one. I've got plenty of lines of "route inside this" which, listed in ASDM are static routes. As far as NAT goes, however I have yet to see / find anything that specifically says what the config is for that on the ASA. My last attempt was about two hours of googlng and research and still came up empty handed and confused (could you tell from the screen name?). If you can steer me in the direction of what in the output of a sh run is considered 'NAT', or alternate commands to use / what to look at in ASDM, I'm all ears.

[PhilipDAth]

Look for lines in the ASA config that have the word "nat" in them.

I think if you ask the service provider that there will also be a line on the 2900 routing a public IP (or maybe even a block of public IPs) via the Gigabit interface in the router facing towards you, or via the IP address assigned to the ASA.  I think that the ASA will be NATing your outbound web browsing to this IP address.

[confuzzled]

Ha! I thought so. So, if thats the case, then we have a "zero nat" configured ASA. Not even kidding.

And yes, the option to tie the .18 into a public IP has been there. I've been trying to get this to work "natively" because I'm pessimistic that come some kind of auditing time, they're going to say "well such and such agency (mine) does their own firewalls and security" - and then I'll be subject to the same level of reports, auditing, etc that they are.

As it stands now, since we're basically a LAN segment, I can push a lot of the "blame back" so to speak.

Looks like the public IP route might be the only option though.

Thank you so very much for your help.

[PhilipDAth]

If there is no NAT configuration then the ASA must be routing your subnet block behind the ASA, and the upstream must be NATing that for you.

Regretfully, it seems that you wont be able to simply transfer the IP address from the ASA to the MX and have it work, since that IP address has no access to the Internet.  You will need your service provider to provide you with a subnet that can be applied to the outside of the MX that they will NAT for you, or a directl public IP address.

Sorry I couldn't resolve this one for you.

[confuzzled]

No need to apologize @PhilipDAth, I think you're right. To add further complexity to this mess I inherited just over a year ago; the "idjut" that set it up internally matched our internal schema to the external WAN IP ranges, with the 10.x in the middle. example, if we had a public website and it was 164.x.x.5, we have 20 different ranged beginning with 164.x.x.x.

So 164.x.x.x > 10.x.x.x > 164.x.x.x.

Talk about a migraine. 

But now you know why I'm trying to replace these haha. But again thank you, this gave me several very good pieces of information, and I now can see that the assigning the public IP route is going to not only be way faster, but much much easier than trying to force it to work the other way.

@Tat0rt0t Thank you as well for all of your considerable expertise, and also your willingness to break things down (including a demo map!) you guys rock. You've sold me that the public IP route is the best way to go on this.

[Netify]

Likely no longer an issue, and just seeing this thread - but a couple of thoughts if others face a similar issue:

The key thing to remember is that the MX will operate as a router / firewall just fine without cloud access ONCE CONFIGURED - your primary issue is/was just that you needed to get the initial configuration on it.

Of course, the long term solution must include getting access to the internet for the MX - but there's 'then' and there's 'now'.

@Tat0rt0t nailed what I was going to suggest, but I'll throw in an alternate angle:

- Take the MX84 home one night if you can

- Plug it into your internet connection (behind your home firewall, let it pull a DHCP address that will be NAT'ed to the internet)

- Configure the 84 with basic connectivity / static IPs you intend to use, etc. (you would need to configure your home router with a custom NAT which would allow it to pass traffic (may have to call it a 'DMZ Host') to that internal/static IP (your intended 10.x.x.18 you plan to take from the ASA);
** This gets the device configurable / configured as you will be deploying it **

- Identify if the ASA is running a dynamic routing protocol (OSPF or BGP peering, etc.) that you will need to emulate in the MX config, and build that config as necessary in the MX (to talk to the 29xx router - though I think you said ASA was just dumping to default gateway)

- After that, the MX should be configured exactly as it would need to be for your final installation

- Take device to the office, drop out the ASA in your outage window, plug in the MX, and everything should behave as required, assuming the firewall rules were properly configured, etc.

- THEN... plug in a USB or Cellular to ethernet transceiver to act as your 'out of band' cloud management link, while the MX sends all actual traffic out the GigE port

You could reverse this process / eliminate the 'take home' phase if you did the transceiver first and used it from the start. But this is an alternative to getting things moving.

Cheers.