MX250 and MA-SFP-1GB-TX module in WAN ports

[ronnieshih75]

Once again, I need to resort to the community to solve problems where meraki support is lacking information on.

Story begins:  I have two MX250 units that are supposed to replace two MX100 routers at our headquarter office.  Our two internet service providers provide only single copper handoffs off the network interface units.  And we have an active / warm spare meraki router setup at this location, therefore, external public switches are required.  And the 2 external switches also only support copper ports.  I've been told by both our Cisco reps and Cisco Meraki support that we need to install the MA-SFP-1GB-TX module in the WAN or internet ports then we can use ethernet cables to connect to the external switches for internet connectivity.  Well, I've found that these modules do not admin up by the OS and stay dark as indicated by the ports' status LED. (see picture 1) While they do blink on the external switches side ports, however the ports aren't even arp-ing on the external switches.  Oddly, these modules come up when I move them to a SFP downlink ports meant for switches. (see picture 2).   So one of the meraki support techs I spoke to said that I must select the 1Gbps full duplex setting which I have set via the local status page. (see picture 3)  The modules still do not come up.  Now consulting the SFP data sheet, the MA-SFP-1GB-TX modules are supported on "All MXes", see this doc:  https://documentation.meraki.com/General_Administration/Cross-Platform_Content/SFP_and_Stacking_Acce...

So I'm at a loss.  Then the next thing Meraki support asked me to do is to download the out-of-band support data, which ironically, without getting these routers to the internet to upgrade whatever firmware these routers are on, this feature is not supported in the local status page.  Yes, I have quadruple checked.  So I'm dead in the water here.  These routers aren't traditional Cisco routers where I can just console in to pull files or just upgrade the firmware locally.  These are getting packed and shipped back if no one has an answer here regarding whether these modules work in the WAN ports.

[ww]

What if you plug the utp in one if the normal ports of that switch?

Or does the switch have a Sfp+ port  so you can test with a sfp+dac cable between the mx and switch? 

Or can you connect the mx temporary directly to the isp to see if you can get it to update

[ronnieshih75]

That's what my second picture above shows.  The MA-SFP-1GB-TX module works in a SFP port meant for downlink to switches, but does not work in either one of the WAN ports.

[ww]

So can you plug the isp in the switch. And use a sfp+ port of the switch to connect to the sfp+ wan of the mx to see if you get a link and update the mx.

[ronnieshih75]

My question is how would you route traffic of a router out the internet with it being behind a switch?  The downlink SFP switch ports are in either trunk or access mode with a vlan number, but no default gateway and it won't get a DHCP IP on any port.  The SFP ports are meant to do transport and not route, that's what the WAN ports are for.  So now we are trying to come up with some arcane workaround fiddling with our headquarter's existing network.  

[jtorre]

You can configure that DMZ switch port(ISP cable) w/ dry vlan say 150 then same config w/ the SFP port going to the WAN port of the MX250.

[ronnieshih75]

If you've read my original post.  I do not have any working WAN port on the MX250 due to SFP module issue with MX250 on old firmware.  

[PhilipDAth]

My guess - the MX250 has shipped with firmware too old to support the MA-SFP-1GB-TX.

My guess - you'll need to bring the MX250 online via any Internet connection first, and then it will work.  This means you are probably going to need to plug the MX250 in via a TwinAx cable to a switch somewhere that also has a SFP+ port just long enough for the firmware upgrade to complete.

[ronnieshih75]

Connecting a MX250's SFP+ port to one of our existing MS350 switch's SFP+ port via a twinax cable does absolutely nothing, as I imagined it would be.  I literally JUST TRIED IT.  The MX350 has internet access via its VLAN1 interface.  The physical interconnection between the MS350 switch and the MX250 router serves as only a transport, the MX250 does not obtain any sort of IP address because its SFP or SFP+ ports do not obtain IP addresses.  The only thing that came out of trying this was that the MX250 started serving DHCP IPs off its default 192.168.0.0 network which I got an alert for on our prod network so I promptly disconnected the router.

The only way for these routers to get updates is for me to buy an old Inseego SkyUS-DS 4G modem on ebay, which is supported on all MX routers and plug that into the USB port of the router for it to get firmware update  over 4G.  That's yet another second fail on Meraki's part, they have no updated list of supported USB 4G modems.  

[ww]

Did the wan port get a link or not?

[ronnieshih75]

Absolutely not.  It wouldn't, because the router still cannot get to the internet to update its firmware to support the modules.

[GIdenJoe]

I have customers running the MA-SFP-1GB-TX modules on the WAN ports without issue.
There was a recent version however (16.16) that had issues with connecting to the switches.

[ronnieshih75]

And how did you manage to get the updated firmware on there?  SFP handoff from provider's NID?

[GIdenJoe]

The issue was confined to downlinking to meraki switches.  The issue didn't occur if you connected directly to a provider router.  We had a test store with that issue so we avoided the release until it was fixed before upgrading.

If you right now have two MX devices that don't work then it's tough to argue there is a hardware fault but it could be possible.  However do the usual checks like reseating the TX modules (there were some issues with those modules not fully clicking inside the SFP/SFP+ slot.)

then the usual factory defaults, trying some other modules like SFP modules and link up to a switch with fiber just to get the upgrade going.  It's of course handy if you're a bigger Meraki shop and have some spare parts here and there.  Of course if you have multiple switches they should be double uplinked so you should have the ability to disable an uplink and use those SFP's in a switch and the MX.

[cmr]

@ronnieshih75 I would follow what @GIdenJoe said, we recently found that some 1Gb RJ45 transceivers would not properly plug into Meraki SFP+ ports as the ports were slightly sunk into the fascia.  This caused no issues with fibre transceivers as they are smaller.  We run MX250s with Cisco GLC-T in the WAN ports without issue, but I do remember that some Meraki GLC-T modules did need a firmware update to be recognised (as per @PhilipDAth's comment.

Do you have any non-Meraki GLC-T transceivers or fibre ones?  On another note, I'm sure you know that the DAC will only come up at 10Gb (usually) so you mustn't force the port to 1Gb.

[ronnieshih75]

Discovered that the modules indeed weren't seating in, clicking in.  But this didn't matter though.  We finally got one seated in, clicked in.  First tried two SR modules with single mode LC fiber cable in between, switch side would not light up, MX250's status LED did light up finally.  Perhaps I just need to reboot the MS120-8 external switch but it's a prod switch and I did not want to reboot it during the day.  The two MS120-8 switches have only two 1GB SFP ports so using DAC is out of the question, although I do have DAC cables and did try it and both switch and router side stayed dark.  

Unfortunately, I do not have aftermarket non-Meraki SFP modules just sitting around to play with.

I will be stopping by a branch office to rip off an Inseego SkyUS-DS 4G modem and use it on the MX250's USB port.  This is most likely how it will get to the internet.  This was a solid USB 4G modem supported on all MX routers in the past that we used for tertiary internet failover.

To be continued after Thanksgiving.

[ronnieshih75]

Again, I found info off older forum posts:

https://community.meraki.com/t5/Security-SD-WAN/Mx250-wan-compatibility-changed/m-p/122179

To summarize, MX250s need to run firmware v16.10 and newer to support newer MA-SFP-1GB-TX modules.  And simply to make it work, get the old Cisco GLC-T modules.

Excerpt from the post:

[n0tHiNg]

Hi, its happened to me couple months ago. What I did was bring up the port using SX or LX module first.

No need to configure anything, just make it physically UP. then only it can detect the TX module.

Yeah its a bug I believe..

[ronnieshih75 · Accepted solution]

Received Cisco branded GLC-TE SFP modules yesterday, and guess what?!  They work right out of the box.  I finally got the routers up and firmware upgraded on those modules first.  Then we plugged in the MA-SFP-1GB-TX modules, and now they also finally work after that firmware upgrade.  Lesson learned.  I'm actually ditching the Meraki MA-SFP-1GB-TX modules for Cisco ones.  The Cisco ones also seat properly.

SOLVED.  This was a double fail:  1. Meraki support couldn't tell provide a fix over the phone, nor did the 3 guys I spoke to know about this.  And 2. The hardware is poorly manufactured with fitment issue, plus firmware upgrade required first for the meraki branded modules to work

[cmr]

Glad what we did worked for you, hopefully you can just enjoy the MX simplicity now 👍