Basic Speed/Duplex functionality of device ports

[707eric]

We currently have multiple MX105's and MS125's. All the 10GB SFP+ optical ports lack the ability to manually set the port to 10GB Full. The only option is auto-negotiate but you can't auto-negotiate optical links. This becomes major issue if network devices other than Meraki are added to the network. Meraki's answer seems to be "Don't use any non-Meraki devices" yet my solution is looking like, "Don't use any Meraki devices"

Does anyone know if there is any places to fix this most basic feature on the 10Gb devices?

[alemabrahao]

On MX, you can access the local device status page and set It manually:

 On switch, you can force 1Gb full duplex.

[alemabrahao]

[707eric]

I don't want to set the port to 1GB. I want set the 10G ports to 10GB full and not 10GB auto-negotiate.

[alemabrahao]

Unfortunately, it is not possible, but you can use it in auto-negotiate. mode with other devices that are not Meraki that will work without any problem.

[alemabrahao]

Or request a new feature for Meraki. 😉

[707eric]

No that's not true because they are optical ports. 10Gb isn't supported on optical ports IEEE standards even says as much.

The very reason why the devices dont work with other network devices is because  they follow IEEE standard and Meraki doesn't.

[alemabrahao]

Well, I have been using It with no Meraki and It has been working well.

But, as I suggested, make a wish for Meraki.

[alemabrahao]

Cisco Meraki switches and security appliances with SFP interfaces allow both Cisco Meraki-branded and 3rd party optics. Our products do not lock out optics that have been detected as non-Meraki parts. However, please note that in most cases only Meraki-branded accessories have been certified to work with Cisco Meraki products. For any exceptions please see the section below labeled: Are other Cisco SFP/SFP+ parts certified to work with Cisco Meraki products?

[alemabrahao]

There are many different standards in the industry for varying distances and fiber types. While Cisco Meraki strives to support the most common IEEE802.3 standards, if you are in need of a standard that Cisco Meraki does not carry a certified option for, you will need to use a non-Meraki part. If you have problems using the uncertified part with a Cisco Meraki product, please contact Cisco Meraki support. Though we do not officially support non-Meraki parts, many are still compatible and Support will do their best to ensure all the requirements are met for checking non-Meraki SFP connectivity and viability. For more information on compatible and supported SFP modules, check our SFP and Stacking Accessories document.

[707eric]

The issue isn't supporting Meraki branded or non-Meraki brands parts. The issue is more basic than that. If I buy a device, be it managed switch or firewall at the prices we are paying for these devices, I should have the fundamental basic capability to set speed and duplex on a port. I should have the ability to set a 10Gb optical port to 10Gb full.

Setting speed/duplex on a port is one of the most basic functions of a managed network device.

[BlakeRichardson]

@707eric  I am using 10Gb SFP+ between our MS425-32 and a Fortigate with no problems.

Are you referring to 10Gb fibre or copper connections? 

[707eric]

10gb copper connections act as expected. They support 10Gb auto-negotiation which falls in line with IEEE standard. 

The issue is Meraki doesn't follow IEEE standard when using fiber. Fiber should be manually set or only support 10g Full Duplex not 10G auto-negotiate. The Cisco devices we use follow the standard and require the port be set manually. The Meraki doesn't allow the port to be set manually.

In fact accessing the local interface of the  devices you wouldn't even know they support 10G. 

[cmr]

@707eric we have MS225s connected to Cisco 3850s using 10Gb short range optics and they auto negotiate to 10Gb full without issue.  We also have MS355Xs connected to Pure flash arrays using 10Gb SR without issue.  These have worked for the last 2-3 years reliably.  I know you want to be able to manually set them, but we haven't found it necessary.

[cmr]

I also checked the IOS config and all we have there on the 10G port is:

[alemabrahao]

That's not true, you must have configured something wrong on the other side.

[707eric]

I've been configuring Cisco devices since back when FE ports were considered fast. It's pretty difficult to screw up setting port speed and duplex on a port.

[alemabrahao]

Me too, and I only needed to set the speed in very specific cases, other than that I always used  duplex auto and never had any issue.

[707eric]

That just mean you don't use much fiber because fiber doesn't support auto-negotiate. 

[cmr]

@everyone I think we are arguing a moot point here.  As fibre only runs at the speed of the optic and in full duplex mode, it doesn't matter if you set it to auto or not as it cannot negotiate anything else.

Let's just agree to disagree 😎

[707eric]

I can agree you are wrong.  A switch Identifying a port is set to auto-negotiate is not the same as the port identifying as not auto-negotiate full.

The point isn't moot either because I can make 10Gb SFP+ work at 1GB if I set both ends statically to 1GB. Problem is Meraki doesn't provide the option to do 10GB Full only 1GB Full.

That's cool the lower end Cisco auto-negotiate. The higher end gear doesn't.

[alemabrahao]

I agree that Meraki is not perfect (no vendors are), but I have tested It with Extreme switches, Cisco Switches, HP Switches, etc... and I had no issues. My question is, Are you having any connection issues, or is the fact that you can't set 10Gb full duplex manually bothering you?

[707eric]

Yes, the reason I am bothered by it is because I have multiple Cisco core with 10GB SFP+ ports.  Using compatible SFP+ on both ends of the link the connections is not stable be the Meraki runs at Auto-negotiate on the optical link and the Cisco runs at Full.

It's been tested with multiple Meraki firewalls, multiple MS switches, and multiple Cisco cores. It would not be an issue if Meraki devices supported something as  basic allowing a person to set all supported speed and duplex on the port.

It bothers me because it would seem like the should focus on getting just the basics working correctly before throwing features at the devices.

[cmr]

@707eric as mentioned earlier in this thread, we link MS to Cisco Catalyst core with 10Gb fibre and set both ends to auto.  It's been working without issues like packet loss etc. for years.  I'd suggest trying that.

Cisco's own documentation suggests only setting speed and duplex where you have issues by not setting it - normally when the other end has fixed speed and duplex.  Where you are in control of both ends auto should be the most reliable option.

[alemabrahao]

@707ericI agree with @cmr 

[707eric]

The only reason I am wanting to set it is because there  issues when it isn't set. I am in control of both ends but the Cisco optical port doesn't allow for being set to auto, it only allows for 10Gb Full with auto-negotiate.  It's common with Cisco because the standard only specifies auto negotiate for twisted pair.

I understand the argument "SFP+" can only run at their designated speed. That being the case, the ports should only be able to do 10Gb Full. I understand auto is easier BUT when you have a device with a 10Gb SFP+ port that doesn't support being set to 10Gb Full BUT does allow you to set it to 1Gb Full. That's renders those arguments moot.

I mean why would they be totally out of compliance and allow the port run at 1Gb Full but not let it it beset to 10Gb like its designed?

[cmr]

@707eric what Cisco switches are you having problems with?  We've identified a number of devices that do work with Meraki switches and 10Gb optics, but no specific models that don't.  Then hopefully someone here will have the same switches and possibly a solution.

[GreenMan]

It's a different IEEE standard for fibre links - being full duplex only is part of the standard for the vast majority of optical standards, so Meraki is compliant.

[707eric]

If that were true, the 10GB fiber links would shows they are running at 10GB full and not 10GB Auto-negotiate and advertising available speeds.

[GreenMan]

As far as I'm aware, with a 10G optic, it runs at 10g, full duplex, regardless;   there's no need to negotiate, because it can't negotiate down to either 1G or half duplex.    Check out the first paragraph here:  https://en.wikipedia.org/wiki/10_Gigabit_Ethernet  

This is one of the reasons why SFP+ is physically compatible with 1G SFPs - if you want a 1000Base-SX link, use a 1000Base-SX SFP.   I'm pretty sure the same applies to optical Gigabit links too

[707eric]

Correct 10Gb auto-negotiate is only supported with twisted pair its not supported on optic

• https://ieee802.org/3/10GBT/public/jan04/lynskey_1_0104.pdf
• https://www.ieee802.org/3/an/public/jul04/lynskey_2_0704.pdf

This also means 10Gb optic ports are not supposed to support 10GB auto-negotiate and only support  10GB Full Duplex. There is a difference between the two.

. 

But Meraki doesn't adhere to that. So if a person uses a device like Cisco Catalyst that does adhere to the standard and won't allow a 10gb optical port be set to auto-negotiate you run into a port speed/duplex mismatch.

The problem here is the Meraki don't adhere to standard like other enterprise gear. 

[cmr]

@707eric see my post above with Catalyst 3850s set to auto on the Ten Gig port and working perfectly. 

[alemabrahao]

I'm pretty sure that you don't need to set It manually.

But ok.

[alemabrahao]

My switch is connected to a Fortinet with a 10GB optical in auto negotiate.

[707eric]

Fortinet doesn't follow the standard. They clear state they support auto-negotiate.

[alemabrahao]

🙄

[alemabrahao]

I don't want to discuss this further, but I'll share another image (it's the last time):

On the other side, we have a Juniper router.

I know the standards and I know it's the most basic feature. But it really doesn't seem like a Meraki issue. I really appreciate it if you share with us when you resolve this issue.

[ww]

How would a mismatch be possible if 10Gb sfp+ can only run 10Gb full duplex regardless

[707eric]

I don't know10GB SFP+ isn't supposed to run at 1GB either but they do. And the SFP+ are 10gb because if I use them to connect two  Meraki's they run at 10Gb.

[707eric]

Except they do negotiate do, at least Meraki's do.

[BlakeRichardson]

I feel like we are flogging a dead horse here, as mentioned 10Gb fibre SFP+ modules only work at 10Gb. If you want to set all of your port speeds manually then maybe Meraki isn't the brand for you. Meraki has never said don't use other brand hardware / mix hardware I have no idea where you read that as it's simply not true.

Why someone would want to waste time manually setting port speeds is beyond me. 

[GreenMan]

You mention that your fibre links are not stable.   I believe speed and duplex is likely to be a red herring here and that other causes for your issues should be looked at.   Can you advise which optical SFPs you are using, at both ends, the specification of the fibre you're using (e.g. multi-mode 50/125u OM3) and the length of the fibre run(s)?

Did you also check that your patch leads match the specification of the structured cabling?

[707eric]

We have already troubleshoot all of this with Meraki support fiber and SFP are all fine. The connection works fine when Meraki's are not part of the equation.

What get's me is in multiple firmware release they acknowledge the issue:

Updated the range of port settings available for MX95 appliances on the device local status page.

Added more speed and duplex configuration options for 2.5Gbps-capable ports on the device local status page.

Yet they say:

Hello, Setting 10G through the Dashboard manually is not currently supported. You can submit this as a feature request using the "Give your feedback" button at the bottom of Dashboard pages. These requests are regularly reviewed by our developers and help shape the future of our products. 

When the issue exist locally and from the dashboard.

[cmr]

@707eric those updates are not regarding 10Gb fibre connections, they are for Cu ports that can negotiate speeds in between 1Gb and 10Gb.

Can you confirm that you have tested with both the Cisco Catalyst and Meraki set to auto negotiate?  i.e. the same as we have been using for years with OM3 LC-LC patch leads and OM3 LC terminated structural fibre. 

[GIdenJoe]

What is all this fuss about?
If you have an SFP+ port and you insert an SFP+ fiber module it runs at 10 Gbps period.  If you insert an 1 Gbps SFP it will run at 1 Gbps.  Unless you are using some weird dual speed SFP+ modules you shouldn't have any issue.

I guess the only issue you could have is with copper SFP's but since Meraki itself only sells 1Gbps variants one would assume 10 Gbps over copper SFP is not officially supported.

[cmr]

@GIdenJoe it appears that @707eric has an unspecified reliability issue with either a Meraki MX105 or an MS125 when connecting to an unspecified Cisco Catalyst switch using a 10Gb unspecified fibre optic over unspecified fibres.  He believes that the ability to manually set 10Gb Full duplex on the SFP+ port in the dashboard will solve the issue, even though many of us here aren't so sure.

@707eric can you please fill in the gaps above so we can help more? 

[GIdenJoe]

Well for one: ports in 1 Gbps are higher are always full duplex.
And secondly the only SFP port types on Cisco switches where you can/have to set speeds is on Nexus switches that run multiple speeds.

There simply is no negotiation between SFP's on both sides.  It just sends the traffic over the TX fiberoptic and receives on the RX fiberoptic.

I would start suspecting there is simply a wrong type of SFP in use in one or both of the switches or the fibercable is not up to spec for the distance/speed.

Both SFP modules at both ends must be of the same standard.  A 10GBASE-SR will not successfully decode data coming from a 10GBASE-BX40 SFP+.  First you need to be sure your fiber is okay and the correct modules are used at both ends.

[cmr]

@GIdenJoe I am in complete agreement with you.  I suspect @707eric has a Nexus switch with perhaps an SFP28 port or optic and that is where the issue lies.  Either that or SR-LR or LRM or OM2 mixed with OM2 or OM4 or somesuch fun and games...

[707eric · Accepted solution]

"Hi Eric,

I just wanted to let you know that we are still investigating this issue, and I will provide you with an update soon. Thank you,"

Nah, Meraki acknowledged the issue and investigating it now.

The gaps were there because those variables were not the issue. No reason to spend time on things that weren't the issue.