QSFP28/QSFP+ stacking with off-brand cables. Who has done it? What did you use?

[DR1]

Hello,

We are trying to validate an installation scenario that involves stacking  three MS355-24X2 switches that are spread between three physical racks.

The distance between the racks makes it impossible to use the Meraki branded stacking cables.

We have managed to get things 'sort of working' with QSFP+ DAC cables to create a 40GB stacking loop, but have CRC errors during highbandwidth transfers and other anomalous issues.

Since Meraki support and its official stance on this makes them effectively useless and any client, I've come here.

Has anyone needed to do Meraki stacking with off-brand DAC/AOC or transceivers? What did you use?

Has anyone managed to find out what the required coding is for transceivers to work optimally on the QSFP+ and/or QSFP28 stacking ports on any Meraki series switches?

[Jerome_EVAGroup · Accepted solution]

Hi @DR1 

I had the case last year, we set up a remote stacking between MS425 using MA-QSFP-40G-CSR4 and MPO12

Some former topics say that this is not supported by Meraki but here the details provided by my Cisco Solution engineer implying it is supported as their DAC / Stack cables are standard.

SR4 and CSR4 are both MPO interfaces. 

My recommendation would be that you will be ok with the MPO12

The document shows this but 
Channel Pairs (Ethernet link Tx and Rx):
1 and 12 (Tx1 and Rx1)
2 and 11 (Tx2 and Rx2)
3 and 10 (Tx3 and Rx3)
4 and 9 (Tx4 and Rx4)
Channels 5, 6, 7 and 8 are unused

I drafted quick specs after ends based on this first experience

To note, I didn't follow up this project from start to end but they follow my initial recommendations and it ends up working fine.

Cisco Spec slide about their QSFP-40G-SR

I hope it helps. I'm pretty sure the same could be done with QSFP28. 

A similar discussion from last week

https://community.meraki.com/t5/Managed-Services/MS355-long-distance-stacking/m-p/102794#M392

If you have access to a Cisco Solution engineer, you may ask him the fiber breakout specifications for MA-CBL-100G / 100GQSFP28

https://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/data_sheet_...

https://www.cisco.com/c/en/us/products/collateral/interfaces-modules/transceiver-modules/datasheet-c... 

https://meraki.cisco.com/lib/pdf/meraki_datasheet_sfp.pdf

Edit:
To confirm, MS425 Stack ports have been set to "stacking" so technically no different than a stack dedicated port on other series.

[DR1]

@Jerome_EVAGroup 

I'm no longer an MSP and dont have access to the forum post you linked. Would you be willing to send it to me as a message? I've requested forum access but doubt I'll receive it without jumping through the hoop of becoming a partner.

Regarding what you shared:

This is invaluable! Thank you for sharing the details and first hand experience.

We have had success with 40G QSFP+ 3-meter and 5-meter DAC cables(3rd party sourced from fs.com) on the stacking ports of the MS355. The 7-meter DAC failed however, and 100G equivalent DAC's had a significant number of CRC errors.

Despite some CRC errors using the 40G DAC cables (amounting to .00000001% of all packets), we have had a mostly stable loop going on over a month.

During this time there has been an aggregate transfer total around 500Terabytes. Our approximate calculations has the vast majority of that total passing over the stacking cables.

There have been two anomalous events, but we haven't been able to isolate the problem as being the result of the stacking and not something else.

I'll follow your advise and try to get the fiber pin-out from a CSE.

Whatever success or failure we have, I'll post back here.

[DR1]

Has anyone made headway on this?

I've continued to investigate and test but haven't found a QSFP28 transceiver capable of working on the MS255-24X2 stacking ports yet.

[pfdan]

Hey @DR1 ,

We're in a similar situation. 3 x MS225s distributed across three floors and we don't want to use up all the 10G SFP+ ports on the front of the switch as trunk up to the other switches from the one in the server room.

Based on what was suggested in the long post above, I should get away with these

https://www.fs.com/uk/products/36157.html

and this

https://www.fs.com/uk/products/68017.html

I think? Either way, they're cheap enough that it should be an issue to try them out and see what happens.

Did you mange to try anything with QSFP+ over a fibre connection? Unfortunately I don't think I'll be able to offer any experience in the future on you endeavour to use QSFP28 transceivers as we don't have any in our switches.

[PhilipDAth]

Could you use the switches without stacking them?

[NolanHerring]

No real feedback to provide but I will only mention that if your using 3rd party equipment, your essentially 'on your own' with regard to support. Same thing applies when you use 3rd party antenna for example. Any issues you have (like what your running into), won't be able to be troubleshot by Meraki support, mea culpa etc.

Whatever your findings are, if you do have success please feel free to share here 😃

Good luck to you