Should I use optical fiber or twinax cable?

Earlier this week I received a question regarding EMC’s recommendations for using twinax cable.  My response is included below, but I need to tip my hat to Mark Lippitt for reviewing, correcting, and adding a couple of different considerations that I overlooked.  One other point worth mentioning is that we are working on an update for the February edition of the EMC FCoE TechBook that will also include the following information.

Question: “Should I use optical or twinax connections when connecting my CNAs to the ToR switch?”

Answer: You can use optical connections for any FCoE link.  The downside is that they are more expensive and use more power.  The upside is that you don't have to worry about distance (up to 300 meters) and, perhaps more importantly, you don't have to worry about which vendor’s cable you use…

Typically, we see twinax (copper) being used between the server and the Top of Rack (ToR) switch.  The upside in this configuration is that the twinax connection is much cheaper than the cost of an optical link.  The downside is that you are limited in distance and there‘s also some cable interoperability issues you’ll need to deal with.

In regards to active versus passive twinax cables, it depends on what you are connecting together:

Note: The following information assumes that either FOS 6.4.1 (or later) or NX-OS 4.2(1)N2(1) (or later) is being used.

• If you are connecting a CNA to a Cisco ToR switch (Nexus) and the cable length is 1, 3, or 5m, you can use a Cisco-supplied 1, 3, or 5m passive twinax cable or a EMC active twinax 1, 3, or 5m cable.
• If you are connecting a CNA to a Cisco ToR switch (Nexus) and the cable length is 7 or 10m, you need to use a Cisco-supplied 7 or 10m active twinax cable.
• If you are connecting a CNA to a Brocade ToR switch (8000) and the cable length is 1, 3, or 5m, you can use a Brocade-supplied 1, 3, or 5m active twinax cable or an EMC-supplied 1, 3, or 5m active twinax cable.

The above assumes that you are using a CNA that supports a given flavor of twinax.  As of right now, the QLogic 815x and Emulex OCe1012, support either active or passive twinax from both Cisco and QLogic.  The Brocade CNAs support active twinax from Brocade or EMC’s active twinax cable. 

If you want to connect a Brocade CNA to a Cisco ToR switch (Nexus), you could use the EMC active twinax cable.  Otherwise, you'll need to use optical since the Cisco switch will log an error if you attempt to use the Brocade cable.

If you want to attach EMC native FCoE storage (CLARiiON) to a Brocade switch using twinax you can use either:

• EMC's 1, 3, or 5m active twinax cable; or
• Brocade’s 1, 3, or 5m active twinax.

If you want to attach EMC native FCoE storage (CLARiiON) to a Cisco switch using twinax:

• You can use EMC's 1, 3, or 5m active twinax cable.
• You CANNOT use Cisco’s passive 1, 3, or 5m passive twinax.
• We are testing Cisco’s 7, 10m active twinax with native FCoE storage, but this is not completed yet.

With regard to compatibility algorithms:

• Cisco will always bring up a cable if it can. If it does not recognize the cable, it will post an error message to the system log, but bring up the cable.
• Brocade and EMC must recognize the cable. If they do not recognize the cable, they will not bring up the link.

Comments

[Andrew Fidel]

This needs to be fixed. The only thing you should need is a standards compliant sfp+ cable. Even in the IT industry this coding to only using special marked up cables is an aboration.

[Danny DeMichele Entrepreneur]

I think the above suggestion is good. And I think I will also prefer this. Thanks for sharing such useful information with me.

[Erik Smith]

Thanks Danny. I'm Glad you find the information useful!

Erik

[Jim Nickel]

Would be super-cool if you would list the EMC part #'s for the 1, 3, and 5m active Twinax cables.

Jim

[Erik Smith]

Hi Jim, good idea, thanks for the suggestion. I'll post the model numbers shortly.

[Paolo]

Hi Erik,
Could I connect a “SFP+ 10Gb Twinax Cable” between two servers, without a “ToR switch”?
Is SFP+ Twinax also a “Crossover Cable”?

The Configuration for example could be:
two UCS C-Series Rack-Mount with Qlogic QLE 8152-CNA 2port 10Gb SFP+ Copper

Thanks much!

Paolo

[Erik Smith]

Hi Paolo, although I've never actually tried it, I believe you can connect an SFP+ twinax cable directly between two servers without using a switch. The cables terminate into SFP+ modules that are standardized to all use a common connector pinout. As a result, they are all effectively "crossover" cables.

Keep in mind that currently FCoE will not function properly in this configuration.

[Allen Smith]

Very resourceful insight! I am currently using optical fiber, now I have seen the big advantages and disadvatages. Thanks for posting this relevant article.

[Erik Smith]

Thanks Allen!