Brass Tacks

Interoperability issue between Brocade CNA and Cisco Nexus 5596

The other day one of my colleagues, Mugdha Kulkarni, brought an interoperability issue to my attention involving a Brocade 1010 / 1020 CNA (Windows Server 2008, Driver 2.3.0.2) and a Cisco Nexus 5596 (NX-OS 5.0(3)N1(1c)). 

The problem was the Brocade CNA was unable to login to the Nexus 5596.  Complicating matters was the fact that the problem would not reproduce on a Nexus 5548.  This was of particular interest to me since based on what we’ve observed (and what we’ve been told by Cisco), the 5548 and 5596 should be identical from a protocol interoperability point of view.     

With this in mind, I spent a bit of time in the lab yesterday trying to figure out if the problem was related to a specific Nexus platform or NX-OS revision.  Based on the testing results and after looking at the traces, my working hypothesis is the problem is due to a DCBX version negotiation problem. 

We are actively working on the problem but unfortunately as of 6/30/2011, there is no workaround.

UPDATE:  As of 10/19/2011, there is still no workaround for this problem but we were able to confirm that the issue is being caused by both devices failing to agree on which version of DCBX to use.  As with almost all interoperability issues, the problem is being caused by unique characteristics of both implementations.  As a result, the "fix" for the problem is for either vendor to slightly alter their DCBX negotiation behavior.  Also as is typical for these types of interop issues, both vendors are providing a "fix" so that the issue can be addressed in as many different environments as possible.  With this in mind, I received confirmation from Brocade that this issue will be fixed with driver version 3.0.3.0 (scheduled to be released early in Q1-2012).  I also received confirmation from Cisco that this issue will be fixed in their next major NX-OS version (also scheduled to be released early in Q1-2012).  

Thanks for reading!

Twinax support table (6/28/2011)

The twinax support table has been updated to include Broadcom CNAs and VNX.  Please use the following version as of (6/28/2011).  This information will also be added to the July version of the EMC FCoE Tech Book

 

The model numbers for the EMC twinax cable are included below. 

 

Thanks for reading!

FC/FCoE Connectivity options as of 6/27/2011

About a month ago I received an email from someone asking "When will the FCoE Tech Book be updated to properly reflect EMCs support for native FCoE on our arrays?" 

My initial reaction was "OH!, <golly-gee>" and then the inevitable"Of all the <interesting> pieces of information to <absentmindedly> leave out".   

My actual response was "what format would you like to see the options presented in?"

In response, they provided a link to a blog post on virtualeverything and said “it would be really nice if you could do the same for EMC Storage.” 

After looking at the format of the blog post referenced, I completely agreed and decided to copy it.  I also decided to add Brocade FCoE information as well as a list of configuration rules to hopefully avoid the “why don’t you support x with y” types of questions.  That having been said, if you have any questions, please let me know. 

I intend to update the configuration rules and diagrams on an as needed basis.

Configuration rules

The following configuration rules were used to create the topology diagrams.  If there is a discrepancy between the diagrams and the configuration rules, the configuration rules take precedence:

1. The Nexus 2232 requires a connection to a Nexus 5000 in order to function.
2. The Nexus 2232 does not support native Fibre Channel ports.
3. The Nexus 4001i requires a connection to a Nexus 5000 in order to support FCoE.
4. The Nexus 4001i does not support native Fibre Channel ports.
5. The Nexus 7000 does not support native Fibre Channel ports.
6. The Fibre Channel ports on the Nexus 5000 cannot be used for end device connectivity while the switch is in NPV mode.
7. FC-SW interoperability between Brocade and Cisco is not supported.  When attaching a Brocade switch to a Cisco FCF and vice versa, either NPV mode must be used on the Cisco switch or AG mode must be used on the Brocade switch.
8. The Fibre Channel ports on the Brocade 8000 cannot be used for end device connectivity while the switch is in NPV mode.
9. All host connectivity to the UCS 6100 is via FCoE.
10. The Fibre Channel ports on the UCS 6100 cannot be used for end device connectivity while the switch is in NPV mode.
11. The UCS 6100 does not currently provide an FC zoning interface.  As a result, when the 6100 is running in FC-SW mode, it must be connected to another Cisco switch that is also running in switch mode.
12. Cascading switches that are running in NPV / AG mode is not supported.
13. The Nexus 4001i does not support external VN_Port connections
14. EMC does not currently support FCoE on the Nexus 7000 or MDS.  We expect to release support for this in September.
15. Storage ports cannot be connected to the Nexus 5000 while it is running in NPV mode.
16. The Brocade 8000 and DCX do not support VE_Ports
17. Cisco UCS does not currently support VE_Ports.
18. FCoE NPV uplinks between Cisco and Brocade switches are not supported.
19. VN_Port to VN_Port connections (a host directly connected to storage) without the use of an FCF is not supported with FC-BB-5. 

Supported connectivity options:

The supported connectivity options have been broken down into several categories.  Each category focuses on the connectivity options for a particular connectivity component.  The categories are:

1. Configurations specific to Nexus 5000
2. Configurations specific to Nexus 2000
3. Configurations specific to Nexus 4001i
4. Configurations specific to Nexus 6100
5. Configurations specific to MDS
6. Configurations specific to Brocade 8000 / DCX

Configurations specific to Nexus 5000:

Please note:  The first configuration shown is not specific to the 5000.  It has been included to call your attention to the fact that native FC supports a host being directly connected to storage but FCoE does not.

  

Configurations specific to Nexus 2000:

 

Configurations specific to Nexus 4001i:

 

Configurations specific to 6100:

 

Configurations specific to MDS:

 

Configurations specific to Brocade 8000 / DCX:

 

Thanks for reading!

Enabling FIP Keep Alive on Brocade 8000

I received a question from a colleague in Singapore yesterday asking about the D bit in FIP FKA_ADV_Period Descriptor.  Apparently, a customer was wondering if it could be changed from “one” to "zero" on a Brocade 8000 switch.

A portion of the customer's email is shown below:

The Discovery Advertisement frame received from the switch has the “D” bit set to 1. This seems to be disabling the FIP Keep Alive frames.  We would like to have these Keep Alive frames. What can we do to modify the current behavior?

FYI, according to the standard

http://www.t11.org/ftp/t11/pub/fc/bb-5/09-056v5.pdf Table 41 FIP FKA_ADV_Period descriptor format

7.8.6.3.13 FIP FKA_ADV_Period descriptor

D: the value of the D bit shall be administratively configurable on FCFs. The D bit shall be set to zero unless administratively set to one. VE_Port capable FCF-MACs shall ignore the value of the D bit provided in received Discovery Advertisements. ENode MACs shall process the value of the D bit provided in received Discovery Advertisements.

When the D bit is set to zero, the receiving ENode MAC shall verify periodic reception of Discovery Advertisements and shall transmit periodic ENode FIP Keep Alive and VN_Port FIP Keep Alive frames as specified in 7.8.5.2. The VF_Port capable FCF-MAC shall verify periodic reception of ENode FIP Keep Alive and VN_Port FIP Keep Alive frames as specified in 7.8.5.2.

My response was:

Hi, you can display the FIP configuration settings by using the “fcoe --fipcfgshow” command.

----Example----

Elara_IOP05472_AG:admin> fcoe --fipcfgshow

FIP Unsolicited Advertisement Interval = 8

FIP Keep Alive Timeout Disabled.

========================================

VLAN            fcmap

========================================

1002            0x0efc00 [Default fcmap]

========================================

Elara_IOP05472_AG:admin>

 --------

You can configure the FIP FKA-adv-period (shown above as FIP Keep Alive Timeout Disabled.) using the “fcoe --fipcfg -advinterval <value>" command.

----Example---- 

Elara_IOP05472_AG:admin> fcoe --fipcfg -advintvl 8

FIP Unsolicited Advertisement Interval is set to = 8

Elara_IOP05472_AG:admin> fcoe --fipcfgshow

FIP Unsolicited Advertisement Interval = 8

FIP Keep Alive Timeout Enabled. FKA_ADV_PERIOD = 8

========================================

VLAN            fcmap

========================================

1002            0x0efc00 [Default fcmap]

========================================

Elara_IOP05472_AG:admin>

-------- 

Thanks for reading!