Brass Tacks

FCoE Multihop and VE_Ports

In the April EMC Support Matrix, EMC will be posting support for NX-OS 5.0(2)N2(1) for the Nexus 5010, 5020 and 5548.  In addition, we will also now support the creation of Virtual E_Ports (VE_Ports) on these products.  This post discusses what VE_Ports are, the protocol used to instantiate them and the exact configuration steps required to create them.

VE_Port overview 

VE_Ports allow for the formation of FCoE ISLs and the creation of an all FCoE fabric or more precisely a Multihop FCoE fabric.  As you will see, once a link between two FCFs has been established using FIP, VE_Ports / FCoE ISLs are initialized using the same protocol that is used to initialize E_Ports / native FC ISLs.  Just in case you are interested, the FIP protocol used to establish FCF to FCF connections is described in my previous post, “FIP, FIP Snooping Bridges, and FCFs (Part 1 – “FIP”, the FCoE Initialization Protocol)”. 

From a logical connectivity point of view, an all FCoE fabric is identical to an FC Fabric and it supports all of the same functionality such as zoning, a distributed name server as well as support for RSCNs.  As a result, the same types of scalability limits apply to both FC and FCoE Fabrics such as the maximum number of; hops, VN_Ports and Domains.  The FC scalability limits have been posted in the EMC Support Matrix for years and are currently no more than 5 hops, no more than 6000 N_Ports and no more than 55 Domains.  With an all FCoE fabric, I feel comfortable with 3 hops (because I’ve tested it), and have no reason to believe that the number of VN_Ports or Domains will need to be limited to less than what is supported in native FC.

From a physical connectivity point of view, the connectivity options currently supported for an all FCoE Fabric are shown in the following diagram.  Each of the connectivity options are explained in detail below the diagram.

  

For the sake of clarity, the diagram shows the physical connectivity options that can be used within a row of equipment racks.  The diagram is not intended to indicate a limitation in the types of topologies that are supported; rather it is just intended to help highlight all of the different possibilities.  Each Rack is described in detail below: 

End of Row– The End of Row (EoR) cabinet contains the aggregation layer switches.  In order to maintain the highly available characteristics of FC, two 5548s are shown and they have not been connected together via FCoE ISLs.  This allows for the two fabrics to remain logically isolated.  The 5548s may be part of the same VPC Domain. 

The 5548s in the End of Row cabinet will support:

1. VE_Port (FCoE ISL) connections from other Nexus 5000s,
2. VF_Port (ENode) connections from FCoE initiators and targets,
3. native FC E_Ports (ISLs) to other FC capable Cisco switches
4. F_Port (FC initiator and target) connections from FC initiators and targets.
5. N_Port connections to FC capable Cisco/Brocade/QLogic FC switches
6. uplinks from an FSB, and
7. Fabric Ports to connect to the 2232 FEX module. 

Rack 7 & 8 Storage– Storage ports (either FC or FCoE) can be connected to a Top of Rack (ToR) switch or directly back to the 5548 in the End of Row rack (EoR).  Connecting a storage port directly to the top of rack switch in a given rack makes sense if the storage port is accessed primarily by servers residing in that rack.  If a storage port is going to be accessed by many servers in different racks, the optimal placement of the storage ports may be on the 5548 in the EoR rack. 

Rack 5 & 6 FIP Snooping Bridge (FSB)– FIP Snooping Bridges such as the Cisco/IBM 4001i can either be connected to the Nexus 5000 at the top of the rack or connected to the Nexus 5548 at the EoR. 

Rack 4 Nexus 5000 (VE_Ports)– The Nexus 5000 can be connected to the 5548 in the EoR via VE_Ports while running in FC-SW mode over FCoE or over FC while running NPV and FC-SW modes.  Currently you cannot utilize FCoE for uplinks to the 5548 at the EoR while running in NPV mode.

Rack 2 & 3 Nexus 5000 (VE_Ports)/Nexus 2232 (Fabric Ports)– The Nexus 2232 can be connected to a Nexus 5000 via Fabric ports and then the Nexus 5000 can be connected to the Nexus 5548 at the EoR via E_Ports. 

Rack 1 Nexus 2232 (Fabric ports) - The Nexus 2232 at the ToR can be connected to the Nexus 5548 at the EoR via Fabric ports. 

A logical representation of the physical topology is shown below. 

  

Although it isn’t shown in the logical topology above, the FCoE ISLs must be on physically separate links rather than trunked on the ethernet uplinks used to carry non-FCoE traffic.  This is not a requirement from an FCoE protocol perspective but it is required when using the current version of NX-OS.  This requirement may eventually be removed. 

An important point to note in the above diagram is that from an FC logical point of view, a Fabric A / Fabric B topology has been maintained.  If the Customer desires, the ToR switches could be connected together to allow for the use of vPC and active/active NIC Teaming /Bonding.  Although this breaks the “Air gap” separation requirement (more on this later), the FC portion of the network will remain logically isolated. 

FCoE VE_Port Virtual Link Instantiation

Note: This section describes the FCoE VE_Port Virtual Link Instantiation process currently being used between Cisco Nexus products.  The reason for this is that Cisco is the only FCF vendor currently providing this functionality.  When other vendors provide VE_Port functionality, this section will be updated to show the differences (if any).

The Virtual Links used to support the instantiation of Virtual E_Ports (VE_Ports) are created in a manner that is similar to the process used to instantiate a virtual link that supports the instantiation of a Virtual F_Port (VF_Port) as described in my previous post  “FIP, FIP Snooping Bridges, and FCFs (Part 1 – “FIP”, the FCoE Initialization Protocol)”. One major difference is that FIP VLAN Discovery is not used.  For the sake of this example, the following topology will be used.

 
 

FIP Discovery Solicitation

The process starts with both sides of the link transmitting DCBX frames.  Once the DCBX parameters have been exchanged, both sides will transmit Solicitations on every VLAN that FCoE has been allowed on.  In this example, we will assume that the Ethernet / vFC interface on the Nexus has been configured to allow all VLANs / VSANs.  In the diagram below, only the details for the Solicitation frames for VLAN/VSAN 100 are shown.  The Solicitations for VLAN/VSAN 200 and 300 would contain similar information.

  

The Solicitations are transmitted to the multicast Destination Address (DA) of ALL-FCF-MACs.  The SA of these frames is the Chassis MAC of the Nexus.  The 802.1Q tag will contain the VLAN that the Solicitation is being performed on.

The Available for ELP bit will need to be set to one in order for FIP to proceed to the next phase (FIP Advertisement).

The FCF bit indicates that the Frame was transmitted by an FCF. 

The FC-MAP is checked by both sides to ensure that this value matches.  If it does not, FIP will not proceed to the next phase and the virtual link will not be instantiated.  The FC-MAP value prevents unintentional FC Fabric merges and should be administratively set to a value other than the default if you have multiple FCoE Fabrics in the same data center and you do not want an accidental connection between two FCFs to result in a fabric merge. 

The Max FCoE Size field is set to the maximum size FCoE Frame supported by each side.

FCoE Discovery Advertisement

If the Available for ELP bit and FC-MAP values match on both sides of the link, the FIP process will continue with both sides transmitting a Discovery Advertisement. 

 
The information contained within the FIP Discovery Advertisement is similar to what is contained in the Solicitation.  One difference worth mentioning is that the Advertisement is padded so that the Frame is equal to Max FCoE size.  The presence of this padding allows both ends to validate that the network infrastructure between the two VE_Ports is capable of supporting mini Jumbo frames.

Once both sides have received and validated the information contained within the Advertisements, the Virtual link can be instantiated by both sides transmitting ELP on each VLAN/VSAN that supports FCoE.  The frames exchanged to complete the VE_Port instantiation are practically identical to those frames used to initialize an FC E_Port and will not be repeated here.  For more information, refer to the Networked Storage Concepts and Protocols TechBook for additional information.   

Configuring Cisco VE_Ports

VE_Ports allow for two Cisco Nexus 5000 products to be connected together and merged into the same fabric using FCoE links.  VE_Ports can be created as either individual links or as shown in the following example configured to use a Port channel.  The topology used in this example is shown below. 

 

Setup steps:

On Nexus 5000 A:

1.  Create the appropriate VSANs

NEXUS5000A# config t

Enter configuration commands, one per line.  End with CNTL/Z.

NEXUS5000A(config)# vsan database

NEXUS5000A(config-vsan-db)# vsan 200

NEXUS5000A(config-vsan-db)# vsan 300

NEXUS5000A(config-vsan-db)# vsan 400

NEXUS5000A(config-vsan-db)# exit

NEXUS5000A(config)#

2.  Create the appropriate VLANs and associate them with the appropriate VSAN.

NEXUS5000A(config)# vlan 200

NEXUS5000A(config-vlan)# fcoe vsan 200

NEXUS5000A(config-vlan)# vlan 300

NEXUS5000A(config-vlan)# fcoe vsan 300

NEXUS5000A(config-vlan)# vlan 400

NEXUS5000A(config-vlan)# fcoe vsan 400

NEXUS5000A(config-vlan)# exit

NEXUS5000A(config)#

3.  Create the port-channel that will be used to connect the two switches.

NEXUS5000A(config)# interface port-channel 777

NEXUS5000A(config-if)#switchport mode trunk

NEXUS5000A(config-if)# exit

NEXUS5000A(config)#

4.  Create the vfc interface that will be associated with the port-channel, specify it as an E_Port and allow the appropriate VSANs.

NEXUS5000A(config)# interface vfc777

NEXUS5000A(config-if)# bind interface port-channel777

NEXUS5000A(config-if)# switchport mode E

NEXUS5000A(config-if)# no shutdown

NEXUS5000A(config-if)# exit

NEXUS5000A(config)#

5.  Add the ethernet interfaces to the port-channel

NEXUS5000A(config-if)# int e1/1-2

NEXUS5000A(config-if-range)# switchport mode trunk

NEXUS5000A(config-if-range)# channel-group 777 mode active

NEXUS5000A(config-if-range)# exit

NEXUS5000A(config)#

6.  Repeat steps 1 – 5 on Nexus 5000 B.

7.  At this point the VE_Port port-channel should initialize and can be viewed using the show fcoe database command.

NEXUS5000A(config)# show fcoe database

-------------------------------------------------------------------------------

INTERFACE       FCID            PORT NAME               MAC ADDRESS

-------------------------------------------------------------------------------

…

VE Ports:

-------------------------------------------------------------------------------

INTERFACE       MAC ADDRESS             VSAN

-------------------------------------------------------------------------------

vfc777                 00:05:73:af:13:68               200

vfc777                 00:05:73:af:13:68               300

vfc777                 00:05:73:af:13:68               400

NEXUS5000A(config)#

Afterword

Last Sunday, while I was in the process of writing the technical content of this post, a friend of mine from Cisco sent me an asynchronous notification that J Metz had just posted a blog regarding FCoE Multihop.  My initial response was “Darn!” (or some variant thereof), followed by “he beat me to it!”  Well, here we are a week later and I’m finally getting around to posting my FCoE multihop post.  For the sake of transparency, let me state that because I wanted to avoid contaminating my thought process, this morning was the first time I actually read through the entirety of J’s post and the comments that follow.  For the most part, I found myself agreeing with J and Brad, but I understand where Brook is coming from and I think he raises some valid points that need to be seriously considered.  One point that I want to stress is this, we don’t care which protocol or topology you use to connect to our storage!  We care very much that whatever protocol or topology you do use, you are able to design, implement and maintain it with as little effort (or drama) as possible.  As long as this is possible we will strive to support whatever protocols and topology our customers are asking for...

Thanks for reading!

Connectivity changes for March 2011

The updates for the month of March are listed below:

Brocade 

Firmware added to the March ESM to support the FCoE 10-24 blade in DCX (Courtesy of Erik Paine and Aditya Nadkarni)

• FOS 6.4.1_FCoE – This version adds support for non-disruptive CP failover and is capable of supporting connections from both FCoE initiators and targets.

Hardware added to the March ESM (Courtesy of Erik Paine and Aditya Nadkarni)

• FCoE 10-24 blade

Cisco MDS

Firmware added to the March ESM (Courtesy of Rich Hultman)

• Cisco NX-OS 4.2.7d

Cisco UCS (Courtesy of Shreedhan Nikam)

Feature support added:

• F-port trunking and channeling in NPV mode

Thanks for reading!

PT2PT and VN2VN - Directly connecting an FCoE initiator to an FCoE target (Part 1 – Overview)

Recently, I received a few questions asking “how do you connect an FCoE initiator directly to an FCoE target without using an FCF?”  The short answer is, “with the FC-BB-5 version of FCoE you can’t!”  The reason is, this functionality was intentionally deferred to FC-BB-6. 

To help clarify what is, and what is not supported today, I will describe the basic connectivity options supported by FC-BB-5 and then describe the basic features of the PT2PT and VN2VN protocols being worked on in FC-BB-6.  In the near future, I will provide a part 2 and 3 to this post that will explain the PT2PT and VN2VN modes of operation in more detail.  

FCoE connectivity options

The relationship between the connectivity options supported in FC and those supported in FC-BB-5 and FC-BB-6 are shown in the following diagram: 

 
Figure 1

As shown, Fibre Channel Switched Fabric (FC-SW) maps directly onto FC-BB-5.  When I say FC-SW, I’m referring to environments that utilize an FC switch for connectivity, as opposed to directly connecting an HBA to a storage port or utilizing a hub for the same purpose.  The reason FC-BB-5 tackled this particular connectivity option first is that the overwhelming majority of data centers utilize FC-SW.

Fibre Channel Arbitrated Loop (FC-AL) maps somewhat more loosely to the proposed VN2VN protocol being developed in FC-BB-6.  VN2VN and FC-AL are similar in that they both allow for many devices to connect to one another (up to 127 with FC-AL and at least that many with VN2VN) and have similar issues with sharing a common address space.  As you will see, special care had to be taken when the protocol was being developed to prevent 24-bit address conflicts. 

Fibre Channel point-to-point also maps somewhat loosely onto the proposed PT2PT protocol.  PT2PT and FC point-to-point are similar in that they both expect only one device to be present at the other end of the link and bad things will happen if this turns out not to be the case. 

FC-BB-5 connectivity options

With FC-BB-5, the FCoE initiators and targets need to be connected to an FCF either directly (figure 2) or through a FIP Snooping Bridge (figure 3).    

 

Figure 2

 

Figure 3

As I described in my previous post “FIP, FIP Snooping Bridges, and FCFs (Part 1 – “FIP”, the FCoE Initialization Protocol)”, the FIP login process relies on the use of FIP frames and on the fact that an entity exists somewhere in the network that is listening for frames destined to the Destination Address (DA) of ALL-FCF-MACs.  If there is no such entity, FIP is unable to determine which VLAN FCoE is supported on and is also unable to determine the DA that the FIP FLOGI should be transmitted to.  In addition, in both of the cases, it is important to note that all FCoE frames are transmitted with a DA of the FCF-MAC that they logged in with.  The point is, without an FCF in the network, the login process would not complete successfully and FCoE frames will have no valid destination that they can be transmitted to.  As a result, with FC-BB-5, the following topologies cannot be supported:

  

Figure 4

  

Figure 5

As you may have noticed, in the previous topology diagrams I’ve included a “DCB Cloud”.  This probably didn’t look too strange until the diagram showing the FCoE initiator and target directly connected by a physical cable.  The reason I am including the DCB cloud has to do with the fact that with FCoE, a multi-access link is assumed and this had an enormous impact on the development of the PT2PT and VN2VN protocols.

For the sake of completeness, let me also point out that FCF to FCF connections (VE_Port to VE_Port) are supported with FC-BB-5.  This allows for the creation of FCoE ISLs and the creation of an all FCoE Fabric should you want to create one.   

FC-BB-6 connectivity options

FC-BB-6 will support all of the FC-BB-5 connectivity options (shown above) as well as some new protocols and a new network element that will allow for much more flexibility.  The new protocols, PT2PT and VN2VN, will be described in detail in the next posts (part 2 and 3).  The new network element, referred to as an FDF (FCoE Data Forwarder), will also be described in detail in a future post (once it is finalized).  For now, just think of an FDF as an enhanced FSB or a simplified FCF. 

PT2PT

The FCoE PT2PT (Point to Point) protocol is useful if you want to connect two (and ONLY two) FCoE ENodes directly together.  The ENodes can either be physically connected with a cable or connected through a Lossless Ethernet Bridge (see figures 4 and 5 above). 

When physically connecting two ENodes together, you can either use twinax (copper) or optical fiber.  However, as I mentioned in my previous “optical or twinax” post, the downside to using copper is the need to verify that both ENodes support the same twinax cable type.  If you use an optical fiber, it is more expensive but you will not have to worry about physical layer interoperability.

Using a Lossless Ethernet bridge to connect two PT2PT ENodes together may prove to be useful if you want to use twinax, but the ENodes do not support the same twinax cable type.  For example, ENode A only supports active twinax and ENode B only supports optical.  A more realistic example would be that ENode A checks to make sure that a particular brand of twinax is in use and ENode B doesn’t support that cable type.  When using PT2PT over a Lossless Ethernet Bridge, you are going to need to ensure that either there are only two FCoE PT2PT ENodes in that broadcast domain, or create a VLAN and put only the two FCoE PT2PT ENodes that should be connected into it.  In terms of supported L2 Ethernet topologies, PT2PT will operate just like any other Ethernet-based protocol and will not be limited to connectivity between ports on the same bridge.  It has been designed to support connectivity between any two ports in a L2 segment and can operate independently of the type of Link Aggregation in use.

One of the strengths of the PT2PT protocol is that it allows for the PT2PT link to initialize very quickly (< 2 seconds).  This is of particular importance to mainframe environments and, in fact, a major manufacturer of mainframe products was the driving force behind the creation of a PT2PT type of protocol.    

PT2PT limitations

In terms of drawbacks to the PT2PT protocol, since PT2PT is a peer-to-peer type of protocol and may be used when an FCF is not present, the protocol was designed to operate independently of the FC Fabric Services.  As a result, there will be no protection provided by FC zoning and no notification via RSCN if the other device should disappear or re-appear for some reason.  Instead, the ENodes will need to rely on FIP Link Keep Alive (LKA) Frames to ensure that the other ENode is still present and Advertisements to discover when another PT2PT is added to the network.

VN2VN

The FCoE VN2VN (VN_Port to VN_Port) protocol allows an arbitrary number of VN_Ports to communicate with one another in a peer-to-peer fashion.  As with PT2PT, the ENodes can either be physically connected together with a cable or through a Lossless Ethernet Bridge (see figures 4 and 5 above).  Also, since VN2VN allows for an arbitrary number of VN_Ports, the topology shown in figure 6 is also allowed. 

 

Figure 6

One of the strengths of the VN2VN protocol is that it allows multiple VN_Ports to communicate with one another without the presence of an FCF.  As with PT2PT, VN2VN will operate just like any other Ethernet-based protocol and will not be limited to connectivity between ports on the same bridge.  It has been designed to support connectivity between any two ports in a L2 segment and operates independently of the type of Link Aggregation in use.

VN2VN limitations

As with PT2PT, since VN2VN is a peer-to-peer type of protocol and may be used when an FCF is not present, the protocol was designed to operate independently of the FC Fabric Services.  As a result, there will be no protection provided by FC zoning and no notification via RSCN if the other device should disappear or re-appear for some reason.  Instead, the ENodes will need to rely on FIP Link Keep Alive (LKA) Frames to ensure that the other ENode is still present and Advertisements to discover when another PT2PT is added to the network.

In addition, the VN2VN protocol places some additional requirements on the VN_Ports.  The biggest of these is the requirement that each VN_Port keep track of all VN2VN ENodes that are on the same network segment.  This effectively means that each ENode will need to maintain a local copy of a Name Server.  As I’ll show in part 3 of this post, there are very good reasons for why this was necessary, but it may prove to be a rather heavy burden for ENode implementations to bear.

One other point worth mentioning about VN2VN is this: a corner case with a network join scenario could result in duplicate addresses (FPMAs) for a brief period of time.  The solution to this will be to implement some form of dynamic ACLs.  As a result, when E-Lab eventually qualifies a VN2VN solution, we will probably require the use of an FSB or impose a default ACL requirement.  I’ll provide more detail in Part 3.

With all of this in mind, even though I was a proponent of the PT2PT protocol and presented my own version of the PT2PT protocol in FC-BB-6, my personal favorite is now VN2VN.  It is much more flexible than PT2PT and will make our customers’ lives easier. 

Thanks for reading!

Conversation topics for EMC World 2011

Since I end up spending much of my time in the lab, you probably won’t be surprised to learn that I consider EMC World to be one of the high points of the work year…Setting aside the much-needed (and appreciated) opportunity for social interaction, I find it extremely helpful to speak with end users about new technology and gain insight into how well we are meeting their needs.  Based on what I hear, I frequently find myself making changes to my testing methodology or trying a new topology in an effort to better serve the people who keep the lights on. 

Over the past 5 years, I’ve found that some of the best suggestions originate in the “Birds of a Feather” (or BoF) sessions.  For those of you who have never attended a BoF session, it’s a somewhat structured opportunity for EMC to raise points of discussion and for our customers to provide their thoughts.  For example, last year during the “Future of Storage Networking” BoF, I presented the work E-Lab had done with FCoE and FC-SCM.  I found the feedback I received on FC-SCM to be especially helpful.  In fact, the comments caused me to reconsider some of the work that I was involved with at T11 and push for changes in the FC standard to better support our customers’ needs for the long term.  The work to address these long term goals is still ongoing and is now being tracked under the name of Target Driven Zoning or “TDZ” (much more will be provided on this topic in a later post).

In preparation for EMC World 2011, we were discussing the “Future of Storage Networking” BoF session, and one of my colleagues suggested asking all of you for your thoughts on the proposed discussion topics.  So far, we’re thinking about including the following:

1. Realistic topologies and the separation of storage and IP traffic. -  We're hearing from many different sources that storage and IP traffic should be separate (at least for now).  By April, this discussion may start to heat up as the need for dedicated links for VE_Ports becomes more than just a theoretical discussion.  All of this leads me to the question, what FCoE multi-hop topologies are you interested in?  Is it FIP Snooping Bridges at the Top of the Rack (ToR) and FCFs at the End of the Row (EoR) or FCFs in both the ToR and EoR?  
2. Active / passive twinax and optical connections - There's been a bit of discussion about this topic and it may be instructive to let our customers voice their opinions on their connectivity preference and pain points.
3. FCoE TechBook - An opportunity for end users to suggest improvements to the TechBook.  We’ve been working on the FCoE TechBook for almost three years now, trying to include concrete examples and case studies, and would like to make sure we are publishing the most useful information possible.  Any suggestions to improve the guide, we would really appreciate hearing.
4. Why are you using FCoE vs. iSCSI vs. NAS?  What are some of your use cases?
5. Software-based Intel and Broadcom FCoE initiators -  What are you expecting relative to your experience with HBAs in terms of set-up, performance, PowerPath, MPIO, OS, etc?
6. FCoE and LAN - We’ve had a couple of data center customers say they have no interest in FCoE because the LAN in the data center has nowhere near the availability of their SAN, and until it does, they can't see betting their SAN traffic on Ethernet.  Is this a common perception?
7. 16 GB FC - EMC is committed to providing 16 GB FC. What are your plans for adoption in the next year / 5 years? 
8. TRILL – We’re hearing TRILL is probably going to be interoperable at its edges only. Based on how widespread end-to-end from single networking vendors tends to be, will interoperability inside a TRILL cloud influence your buying decision?
9. Target Driven Zoning - Last year I introduced FC-SCM and this year I'd like to introduce TDZ, a means to facilitate the network's automation of the zoning process for FC and FCoE, as a topic for discussion. What are your thoughts?

Which of these topics are most important to you?

Are there any other topics that you would like to have to opportunity to provide feedback on?

Thanks for reading!