Brass Tacks

FC and FCoE versus iSCSI – “Network-centric” versus “End-Node-centric” provisioning

Which is easier to use: FC, FCoE, or iSCSI? 

This is the question we wanted to answer as we set out few months ago to test a series of nine configurations in the lab.  Our approach was to determine the number of provisioning steps required when using each protocol to connect to storage and then compare them.  We felt if one of the protocols was significantly more labor intensive than another, then perhaps this would shed some light on some of the recent trends we had been noticing.  Actually to put a sharper point on it, we wanted to determine why iSCSI seems to be more attractive than FC or FCoE in low end server virtualization environments.  A summary of the data we captured as well as the testing process used have been included below, but they were summed up perfectly by Mark Lippitt, one of EMC’s Distinguished Engineers, when he said something along the lines of:

“FC and FCoE provisioning are Network-centric and iSCSI provisioning is End-Node-centric.”

Before I explain what he meant by this statement, let me start by stating (yet again), we don’t *really* care what protocol you use between your host and storage; we’ll support them all!  We do want to make sure that whatever protocol you have chosen to use is the best fit for your environment and it provides you with easy and error free operation.  Getting to the bottom of why iSCSI is successful in certain circumstances is important to us because it represents a change.  

The testing results are shown the following table and I believe clearly show:

1. Native FC requires the least number of configuration steps when compared with both iSCSI and FCoE. 
2. Configuring FCoE requires many more network configuration steps and requires fewer configuration steps on each individual host when compared with iSCSI.   
3. Configuring iSCSI requires much more work on each individual host and requires fewer configuration steps in the network when compared with FCoE.

I’ll describe why this matters in a bit.

A few notes about the numbers in the table:

• The steps that were counted and entered into each cell have been included for your reference at the end of this post.  See “Configuration steps”
• Since we had to deal with the comparison of “mouse clicks” and “CLI commands”, we tried to break them down into logical steps.  With the CLI this was basically one step for every CLI command.  For the GUI, we tried to count one step per dialog box unless I had to type in a value and then we counted each value entered as one step.  Perfect? No!  But we tried to be fair..
• I have not counted steps that are common to all protocols such as attaching cables.
• I’ve also included information that shows how TDZ would reduce the total number of provisioning steps. 

Getting to the point

As you may have noticed, in terms of the number of provisioning steps, the difference between FC, FCoE and iSCSI is not that great.  In fact, based on the numerous “FC is harder to use that iSCSI” complaints I’ve heard over the past couple of years, I was very surprised to see that FC requires 1/3 the number of configuration steps as iSCSI and I was also surprised to discover how many configuration steps are required to configure FCoE.  If you were just to look at these numbers you may come to the conclusion that FC is the clear winner and it will continue to dominate in the SAN, but then how do we explain iSCSI’s growth in the server virtualization space.  Obviously there are multiple factors at play but again Mark had a hypothesis about this that resonates as true to me “reducing the number of user interfaces is more important the reducing the total number of configuration steps”.  And this is where the concepts of “Network-centric” and “End-Node-centric” environments come into play.

Network-centric

FC and FCoE are Network-centric.  Both protocols rely on fact that the network will control what each end device has access to.  This control is centrally managed and requires a special set of FC skills to administer properly.  Since these are fairly specialized skills, I think you are more likely to find people that possess them in large organizations where controlling access to a pool of shared storage is a necessity.  In addition, since control is centrally managed, the end devices have evolved so that they will try and utilize every device that they can discover in the SAN.  This evolved behavior is one of the things that made the T11 FC-SCM technical report so difficult to write and unfortunately so un-implementable.  The bottom line is a Network-centric approach is probably better suited for large organizations that need centralized control of access to storage resources. 

End-Node-centric

iSCSI is End-Node-centric.  iSCSI relies on the fact that the network will allow communication between the iSCSI Initiator and whatever iSCSI Target the Server Admin points the Initiator at.  iSCSI requires no special set of FC SAN Administration skills, so it provides a very low hurdle for entry into the SAN space.  I suspect this low hurdle is the reason for uptick in iSCSIs adoption in the low end of the server virtualization space.  In addition, since control is managed at each individual end point, the end devices have evolved so that they will only discover what they are told to discover.  Maybe this partially explains why the iSCSI iSNS has been adopted by so few.  The bottom line is an End-Node-centric approach is probably better suited to smaller organizations that do not need centralized control of access to storage resources. 

LUN Masking

In both the Network-centric and End-Node-centric provisioning models, there is typically some kind of LUN Masking implemented on the storage array.  I suppose that this could be viewed as a centralized control of storage resource but since it is common across both iSCSI and FC/FCoE, I’m assuming that it would not influence the decision to use one protocol over another. 

Conclusion

So which one is right for you? 

I don’t know!  Figure it out for yourself!!! :-)

Seriously, it depends on what you need:

• if you require a separate high performance (up to 16GB/s) and fault tolerate SAN, then choose FC or FCoE.
• If you are just entering into the SAN space and want to connect to your external storage without having to learn about FC/FCoE, then iSCSI is probably the best fit for you.
• If you are somewhere in the middle, then it's mostly a matter of personal preference and you should choose what works best in your environment. 

Topology tested

 

Configuration steps

Note: Please keep in mind that the steps listed below are just a description of each task and do not necessarily represent all of the details required to complete that particular task.  An example is “Install HBA”, obviously there is a reboot implied in this step.  I didn’t include all of the detail because the document that contains all of details was 50+ pages long..

Hosts

Windows:

FC:

1. Install HBA
2. Install Driver

FCoE:

1. Install CNA
2. Install Driver

iSCSI:

1. Locate the adapter that will be used for iSCSI in the “Network Connections” dialog
2. Right click the adapter, select properties, then select “TCP/IPv4” and click properties
3. Select “Use the following IP Address”
4. Enter an IP address
5. Enter a subnet mask
6. Enter a default gateway
7. Click OK and then close.
8. Install the Microsoft iSCSI Initiator software (can be downloaded from microsoft.com if necessary)
9. Open the Microsoft iSCSI Initiator properties dialog by clicking Start / Administrative Tools / iSCSI Initiator.
10. Add an iSCSI Target by clicking the Discovery Tab and then Add Portal…
11. Enter the IP Address of both iSCSI Targets into the “IP address or DNS name:” field as shown below:
12. Click Advanced and the Advanced Settings dialog is displayed
13. In the Local Adapter field, choose Microsoft iSCSI Initiator from the pull down list.
14. In the Source IP field, choose the IP Address of the Adapter that will be used to access this target.
15. Click OK and then OK again.
16. Click on the Targets tab and the iSCSI Targets should be displayed
17. Select the correct target in the list and then click Log on…, the Log On to Target dialog is displayed
18. Ensure that the Automatically restore this connection when the computer starts checkbox is selected.
19. Click OK.

Linux:

FC (FC drivers supported by EMC are inbox):

1. Install HBA

FCoE (FCoE drivers supported by EMC are inbox):

1. Install CNA

           iSCSI:

1. Download the appropriate driver
2. Unzip the driver
3. Install the driver and interact with installation script
4. Start the iSCSI driver
5. Verify the iSCSI driver was started
6. Edit the /etc/iscsi/iscsid.conf file and verify:
7. node.session.iscsi.InitialR2T is set to yes;
8. node.session.iscsi.ImmediateData is set to No; and
9. node.session.timeo.replacement_timeout is set to 60.
10. Set the run levels of the iSCSI daemon to automatically start at boot and to shut down when the server is brought down using chkconfig –level 345 iscsid
11. Assign an IP Address to the NIC that is to be used as the iSCSI initiator.
12. Use service network restart to allow the IP Address change to take effect.
13. Discover the Symmetrix iSCSI target using iscsiadm –m discovery –t st –p 10.246.54.109
14. Log into the storage using iscsiadm –m node –L all

VMware:

FC:

1. Install HBA

FCoE:

1. Install CNA

iSCSI:

1. launch the vSphere client and login
2. Select a host from the inventory panel.
3. Click the Configuration tab and click Networking.
4. In the Virtual Switch view, click Add Networking.
5. Select VMkernel and click Next.
6. Select Create a virtual switch to create a new vSwitch.
7. Select a NIC to use for iSCSI traffic.
8. Click Next.
9. Enter a network label - A network label is a friendly name that identifies the VMkernel adapter that you are creating, for example, iSCSI.
10. Click Next.
11. Specify the IP settings and click Next.
12. Review the information and click Finish.  Verify the vSwitch that was created is show\
13. Select the host from the inventory panel of the vSphere client
14. Click on the configuration tab and then storage adapters.
15. Click on “add” above the storage adapters field and select Software iSCSI Adapter.  The iSCSI Software Adapter will be shown in the Storage Adapters section.
16. Select the iSCSI adapter and click properties in the details text area and the iSCSI Initiator Properties dialog is displayed. 
17. Click on the Network Configuration tab and then Add. 
18. Select the appropriate VMkernel Adapter and then OK.
19. Repeat for the other VMkernel Adapters.  When done, the iSCSI Initiator Properties dialog will appear as follows.
20. Click the Static Discovery tab and then click Add
21. Enter the IP Address of the iSCSI Target and it’s IQN.
22. Click OK.
23. Click OK and a rescan will be performed.  Once LUNs have been made available from the storage, perform a re-scan and devices should be visible from ESX. 

Network – Not operating system specific:

FC:

Zoning:

1. Create Zone
2. Add WWPN member 1
3. Add WWPN member 2
4. Add zone to configuration
5. Activate configuration

FCoE:

1. Enable FCoE Feature (one time)
2. system qos (The following QOS settings are done once)
3. service-policy type qos input
4. fcoe-default-in-policy
5. service-policy type queuing input
6. fcoe-default-in-policy
7. service-policy type queuing output
8. fcoe-default-out-policy
9. service-policy type network-qos
10. fcoe-default-nq-policy
11. vsan 600 (one time)
12. vlan 600 (one time)
13. fcoe vsan 600 (one time)
14. int vfc 5
15. bind int e1/5
16. switchport trunk allowed vsan 600
17. no shut
18. int vfc 6
19. bind int e1/6
20. switchport trunk allowed vsan 600
21. no shut
22. int e1/5
23. switchport mode trunk
24. switchport trunk allowed vlan 1, 600
25. spanning-tree port type edge trunk
26. int e1/6
27. switchport mode trunk
28. switchport trunk allowed vlan 1, 600
29. spanning-tree port type edge trunk
30. vsan database
31. vsan 600 interface vfc5
32. vsan 600 interface vfc6
33. Create Zone
34. Add WWPN of host
35. Add WWPN of target
36. Add zone to configuration
37. Activate configuration

iSCSI:

1. VLAN 700 (one time)
2. int e1/5
3. switchport access vlan 700
4. no shut
5. int e1/6
6. switchport access vlan 700
7. no shut

Storage – Not protocol specific:

1. Create the storage group
2. Add storage devices to the storage group
3. Create the port group
4. Add Symmetrix interfaces to the port group
5. Create the initiator group
6. Add the initiator to the initiator group
7. Create a masking view

Thanks for reading!

Partner Mac Address: 00:00:00:00:00:00

While setting up bonding on a Linux host running SLES 11, a colleague and I noticed that the output from cat /proc/net/bonding/bond0 showed a “Partner Mac Address:”  of 00:00:00:00:00:00. (BTW, we’ve also observed this symptom on at least one other distro (Debian)).  We were interested in the Partner MAC Address field because we were trying to troubleshoot a connectivity issue and we thought (I believe mistakenly) that this was an indication of some kind of configuration problem on the switch.  It turns out that this symptom is related to a switch configuration parameter and the short answer is: check to make sure that you have set the port-channel on the switch to “mode active”. 

For a more detailed explanation, please read on. 

Bonding overview

Before I dive into the details, a brief description of bonding would probably be helpful.  Bonding (known as NIC Teaming in windows environments) allows for two separate physical interfaces to be treated as a single logical interface.  There are a number of different ways to use bonding, so for the sake of this example, I am going to assume we want to use Active/Active.  More specifically we are going to use the setting “mode=802.3ad miimon=100”.  For more information about bonding, I highly recommend that you review the bonding section of the linux foundation website .

Topology

The approximate physical topology we had configured in the lab is shown below:

The bonding configuration on the host that can be found in /etc/sysconfig/network/ifcfg-bond0 was as follows:

 

And the current status of the bond as found via cat /proc/net/bonding/bond0 was:

 

When we noticed this, we weren’t sure if a Partner Mac Address of all zeros was a problem or not but we were able to ping other devices that could only be accessed via the bond interface.  The question was, were both interfaces up and part of the bond, or not? 

The first way we thought to verify this was to use tcpdump –i bond0 and compare the traffic on the newly configured bond with a known working bond on another host.  When we did this, we noticed there were lots of LACP frames being exchanged between the host and the switch in the known working configuration and there weren’t any LACP frames being exchanged between the host and the switches we were having problems with. 

Next, we reviewed the configurations from both of the Cisco switches in our topology and verified that they were the same:

 

Next, we reviewed the mac address table on both of the Nexus switches using show mac address-table.

 

Since the same host MAC Address was shown in both tables, we came to the conclusion that the bond was working properly. However, we were still curious about the Partner MAC Address value, so on a hunch we changed the switch interface configuration so that both members would be using mode active

and the Partner MAC Address was now properly populated 

To get to the bottom of where the Partner MAC Address value was coming from, we took an xgig trace of the bond initializing both with and without "mode active".  After reviewing the trace, it appears that server pulls the Partner MAC Address out of the LACP frames from the switch, see below

In the case where the Partner MAC Address is set to zero, the port-channel is set to “mode on” and the switch doesn’t transmit any LACP frames.  As a result the server is unable to populate this field. 

I don’t believe there is a problem either way but at least now you’ll have a basic idea of why the Partner MAC Address is zero if you’re ever wondering.

Thanks for reading!

Introducing Target Driven Zoning (TDZ)

At 9:28 on Tuesday 12/6/2011, the T11 FC-GS-7 working group approved a motion to incorporate the text for “Peer Zoning” that was prepared by Claudio DeSanti (Cisco).  The actual text can be found in the T11 document 11-411v2.  To me this moment marked the culmination of a four year journey to take a concept, get it through a standards body and into a standards document.

Target Driven Zoning (TDZ) is a proposed application that utilizes Peer Zoning to reduce the number of steps required to provision new storage by 50%.  Since many customers have been complaining about the task of zoning for years, I’m proposing to achieve this reduction by eliminating the manual task of FC zoning. 

Just to be clear, while I view introduction of the Peer Zoning functionality as a game changer for FC, I don’t view the completion of this effort as some kind of extraordinary accomplishment, actually there’s nothing extraordinary about it.  These sorts of efforts are started all the time in standards bodies.  Sometimes they result in useful protocols that are implemented by many (e.g., FCoE and FIP).  Other results are implemented by few but they provide a critical requirement for the industry (e.g., FC-SP) and yet others turn out to be interesting academic exercises but are never implemented by anyone (e.g., FC-SCM).  I have hopes that Peer Zoning will fall into the category of “useful technology that is implemented (and deployed) by many” but it’s too early to tell as this point. 

This post is intended to give you an overview of the technology and give you enough information to decide if TDZ is something that you’d like to use in your environment.  I also feel it’s important to give credit where it’s due and with that in mind without the help of Mark Lippitt, David Black, Claudio DeSanti, Bob Nixon and Ralph Weber; this entire effort would still be hopelessly stalled.      

Background

The story starts back in mid-2007.  I was on a conference call with David Black and Mark Lippitt (both with EMC) discussing what would happen if zoning wasn’t used in an FC SAN (as was being proposed in the T11 FC-SCM working group).  In case you don’t know, EMC has a fairly strict best practice called “Single Initiator Zoning” that was created based on lab testing results.  As the name suggests, the best practice states “zones should only contain a single initiator as well as the storage targets it needs to access.”  Because of the work that I had done with fabric scalability, I was pulled into the meeting (somewhat last minute if memory serves) and asked to provide my opinion on what was being proposed.  Now, as most people who work with me will tell you, I typically call things as I see them and the intensity of my voice will vary with the level of conviction that I feel about a given topic.  As a result, my reaction to eliminating zoning was something along the lines of “ARE YOU <bleep> KIDDING ME?” and I proceeded to provide specific reasons why I thought that this was the *SILLIEST* idea I had ever heard of.  Perhaps it was the intensity of my reaction (or the simple fact that David simply didn’t have sufficient bandwidth to create a presentation that contained all of my points), in either case, he eventually asked me to present my list of concerns to the T11 FC-SCM working group at the August 2007 meeting in Seattle. 

Preparing for Seattle

In case you aren’t familiar with T11, this is the group that literally wrote the book on FC.  In fact, a better way to say it would be they ARE the book.  From my point of view as an integration engineer who “grew up” with FC, the idea of merely attending (let alone presenting at) this meeting felt something akin to traveling to Mt Olympus and meeting Zeus and the gang.  As a result, I spent much of the month before the meeting preparing by testing specific “unzoned” configurations in the lab, re-reading the various standards involved (FC-GS and FC-SW), iterating on the presentation with Mark Lippitt and working myself up into an emotional wreck in general.  As luck would have it, the FC-SCM meeting was first on the schedule for the week, so off I went to present at my first T11 meeting without having actually ever attended one. 

The FC-SCM meeting

During the FC-SCM meeting I presented “FC-SCM-OpenZoning”.  The main point of the presentation was due to the way Initiators and Targets perform discovery, you can’t simply eliminate zoning.  If you did, the resulting flood of name server queries every time something changed in the fabric would bring the fabric to its knees with as few as 255 N_Ports in the fabric.  Consider the case where an N_Port is bouncing (logging in an out) and you are totally screwed.  After speaking to my last slide and answering a couple of questions, I returned to my seat feeling like the weight of the world had been lifted off of my shoulders.  I think Mark Lippitt and David Black said something to the effect of “Nice job” and my response was going to be “Boy, am I glad that’s over with” but before I could even get the words out of my mouth, Claudio turned around and said something to the effect of “Ok, so you have shown us a problem, now what?  Would you be willing to come back and present a solution?”  I literally couldn’t believe my ears, I was like “are you talking to me?”  After briefly consulting with Mark, I said yes and committed to provide a proposal at the October 2007 T11 meeting in Coeur d’Alene, ID.  The purpose of that proposal would be to describe an approach that would eliminate the manual task of zoning.

Inspiration strikes

After the FC-SCM meeting Mark, David and I went out to lunch.  On the walk down to Pike’s Place Market I was walking behind them while they were talking about the concept of OLZ or “One Large Zone” (what would eventually become FC-SCM).  I found OLZ interesting and would dedicate many hours over the next three years pushing it for it to be adopted, but I wanted to pursue a different approach.  I believed (and still do) that the key to a solution was to solve the RSCN problem.  Basically, the RSCN problem happens in an environment without zoning when a single device logs in or out.  When the device’s status changes an RSCN is sent to all devices that are impacted by the change.  Since all devices register for RSCN in a fabric without zoning, every device would receive an RSCN and would pile on the Name Server at the same time and basically cause a DoS attack.  With this concern in mind my thinking process went something along the lines of:

What if somehow we could send RSCNs only to end devices that are actually impacted by the change (i.e., hosts that are logged into that target and vice versa)?

• Where would this information come from?

Maybe a storage port could tell the switch to only send RSCNs to end devices that are logged into it?

• You would probably identify these devices to the switch by their WWPN

But wait a second, our storage ports have a list of WWPNs that they care about (have been granted access) in the LUN masking database

• If a storage ports could somehow send these WWPNs to the switch…
• A switch could use this information to restrict RSCN distribution

But…switches already use the WWPNs in the definition of a zone

Why couldn’t the storage port just share the WWPNs in the LUN Masking database with the switch and then the switch could just create the zones based on this information???

The rest of the story

Describing all the ups and downs of the story goes way beyond the scope of this post, but a couple of excerpts from a cliff notes version would certainly mention; Claudio agrees to write the text at the August 2011 meeting in Edmonton, delivers the goods in Albuquerque and it all gets approved in St. Petersburg.  Thanks again Claudio! 

Target Driven Zoning technical details

In this section I describe how I envision the Peer Zoning functionality Claudio defined in 11-411v2 will be utilized by an application that I am calling Target Driven Zoning or TDZ.

The Target Driven Zoning concept is simple, since a storage port has all of the information required to create an FC zone, it should just go ahead and create one automatically.      

Before I dive into the protocol level details that will explain how TDZ works, let me take step back and define the Storage Provisioning process as it is done today without TDZ and then show how it would work with TDZ.

Without TDZ

1. Physically attach hosts and storage to the fabric.
2. Using the switch zoning interface, create a zone that contains an initiator and its targets.  Repeat for every initiator/target relationship being added to the environment.
3. Activate the new Zone Set / Configuration.
4. Using the Storage array management interface, add each initiator to a “Storage Group” on the array.

With TDZ

1. Physically attach hosts and storage to the fabric.
2. Using the Storage array management interface, add each initiator to a “Storage Group” on the array.  The storage will then automatically setup the appropriate zones to allow each initiator to access it.

The nuts and bolts of Peer Zoning / TDZ

For the remainder of this section, I’ll be referring to the following configuration.  Starting from the left, the configuration consists of:

1. A Host containing a single HBA/CNA port that has a WWPN (World Wide Port Name) of 10:00:00:00:00:00:00:00.  
2. A FC Fabric containing two FC switches:
   1. FC/FCoE Switch Domain 3; and
   2. FC/FCoE Switch Domain 4
3. A Storage array containing a port that has a WWPN of 50:00:09:71:20:30:40:50

Step 1 – The Target queries the Unzoned Name Server

Due to an assumption on our part of “when creating a Storage Group, users would rather pick WWPNs from a list rather than manually type them in”; TDZ suffers from a causality dilemma that is solved by utilizing the unzoned name server.  

The dilemma is:

How do you pick a host WWPN from a list in the array’s management software when none of the interfaces on that array have been zoned to have access to that host WWPN?

As I describe in the Hard zoning versus Soft zoning blog post, the response from the switch to a Name Server query will usually only contain those devices that share a common zone with the querying N_Port.  When TDZ is being used, there is a very good chance that the storage has not been zoned to have access to anything.  As a result, it will not be possible to provide a list of initiator WWPNs to the storage admin via a traditional NS query.    

 

The unzoned Name Server provides a solution to this dilemma because as the name implies, the unzoned name server allows an N_Port to query the Name Server and obtain a list of all N_Ports registered with the fabric without regard for zoning.

Step 2 – The Name Server returns a list of all ports registered in the Fabric

The response to the unzoned Name Server query will be a list of all N_Ports registered with the fabric. 

As you think about this behavior, you may come to the conclusion that it represents a serious security flaw.  As Peer Zoning was being defined, we felt the question we needed to answer was:

“If any N_Port can just query the unzoned Name Server, what’s to stop a host from using the unzoned name server to discover what targets are out there and then grant itself access to any storage port that it wants?” 

We thought about this question (a lot actually) and we came up with a multi-pronged approach.  As I’ll describe shortly each approach represents a different level of security that can be set on a fabric wide basis.  However, before I get to the descriptions, let me state that no matter which one you choose, you are still protected by LUN Masking on the Storage array.  In other words, if a rogue host were to grant itself access to the target using Peer Zoning commands, the rogue host would still be prevented from accessing data on that target due to LUN Masking.  LUN Masking will only allow certain WWPNs to access certain LUNs on each target.  So even if a host created a zone to allow itself to have access to a target port, it would also need to spoof the WWPN of a host that has been granted access to LUNs on that target.  Let me point out that this is something that can easily be done today on all HBA and CNAs and does not represent a new security hole that is being introduced by peer zoning. 

• Option 1: Peer Zoning disabled (default).  If you don’t find manual zoning tasks to be too much trouble, then you can opt not to enable peer zoning.
• Option 2: Peer Zoning enabled – Authentication required.  This option allows you to enable Peer Zoning but will require any port that wants to use the unzoned Name Server to authenticate with the switch using one of the mechanisms defined in FC-SP (e.g., DH-CHAP)
• Option 3:  Peer zoning enabled – Port based.  This option allows you to enable Peer Zoning but it will only allow unzoned name server queries on certain switch interfaces.  You could simply enable the feature on interfaces where storage ports are located.
• Option 4: Peer zoning enabled – OUI based.  This option allows you to specify that certain vendor OUI’s can use unzoned Name Server queries by default.  (e.g., I trust <your storage vendor of choice> (hopefully EMC) OUIs but not others)
• Option 5: Peer zoning enable – open.  This option allows you to specify that Peer Zoning can be used by any N_Port in the fabric.

Step 3 – The Storage Admin attaches the correct host WWPN to a “Storage Group”

This is more of a Storage Provisioning process step and does not require any FC protocol interaction.  From within the Storage Array software application, the storage admin selects a host and associates it with a “Storage Group”.  When the user clicks “OK” or something similar, the next step will be performed.

Step 4 – The Storage port uses AAPZ

Once the Storage Administrator adds the WWPN of the host to the storage group and commits this change, the storage port(s) that are associated with this storage group will use AAPZ (Add/replace Active Peer Zone) to add the appropriate peer zones to the fabric.  The AAPZ request will contain the zone name, the principal N_Port name as well as the zone members (the peers) that should be allowed to access the principal N_Port name.

Step 5 – The AAPZ is accepted by the switch

Assuming the switch is functioning properly and it is not too busy to process the AAPZ, it will accept the AAPZ. 

Step 6 – New zoneset / configuration is activated on the fabric by the switch

The Once the switch transmits the ACC to the AAPZ, it has one minute to actually update the zoning on the fabric.  An example of a peer zone is shown at the top of the following diagram. 

 

Step 7 – RSCNs distributed to all affected N_Ports

Once the zone set containing the new Peer Zone has been activated onto the fabric, each switch will transmit RSCNs to each affected N_Port.  By the way, there is nothing new here.  RSCNs are an existing part of the FC protocol and are widely used today.

Step 8 – The Host performs FC discovery

Once the hosts in the peer zone have received an RSCN, they will perform FC discovery as normal.  This is the same discovery process that I described in the Hard versus soft zoning blog post with one big exception; the peer members in a peer zone are only allowed to access the principal zone member and not each other.  This is a new behavior specific to peer zones and it is was put in place to ensure that our single initiator zoning best practice can remain intact.   

Step 9 – Storage verifies zoning via GAPZ

At any point after using AAPZ, a target port can use GAPZ to verify that the zoning change is in place.

Conclusion

If you’re interested in hearing more about TDZ or are an EMC Customer and interested in actually giving it a try, please let me know!  A robust amount of Customer demand is all that is required to get this idea out of the prototype phase and into the ready to deploy phase.

Thanks for reading!

Hard zoning versus soft zoning in a FC/FCoE SAN

Over the past few weeks, I’ve answered a bunch of questions that have asked something along the lines of “Should I use Hard Zoning or Soft Zoning?”  And no, I have no idea why these questions are coming up now…    

The short answer

If you’re familiar with FC Zoning as well as the FC protocol and don’t require a detailed explanation, the short answer is; “You no longer have a choice.  As of today, all of the FC/FCoE switches supported by EMC provide hardware enforcement for both “WWPN” and “Domain, Port” based zones”.  For the sake of completeness, there are some situations where hardware enforcement is effectively disabled but as long as your fan-in and fan-out ratios remain below 1:64 or 64:1 and/or you're not using NPIV, you won’t need to worry about this.  

Read on if you’d like a bit more detail.

The long answer...

As I started writing this section it quickly became apparent that I would need to provide an introduction to zoning and this leads to a discussion about FC Login, discovery, etc.  So I’ll start with the basics first and then get a bit more detailed as I go.  To start with, let’s assume a configuration similar to the following.    

Starting from the left, the configuration consists of:

1)      A Host containing a single HBA/CNA port that has a WWPN (World Wide Port Name) of 10:00:00:00:00:00:00:00 and was assigned an FCID (Fibre Channel IDentifier) of 030100 during the Fabric login process.  

2)      A FC Fabric containing two FC switches:

• FC/FCoE Switch Domain 3; and
• FC/FCoE Switch Domain 4

3)      A Storage array containing a port that has a WWPN of 50:00:09:71:20:30:40:50 and was assigned an FCID of 040200 during the Fabric login process.   

A few points to take note of:

• This configuration could be all FC, all FCoE or a combination of both.
• The HBA/CNA and Storage Array ports both have a WWPN that is assigned to the port in some vendor specific way AND an FCID that is assigned to each port by the switch during Fabric Login.
• The middle byte of the FCIDs that I have shown correspond to the physical interface and this is not always the case (especially in Cisco environments)
• I will only be talking about Open Systems hosts in this post.  Apologies to FICON users, you’ll have to wait for FC-SB-5 to be finished before any of this applies to you.

What is a zone, zone set, configuration, etc

A zone is a collection of WWPNs, FCIDs or both that have been associated with one another for the purpose of allowing them to freely access one another.  Zones have to be added to a zone set or configuration and then activated onto a fabric in order for the access defined in the zone to become effective.  For example in the diagram below, I’ve updated the example topology so that it now shows a single zone named “TEST” that is a part of the Zone set/configuration “Test CFG” and contains two WWPN members. 

  

A couple of important points to note:

1)      For all practical purposes, the zone set “Test CFG” could have been activated on any switch in the Fabric.  This is because when a zone set is activated it is pushed to every switch in the fabric simultaneously. 

2)      As soon as the zone set has been activated, the zone members will be allowed to discover one another.  This is where the difference between hard zoning and soft zoning starts but before I dive into those details let’s talk about FC discovery first.

The FC Discovery process

 In order for an HBA/CNA or storage port to use a fabric, it has to login, perform registration and then perform discovery.  This process is illustrated in the following diagram and explained in more detail in the text that follows.

Each time an FC port initializes, it must perform some kind of login process in order for it to be able to use the fabric.  In practice each HBA/CNA uses a slightly different login process but they all have certain things in common.  One example is the need to transmit FLOGI (Fabric Login) and receive an ACC (accept) from the fabric before attempting to do anything else.  I point this out because although in practice all of the HBA/CNA or storage port implementations that I am familiar with will perform the same basic steps, the order of steps 5, 7 and 9 may be slightly different and the exact commands used within each step may vary as well.

Step 0 (Not shown in diagram)

Assuming a port is initializing from the power off or loss of signal/sync state, speed negotiation and Link Initialization will be performed.  At the end of the Link Initialization process, the HBA/CNA or Storage port that is initializing will just be passing IDLE frames back and forth with the switch port it has been connected to and both ports will be considered to be in the ACTIVE state.

Step 1 (FLOGI)

FLOGI (Fabric Login) is the first frame transmitted by an end device that is attempting to attach to a switch.  The FLOGI contains many bits of information about this initializing end device (N_Port), but for the sake of the discussion on zoning, we only need to recognize that the initializing N_Port’s WWPN (World Wide Port Name) is included in the FLOGI.  As shown previously, this WWPN can be used in a zone to allow the initializing N_Port to access another N_Port.

Step 2 (FLOGI ACC)

Assuming the switch is functioning properly, is not too busy and there are no configuration parameters set on the switch that would prevent a particular N_Port from logging in, the switch will respond to the FLOGI with an FLOGI ACC.  The format of the FLOGI ACC is identical to the FLOGI request but the information contained within it will be specific to the responding switch/switch port.  Also, a VERY IMPORTANT piece of information that is relevant to zoning enforcement known as the FCID (Fibre Channel IDentifier) is provided to the N_Port in the FLOGI ACC.  Unlike the WWPN, this FCID will be present in every frame that will be transmitted by the N_Port and as you will see, the FCID effectively becomes the N_Port’s identity while logged into the fabric.  The relationship a WWPN has with the FCID it has been assigned by the fabric is one of the factors that allows for zoning to be enforced in hardware.

Step 3 (PLOGI NS)

One the FLOGI has been accepted by the switch, the N_Port will login with the Name Server.  Again, this has to be done in order for the N_Port to be able to register with and send queries to the Name Server.

Step 4 (PLOGI ACC)

 Assuming the switch is functioning properly and is not too busy, it will respond to the PLOGI with PLOGI ACC.  The format of the PLOGI ACC is identical to the PLOGI request but the information contained within it will be specific to the responding switch/switch port.

Step 5 (Register with the Name Server)

Once the attaching N_Port is logged into the Name Server it may register with it.  There are many different registration commands that an N_Port could use but a typical one is RFT_ID or in plain English “Register the specified FC-4 Type for the provided FCID”.  In this case we will assume that all N_Ports involved either have or will register an FC-4 Type of “SCSI-FCP”.  Note, although it’s a bit out of scope for this post, I will mention that this isn’t usually the case.  Many N_Ports (especially targets) choose to register with the Name Server in multiple ways, the purpose for doing so is to ensure that no matter what Name Server query an initiator uses, the target will always be discoverable.  

Step 6 (Name Server Accepts registrations)

 Assuming the switch is functioning properly and is not too busy, it will accept the registrations.  If this fails, an attaching N_Port should retry.

Step 7 (SCR - State Change Registration)

The attaching N_Port will perform a full registration for RSCNs.  By doing so, the N_Port is requesting the switch to send it a Registered State Change Notification (RSCN) every time something in the fabric changes.  These changes currently include events like the addition and removal of a Domain (e.g., switch) as well as a zoning change being made to the fabric that impacts the device.

Step 8 (Fabric Controller accepts the State Change Registration)

Assuming the switch is functioning properly and is not too busy, it will accept the SCR

Step 9 (Query the Name Server)

At this point in the login process the attaching N_Port will query the Name Server in an attempt to discover what other devices are logged into the fabric and available for it to communicate with.  Since I stated that we would assume all N_Ports in the fabric would register an FC-4 type of SCSI-FCP (see step 5), we’ll assume that the attaching N_Port will use a Name Server query called GID_FT or in plain English “Get Port Identifiers for the devices that registered the specified FC-4 type” and will specify an FC-4 of SCSI-FCP.

And finally … information directly relevant to the topic of Hard Zoning versus soft zoning.

Step 10 (Name Server response to queries)

In response to the Name Server query, the switch will return a list of port identifiers (i.e., 030100 and 040200).  More specifically, the switch will only return the Port IDs for other FC devices that:

1. have registered an FC-4 of SCSI-FCP (since that is what we queried for); AND
2. share a common zone with the device that queried the Name Server. 

For example, in the following diagram Host 1, Storage 1A and Storage 1B are all in the same Zone.  If any of these devices were to query the Name Server, they should only get 3 Port IDs back.  In the case of a switch that only supports “soft zoning”; zoning is enforced by allowing N_Ports to discover only those Port IDs that are associated with N_Ports that they have been granted access to (via zoning).  Port IDs associated with N_Ports that do not share a common zone with the querying N_Port will not be returned in Name Server responses.    

The downside of soft zoning is the switch does nothing to prevent a rogue host from transmitting frames directly to another N_Port’s Port ID.  To some this represents a serious security flaw and it’s hard to argue against this point.  However, in practice, this behavior is more of a problem when you had a malfunctioning adapter or an incorrectly programmed driver that would insist on communicating with another device even if access to that device was removed via a zoning change.  

Step 11+ (End to End PLOGI) see diagram below

Once an N_Port discovers the other port IDs it can communicate with, it will perform an end to end PLOGI with them.  In practice only Hosts (Initiators) end up performing this PLOGI and they are usually smart enough to not PLOGI themselves.

When the initiator transmits the PLOGI, it will set the D_ID (Destination ID ) of the frame to be equal to one of the Port IDs returned from the Name Server and it will set the S_ID (Source ID) of the frame to be equal to its own Port ID (assigned in FLOGI ACC). 

When the switch receives the end to end PLOGI, if it supports Hard zoning, it will check the D_ID of the frame and make sure that the specified S_ID is allowed to access (is zoned to have access) the specified D_ID.  If yes, the PLOGI is forwarded on to the destination if not, the PLOGI is dropped.  This is what is commonly referred to as Hard Zoning or Hardware enforced zoning.   Since the PLOGI is dropped, the login process never continues past this point.  I’ve heard but have not been able to confirm in the lab that some switch vendors implement hard zoning by trapping all PLOGIs from N_Ports at the fabric ingress port and forwarding them to the control plane of the switch.  The control plane then only forwards those PLOGI frames that should be allowed.      

Additional considerations

There is so much more that could be said on the topic of hardware enforced zoning.  One point that I should mention is that not all switches supported by EMC today enforce zoning at the fabric ingress port.  In fact, some of the earlier Brocade products would enforce at the fabric egress port.   The net result was, for all practical purposes, the same as if the frame was dropped at the ingress but if I didn’t mention it someone else would have…

Thanks for reading!

FCoE TechBook update for November 2011

An updated version of the FCoE TechBook (v11.0) has been posted to powerlink and emc.com.

New for this version is a case study that describes how to configure FCoE on the Nexus 7000 and MDS.  The topology that is configured within the case study is shown below. 

Also Shreedhan Nikam did a nice job updating the blade server section.  Currently it contains Cisco UCS and IBM BladeCenter related information and we are expecting to add HP Blade server related information in a month or two. 

Thanks for reading!

Upcoming EMC / Cisco webcast "Designing High Availability into a Converged Networking Environment"

Please join J Metz of Cisco and I (Erik Smith of EMC) for a webcast that will cover some of the HA considerations that you need to think about before moving to a Converged Nework topology. 

The webcast details are provided below.  Please remember to pre-register.

Date:  November 10, 2011

Time:  12-1PM Eastern

URL:  http://cemc.bright-talk.com/

Title:  Designing High Availability in a Converged Networking Environment 

Abstract:   Highly available networks have different meanings for LAN and SAN administrators. This session will explore the design principles for a truly converged system, and identify the means by which architects can build highly available networks for both LANs and SANs using converged technologies like FCoE and DCB-based Ethernet.

Particular importance will be paid to Role-Based Access and SAN best practices using a converged infrastructure, along with recommendations for growth and scale.

Join us for this event and learn more about:

• the requirements for High Availability for both LANs and SANs
• the limitations of designing highly available networks and the approaches to mitigate issues within the Data Center
• how Highly Available converged networks scale with increased storage requirements over time

Who Should Attend:  Storage and Networking Administrators who want to learn more designing high availability converged networks.

NOTE:  Please remember that you need to register in advance.

Looking forward to seeing you there!

Thanks for reading!

A Multi-Hop FCoE topology example

As I mentioned in the previous post, in early November we will be releasing an updated version of the FCoE TechBook that will contain a number of fixes as well as a new Multi-Hop case study.  The exact topology that will be configured in that case study is shown below.

The topology is intended to highlight all of the different connectivity options  available and is not intended to represent a realistic topology.  I'll provide a link to the actual case study as soon as it becomes available.

Thanks for reading!

FC/FCoE Connectivity options as of 11/3/2011

Well…we’ve been talking about it since EMC World in May and on November 3rd EMC will officially release support for a Director Class Multi-Hop FCoE solution!  More specifically, we will be adding the Cisco Nexus 7000 and FCoE support for the Cisco MDS 95xx to the November EMC Support Matrix (ESM).  We are also adding a case study to the EMC FCoE TechBook that will provide step-by-step installation instructions for a couple of different deployment options.  I’ll provide a link to the updated techbook once it’s posted. 

Configuration rules

The following configuration rules were used to create the topology diagrams.  If there is a discrepancy between the diagrams and the configuration rules, the configuration rules take precedence:

1. The Nexus 2232 requires a connection to a Nexus 5000 in order to function.
2. The Nexus 2232 does not support native Fibre Channel ports.
3. The Nexus 4001i requires a connection to a Nexus 5000 in order to support FCoE.
4. The Nexus 4001i does not support native Fibre Channel ports.
5. The Nexus 7000 does not support native Fibre Channel ports.
6. The Fibre Channel ports on the Nexus 5000 cannot be used for end device connectivity while the switch is in NPV mode.
7. FC-SW interoperability between Brocade and Cisco is not supported.  When attaching a Brocade switch to a Cisco FCF and vice versa, either NPV mode must be used on the Cisco switch or AG mode must be used on the Brocade switch.
8. The Fibre Channel ports on the Brocade 8000 cannot be used for end device connectivity while the switch is in NPV mode.
9. All host connectivity to the UCS 6100 is via FCoE.
10. The Fibre Channel ports on the UCS 6100 cannot be used for end device connectivity while the switch is in NPV mode.
11. The UCS 6100 does not currently provide an FC zoning interface.  As a result, when the 6100 is running in FC-SW mode, it must be connected to another Cisco switch that is also running in switch mode.
12. Cascading switches that are running in NPV / AG mode is not supported.
13. The Nexus 4001i does not support external VN_Port connections
14. As of November 3rd 2011, EMC supports FCoE on the Nexus 7000 and MDS.
15. Storage ports cannot be connected to the Nexus 5000 while it is running in NPV mode.
16. The Brocade 8000 and DCX do not support VE_Ports
17. Cisco UCS does not currently support VE_Ports.
18. FCoE NPV uplinks between Cisco and Brocade switches are not supported.

Supported connectivity options:

The supported connectivity options have been broken down into several categories with each category focusing on the connectivity options for a particular connectivity component.  The categories are:

1. Configurations specific to Nexus 5000
2. Configurations specific to Nexus 2000
3. Configurations specific to Nexus 4001i
4. Configurations specific to Nexus 6100
5. Configurations specific to Nexus 7000
6. Configurations specific to MDS 95xx
7. Configurations specific to Brocade 8000 / DCX

Configurations specific to Nexus 5000:

Configurations specific to Nexus 2000:

Configurations specific to Nexus 4001i:

Configurations specific to 6100:

Configurations specific to 7000:

Configurations specific to MDS 95xx:

Configurations specific to Brocade 8000 / DCX:

Thanks for reading!

 Note: This post was updated on 11/7/2011 to correct one of the 6100 topologies.

FCoE TechBook update for October 2011

An updated version of the FCoE TechBook (v10.0) has been posted to powerlink and emc.com.

New in this version is a chapter titled “EMC Storage in an FCoE Environment” that was written by Shreedhan Nikam.  It contains information for VMAX as well as VNX and is a very welcome (and necessary) addition to an FCoE TechBook written by a storage company..

Also new in this version is a section for Brocade DCX that was written by Erik Paine.  It contains all of the information you'll need to know when installing an FCoE blade into a DCX.

Finally, I updated the Nexus 2232PP case studies to show that it is no longer necessary to bind the VFCs to the ENODE MAC of the CNA when creating VFCs for CNAs that are attached to the 2232.  This was a change that had somehow escaped my notice and I’m very grateful to Eng Khong Toh for taking the time to call this oversight to my attention!!

Thanks for reading!

Interoperability issue between Brocade CNA and Cisco Nexus 5596

The other day one of my colleagues, Mugdha Kulkarni, brought an interoperability issue to my attention involving a Brocade 1010 / 1020 CNA (Windows Server 2008, Driver 2.3.0.2) and a Cisco Nexus 5596 (NX-OS 5.0(3)N1(1c)). 

The problem was the Brocade CNA was unable to login to the Nexus 5596.  Complicating matters was the fact that the problem would not reproduce on a Nexus 5548.  This was of particular interest to me since based on what we’ve observed (and what we’ve been told by Cisco), the 5548 and 5596 should be identical from a protocol interoperability point of view.     

With this in mind, I spent a bit of time in the lab yesterday trying to figure out if the problem was related to a specific Nexus platform or NX-OS revision.  Based on the testing results and after looking at the traces, my working hypothesis is the problem is due to a DCBX version negotiation problem. 

We are actively working on the problem but unfortunately as of 6/30/2011, there is no workaround.

UPDATE:  As of 10/19/2011, there is still no workaround for this problem but we were able to confirm that the issue is being caused by both devices failing to agree on which version of DCBX to use.  As with almost all interoperability issues, the problem is being caused by unique characteristics of both implementations.  As a result, the "fix" for the problem is for either vendor to slightly alter their DCBX negotiation behavior.  Also as is typical for these types of interop issues, both vendors are providing a "fix" so that the issue can be addressed in as many different environments as possible.  With this in mind, I received confirmation from Brocade that this issue will be fixed with driver version 3.0.3.0 (scheduled to be released early in Q1-2012).  I also received confirmation from Cisco that this issue will be fixed in their next major NX-OS version (also scheduled to be released early in Q1-2012).  

Thanks for reading!