« Connectivity changes for March 2011 | Main | Connectivity changes for April 2011 »



Feed You can follow this conversation by subscribing to the comment feed for this post.

Account Deleted

Hi Eric,

Thank you for such a detailed description, but I am in a fix here. I followed the exact steps given by you for VE port configuration on the cisco nexus. Although on both nexus switches, my vfc's say it is bound to port-channel 777, port is E and mode is TE.

They are just not able to talk with each other. What should be the priority-flow-control settings on the port-channel interface?? I have set them to auto/off/on but no luck. The pre-requisite of identical FC-MAPS on both switches is also followed. Following is the VFC output. Note that on both the switches, I added vfc777 only under vsan 2 in the vsan database.

switch(config-if)# show interface vfc 777
vfc777 is trunking (Not all VSANs UP on the trunk)
Bound interface is port-channel777
Hardware is Virtual Fibre Channel
Port WWN is 23:08:00:0d:ec:b1:d6:ff
Admin port mode is E, trunk mode is on
snmp link state traps are enabled
Port mode is TE
Port vsan is 2
Trunk vsans (admin allowed and active) (2,10,20,200)
Trunk vsans (up) ()
Trunk vsans (isolated) ()
Trunk vsans (initializing) (2,10,20,200)
1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec
0 frames input, 0 bytes
0 discards, 0 errors
0 frames output, 0 bytes
0 discards, 0 errors
last clearing of "show interface" counters never
Interface last changed at Mon May 9 11:12:26 2011


Erik Smith

Hi Somu, sorry to hear you're having a tough time. If you could send me the following output from both switches that you are trying to connect together via the VE_Port, I'll take a look and see if I can figure out where the problem is.

show running-config
show int brief
show fcoe database
show int port-channel 777

You can email them to me at gid_ft@yahoo.com



Do I understand this correctly that the FCoE traffic is separated from the IP traffic on the ISL link? In short, this is NOT a converged network (FCP + IP over Ethernet), but a separate network physical path for FCoE and a second separate network physical path for IP traffic?


Erik Smith

Hi Brook, you're right. There are good reasons for this but they're hard to articulate without sounding like I've drowned in the Cisco cool-aid! :-)

At first I really didn't like the idea because it's not necessary for any protocol related reasons and it messed with the pretty converged network picture I had in my head at the time. That having been said, the Customers that I've spoken with about this actually like the idea of separating FCoE from non-FCoE at the first FCF. The reason is this approach allows the storage guys to mostly retain control of the SAN while allowing them to take advantage of the economic advantages that FCoE to the ToR allows for.

BTW, I'm looking forward to working on VCS, that should be very interesting as well.

Regards, Erik


G'day Erik,

Can you confirm if you have tested or implemented 2 x Nexus 5548UP Switches that are connected over distance (40Km) using ISL/E-port's? Let's say 1 N5K is in 1 datacentre and the other is in another datentre. We are also going through a CWDM.



Erik Smith

Hi Bruce, the maximum distance I've tested between 2 5500s was 300m.

That having been said, I believe what you are attempting to do won't work with the 5500s as they don't have enough buffering to allow PFC to function properly and prevent frame loss due to buffer overrun. Basically, their high water mark threshold (the point at which they will transmit PFC frame due to a buffer full condition) does not take into account the extra "buffering" taking place on the link. As a result, they don't reserve enough space for all of the frames in transit and these frames can be dropped under congestion conditions.

I believe the Nexus 7k and the FCoE Blade for the MDS will support up to 40km.

A workaround would be to use FC for the connectivity between the 5548's.


I know this is an old post, but maybe i get lucky....
I have one question: suppose i have a server and a storage fcoe connected to separate 5ks. The 5ks are vonnected by ve ports.
The fcoe data packets betweek the servers would use source and destination mac as the fcmap, wright? Or both servers would use destination mac addrss as the fcf mac? For me the second one seems more complicated but i can't find an answer...
Thanks for the nice post!

Erik Smith

Hi Sorin, to understand MAC Address use with FCoE, it's easiest to keep in mind that FCoE is really just FC operating at the L3 Network layer and using Ethernet at L2. This means that a connection between a server and a storage that crosses two N5ks connected by VE_Ports actually consists of 3 L2 links; one between the Server and the first N5k, the second between the two N5ks and the final one between the N5k and the storage. As with other protocols (e.g., IP) that use Ethernet as a transport, the L2 Addresses (MACs) only have meaning on each L2 link. As a result, the are stripped off by the L3 entities (e.g., FCoE FCF, IP Routers) and new ones are used to transport the L3 protocol to the next L3 entity. All of this means that between the Server and the first N5k, you'll use the Server's FPMA and first FCFs FCF-MAC. Between the two N5Ks, you'll use FCF-MACs. And between the N5k and the storage you'll use another FCF-MAC and the Storage ports FPMA. I hope this helps!

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.


Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)


  • This is not an official EMC blog.
    The opinions expressed here are my personal opinions. Content published here is not read or approved in advance by EMC and does not necessarily reflect the views and opinions of EMC nor does it constitute any official communication of EMC.