« Network Virtualization – Networking’s 21st century equivalent to the space race | Main | VN2VN – The Good, the Bad and the Ugly »

09/05/2012

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Ivan Pepelnjak

I would love to know which switch you were using. In the good old days a switch (OK, maybe it was called a bridge then) would drop a frame with a CRC error, not forward it.

There should be no transmit CRC errors in a store-and-forward switch. Cut-through switches are a different story. They were a bad idea in 1990's and they still are.

Dmitri Kalintsev

Hi Eric,

Cut-through switching is extremely uncommon in data networking, or at least it was until recently. Now that you mentioned it, I'm very interested in checking with the modern Ethernet "kings of low latency" to see if they are using cut-throgh.

IMO, it is bad idea; I'm very much with Ivan on this one.

Waldemar Pera

Great post! It's a very interesting trouble and a good opportunity to remember what happened "under the hood".

Erik

Hi Ivan, in this case I’m hoping to avoid naming a specific FCoE switch vendor or product. That having been said, I believe that Brocade, Cisco and Juniper FCoE switches will all exhibit this symptom. I’m only naming these three because they are the only FCoE switches that I have direct experience with.

In regards to NV and whether or not the switches that will be used in the underlay will use cut-through or store and forward. I’ve noticed that most of the 10GbE ToR switches I’ve been looking at recently are all claiming latencies of less than 1 usec. To me this indicates they must be using cut-through because it will take at least 1.2 usec to receive 1500 Bytes when using 10GbE (12375 encoded bits / signaling rate of 10.3125 Gbps )

Erik

Hi Dkalintsev, I think it's becoming much more common especially since switch vendors appear to be using the latency metric as a way to differentiate themselves from their competition.

Erik

Thanks Waldemar!

Brook Reams

Erik,

It seems that part of the reason for the bad behavior is because Ethernet relies on frame flooding to determine an unknown address.

In the case of Fiber Channel, since devices login and register with a name service, finding an unknown address does not rely on frame flooding.

As you point our, cut through switching becomes mandatory as link rates go up so the original store and forward architecture isn't feasible and the CRC can only be acted upon by the edge device at the destination.

A tentative conclusion open for debate follows:
1. Finding an unknown destination should rely on a name service and device registration when cut and forward switching is prevalent.

BTW, very useful post with great detail.

Erik

Hi Brook, thanks!

Not using cut-through with OVNs would potentially solve the problem but could create others.

Another idea would be to add a checksum to the "TNID" field and specify that the Tunnel End Points validate and then discard if the validation fails. Not sure how realistic this is though..

Dmitri Kalintsev

Hi Eric,

I followed up with one of my vendor friends, who confirmed that their low latency data-only gear also operates in cut-through mode. His recommendation was that the Layer 1 errors are to be closely and pro-actively monitored, and identified bad talkers acted upon as soon as possible. Some network devices can trigger port actions (such as shut) based on error thresholds, based for e.g. on CRCs, but whether to use this or not would depend on the particular environment, of course.

I agree with the recommendation to monitor and quick act on errors; and it should work equally well with NV.

Cheers,

-- Dmitri

Krunal

This is common problem with any cut thru switching. Nexus 5500 introduces a feature called stomping in which switch does bit flapping in a CRC portion of the ethernet frame.
9000 byte frame with bit flaps/ bad CRC arrives to the switch port 1.
After first 64 bytes received switch decides to send to port 2 and port starts receiving the full frame and putting on wire.

Now last 4 bytes of frame received on ingress port and switch finds that oh this is bad frame but you cannot pull bits from wire. So switch does bit flapping on the CRC and continues to forward the frame however, it accounts on interface counter RX CRC on ingress port and output error on egress port.

If next switch is stored and forward this bad CRC frame will be dropped on ingress port.

If next switch is cut thru that switch does the same process.

All above does not prevent switch from making wrong switching decision because of bit flap in DST_MAC or ether type. But bad CRC frames are identified by next switch.

Erik

Hi Krunal, I agree with what you are saying and have observed this behavior in the lab as well.

What do you think about adding a checksum that will cover the VNID field?

Erik

Dmitri, thanks for the confirmation. Perhaps the answer is as simple as monitor for bit errors or set an aggressive policy that would force an interface offline after exceeding a certain bit error rate..

That having been said, I was hoping for something a bit more automated and was thinking that by adding some kind of checksum to the encap header, that the networking gear could take care these situations automatically (like store and forward architectures do).

Krunal

I dnt know but as you might be already be aware checksum also cannot detect full error. We have checksum filed for each IP header and Not all routers calculates checksum for all IP packets passed thru it. This is one reason IPv6 header is not protected by a checksum.

Dmitri Kalintsev

Eric,

I guess the "correct" answer is, as always, "it depends". In some environments it may be better to drop the defective frames and try to carry on, while in others it could make much better sense to cut off the offender as soon as possible.

My "gut feel" preference is clearly on the side of isolating and fixing the root cause as soon as possible (because even if you carry on, you still know that you'll need an outage to fix the problematic hardware), but on the other hand if bad frames were dropped (meaning their impact is isolated to the directly affected system/s), and their volume wasn't very high, there would be more operational flexibility in regards to when to do the repairs in a more planned manner.

As I said, "it depends". :)

Regarding the idea of introducing an additional CRC for headers - I'm quite sceptical, as in addition to changes to networking standards it would almost certainly require changes to the networking hardware.

Cheers,

-- Dmitri

dort

I was able to find good information from your articles.

Erik

Thanks Dort.

Will Hogan

Erik,

This was a great post and I learned a lot. Thanks a lot!

Erik

Hi Will, I'm glad you found it useful! Thanks for taking the time to comment.

Erik

Erwin van Londen

Hi Erik,

Good post and great find. It's always a PITA to find these kind of issues without an analyzer.

In FC we don't have "VLAN/VSAN flooding". If an in-frame error is encountered whereby the CRC is incorrect and cut-through switching is used (as all FC switches do these days) we simply terminate the frame with EOFti where we invalidate the frame. We do not drop the frame on the next ingress port but we let the recipient handle this. This does prevent additional overhead on the all the ASIC's and is the fastest way of handling these kind of errors.

As for discussing whether cut-trough routing/switching is better or worse then store&forward is up for grabs. There are pros&cons to both.

Cheers,
Erwin

Erik Smith

Hi Erwin, thanks! I agree with your description of how this is handled with FC.

Bill Hubert

Your post was very informative thank you for that

Erik

Thanks Bill!

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Working...
Your comment could not be posted. Error type:
Your comment has been saved. Comments are moderated and will not appear until approved by the author. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.

Working...

Post a comment

Comments are moderated, and will not appear until the author has approved them.

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)

Disclaimer

  • This is not an official EMC blog.
    The opinions expressed here are my personal opinions. Content published here is not read or approved in advance by EMC and does not necessarily reflect the views and opinions of EMC nor does it constitute any official communication of EMC.