Brass Tacks

Let’s play in an AWS based SANdbox – Part 2 Setting up your AWS VPC and networks

In my previous post, I introduced the SANdbox Github repo and described the work that had been done to create an Virtual Lab that is capable of supporting NVMe/TCP traffic.

In this post I’ll walk you through the steps to create your own SANdbox VPC in AWS that will eventually look similar to the following.

Before I get started, I need to acknowledge two sources of information that were invaluable to me as I was getting started with AWS:

1. The AWS documentation – I found an article that seemed to describe exactly what I wanted to do but I didn’t have enough context/knowledge of AWS at the time to be able to follow it successfully.
2. This “HOW TO Build a Home Lab in AWS for FREE” video by Jon Good. It was incredibly helpful and I highly recommend reviewing it before continuing with the following steps.

Create a Free AWS account

1. Click here to create a Free AWS account.
2. Provide an email address
3. Provide an AWS account name
4. Verify the email Address you provided by taking note of the Verification code they provide you an then entering it into the browser when prompted.
5. Create your Root user password.
6. Provide your contact information
7. Provide credit card information
8. Confirm your identity (I chose to use SMS)
9. Select a support plan (I chose Basic support – Free)
10. Click Complete sign up
11. Click Go to the AWS console
12. Enter or ensure that the correct email address is provided and click Next.
13. Enter the password associated with the email address provided and click Next.

Secure your account with MFA

1. In AWS, under Console Home, click IAM
   1. Note: If you ever lose track of a particular view or service, you can always click the Services icon at the top of the page or search for the service name you want to access and get to it that way. I do this all the time for accessing my “EC2” instances.
2. In the IAM dashboard, click Add MFA.
3. Under Your Security Credentials > Multi-factor authentication (MFA), click activate MFA.
4. Choose the type of MFA device to assign (I chose “Virtual MFA device”) and then click Continue
5. In the “Set Up virtual MFA device” dialog, you will be prompted to:
   1. Install a compatible app on your mobile device or computer (I chose the App “Authenticator” because it was supported on IOS and had good reviews online)
   2. Use the virtual MFA app and your device’s camera to scan the QR code shown within AWS.
   3. Enter two consecutive tokens from the Authenticator APP, then click OK, and you should see “You have successfully assigned a virtual MFA. Click Close
   4. Click on “Dashboard” (left side of the screen) and ensure there is a green check mark next to “Root user has MFA”.
6. Click on the user name at the top right of the screen (e.g., SANdboxBlog) and then click Sign out

Setup a Billing Alert

Note: Jon Good recommended this and I cannot stress how good of an idea it is.  You’d be surprised how quickly you can blow through $10 once you start using non-free instances and Elastic IP Addresses. My only regret is I did not also create a $100 alert :-) … do not make the same mistake I did!  The steps for doing this are very straight-forward and can be found here.

Here’s my alarm configuration after setup.

Create a Virtual Private Cloud (VPC)

The VPC is the top level “container” into which all of your Virtual Resources will be deployed. 

1. To create a VPC, click on Services (at the top left of AWS console screen) and click on (or search for) VPC. Click Create VPC.
2. Under VPC settings, I chose VPC and more.
3. Under Name tag auto-generation, I provided a tag of SANdboxBlog.
4. I decided to use a single Availability Zone as my lab configuration does not require HA.
5. I increased the number of private subnets from 1 to 2.
6. All other settings were left at defaults.
7. The VPC preview should look similar to the following
8. Click Create VPC and the VPC workflow will run. Click View VPC after the workflow completes.

Create and associate Security Groups

Creating and managing security groups is a tedious but necessary task.  It allows you to define the sources (e.g., IP Addresses) that can be used to access the resources inside of your VPC.  I created three Security Groups, one for my LAN (public facing) network and two for my SANs (e.g., SAN A and SAN B). I’ve summarized the SGs I created below for ease of reference.

LAN-Public

Name	IP ver	Type	Protocol	Port Range	Source
SSH	IPv4	SSH	TCP	22	0.0.0.0/0
HTTPS	IPv4	HTTPS	TCP	443	0.0.0.0/0

SAN A and SAN B

Name	IP ver	Type	Protocol	Port Range	Source
mDNS	-	Custom UDP	UDP	5353	0.0.0.0/0
NVMe/TCP Discovery	IPv4	Custom TCP	TCP	8009	0.0.0.0/0
NVMe/TCP IO	IPv4	Custom TCP	TCP	4420	0.0.0.0/0
PING	-	All ICMP – IPv4	ICMP	ALL	0.0.0.0/0

Create the “LAN-Public” Security Group

1. From “Services”, click on EC2. (Not really required but I like to start from the EC2 dashboard for everything)
2. On the left side of the console scroll down to “Network & Security” and click on Security Groups and then Create Security Group.
3. Add a Security Group Name (e.g., LAN-Public)
4. Add a Description (e.g., Allows SSH and HTTP access to instances)
5. VERY IMPORTANT Ensure the correct VPC is selected. The default VPC listed in the VPC field is probably not what you want. Click on the ‘X’ in the right side of the VPC field and ensure you have selected the correct VPC. In my case I selected the SANdboxBlog-vpc. If you haven’t already done this, take note of the last four digits of your VPCs instance ID.  In my case it’s “a245”, you will use this later on. Also, I highly recommend creating a diagram of your environment to keep track of all of the instance IDs. A completed example diagram will be provided at the end of this post. 
6. Create Inbound rules – As described in the LAN-Public table above, we need to create rules that will allow for both SSH and HTTPS access to our instances.

 SSH

1. Under Inbound rules, click Add Rule
2. From the Type pulldown, select SSH. This will automatically populate the Protocol and Port Range fields for you.
3. From the Source pulldown select Anywhere-IPv4. The IP address of 0.0.0.0/0 will automatically be added to your list of sources.

NOTE: Using Anywhere-IPv4 is NOT a best practice. Ideally, you would specify the IP Address of the system you will be using to access your VPC.

HTTPS

1. Click Add Rule
2. From the Type pulldown, select HTTPS. This will automatically populate the Protocol and Port Range fields for you.
3. From the Source pulldown select Anywhere-IPv4. The IP address of 0.0.0.0/0 will automatically be added to your list of sources.

Confirm that your security group looks similar to what is shown below and then click Create Security Group.

Once the SG was created I took note of the Instance number for the LAN-Public Security Group (i.e., 4872) and added this to my topology diagram.

Create the “SAN A” Security Group

1. On the left side of the console click on Security Groups and then Create Security Group.
2. Add a Security Group Name (e.g., SAN A)
3. Add a Description (e.g., SAN A is a private network for NVMe/TCP Discovery and IO traffic.)
4. VERY IMPORTANT Ensure the correct VPC is selected. The default VPC listed in the VPC field is probably not what you want. In my case I selected the SANdboxBlog-vpc and it has an instance ID of “a245”. 
5. Create Inbound rules – As described in the SAN A and SAN B table above, we need to create rules that will allow for mDNS, NVMe/TCP Discovery, NVMe/TCP IO and Ping to be used between the instances in our VPC.

 mDNS

1. Under Inbound rules, click Add Rule
2. From the Type pulldown, select Custom UDP. This will automatically populate the Protocol field for you.
3. In the Port range field enter 5353
4. From the Source pulldown select Anywhere-IPv4. The IP address of 0.0.0.0/0 will automatically be added to your list of sources.
5. Provide a description (e.g., mDNS)

NVMe/TCP Discovery

1. Click Add Rule
2. From the Type pulldown, select Custom TCP. This will automatically populate the Protocol field for you.
3. In the Port range field enter 8009
4. From the Source pulldown select Anywhere-IPv4. The IP address of 0.0.0.0/0 will automatically be added to your list of sources.
5. Provide a description (e.g., NVMe/TCP Discovery)

NVMe/TCP IO

1. Click Add Rule
2. From the Type pulldown, select Custom TCP. This will automatically populate the Protocol field for you.
3. In the Port range field enter 4420
4. From the Source pulldown select Anywhere-IPv4. The IP address of 0.0.0.0/0 will automatically be added to your list of sources.
5. Provide a description (e.g., NVMe/TCP IO)

PING (this is not technically necessary, but you will thank me later).

1. Click Add Rule
2. From the Type pulldown, select All ICMP – IPv4. This will automatically populate the Protocol and Port range fields for you.
3. From the Source pulldown select Anywhere-IPv4. The IP address of 0.0.0.0/0 will automatically be added to your list of sources.
4. Provide a description (e.g., PING)

Confirm that your security group looks similar to what is shown below and then click Create Security Group.

Create the “SAN B” Security Group

Using the information from the SAN A and SAN B table above, repeat the steps outlined in “Create the SAN A Security Group” and create another Security Group for SAN B.

OR... You can save a ton of time and do the following

1. Under Actions, at the top of the “SAN A Security Group” configuration screen, you can chose “Copy to a new security group”.

VERY IMPORTANT After providing a “Security group name” (e.g., SAN B) and a Description, you will still need to ensure the correct VPC is selected. The default VPC listed in the VPC field is probably not what you want.  In my case I selected the SANdboxBlog-vpc and it has an instance ID of “a245”.

2. Click Create Security Group

Create your first EC2 Instances

Amazon Elastic Compute Cloud (EC2) instances can be thought of as virtual servers.  For the purposes of the SANdbox configuration, I’ve chosen to use EC2 instances that are running Ubuntu 20.04.  When I was first getting started and needed to work my way through the process of setting up the environment and familiarizing myself with AWS in general, I chose EC2 instances that were available in the free tier. This means there is no charge up to a certain amount of usage.  I will walk you through doing the same for two instances and will get us to the point where you can connect to each instance with SSH and ping across the network interfaces used for SAN A.  We will not be setting up SAN B right now, just validating connectivity across one SAN is enough (for now). 

1. From the “EC2 Dashboard”, click Launch Instance
2. Provide a “Name” for the instance (e.g., Host1)
3. Under “Application and OS Images (Amazon Machine Image)”, click on ubuntu and then select a particular version. I chose “Ubuntu Server 20.04 LTS (HVM), SSD Volume Type” because it was “Free Tier Eligible”, when we actually create the instances we will use for NVMe/TCP testing, we will make different choices that will have a different cost profile. Leave Architecture set to 64-bit (x86)
4. Under “Instance type”, select micro again because it is Free tier eligible.
5. Under “key pair (login), select Create new key pair
   1. In the Create key pair dialog, provide a Key Pair name, I chose to use SANdboxBlog
   2. Select a “Key pair type” of RSA
   3. Select a “Private key file format” of .ppk
   4. Click Create key pair and a file (e.g., SANdboxBlog.ppk) will be downloaded to your system. You will need this file when we connect to the instance via PuTTY in a subsequent step.
   5. Ensure that your key name (e.g., SANdboxBlog) is shown in the Key pair name – required
6. Under “Network Settings” click on Edit
   1. Under “VPC”, ensure the correct VPC is selected (e.g., SANdboxBlog-vpc)
   2. Under “Subnet”, ensure your public network is selected (e.g., SANdboxBlog-subnet-public1-us-east-1a)
   3. Under “Auto-assign public IP”, ensure it is set to Disable
   4. Under “Firewall (security groups)”, click on “Select existing security group”
   5. Under “Common security groups”, leave it blank
   6. Click Advanced network configuration
   7. Under “Network Interface 1”
      1. Set the “Description” to Public-LAN
      2. Under the “Security groups” pull down menu, choose LAN-Public
      3. Leave everything else at the default setting
   8. Click Add network interface
      1. Set the “Description” to SAN A
      2. Under the “Subnet” pull down menu, select the SAN A subnet
      3. Under the “Security groups” pull down menu, choose SAN A
      4. Leave everything else at the default setting
   9. Note, this instance type only supports two network interfaces, so we will only configure SAN A for now.

7. Under the Summary tab (on the right side of the console)
   1. For Number of instances, specify 2
   2. Then click Launch instance
   3. Click View all instances when the button becomes available.

Take an inventory of your Virtual Infrastructure

1. Now that your EC2 instances are up, ensure that you name them appropriately, I chose Host 1 and Host 2. You can rename them in the EC2 Dashboard by clicking in the Name field.
2. For each Host, take note of at least the last four digits of the instance ID.
3. For each Host, click on the Instance ID and select the Networking tab
4. Scroll down to the Network Interfaces section and take note of the last four digits of each Interface ID (you may need to expand the Interface ID column to see the last four digits). You’ll also want to understand what network each interface is associated with, and the Description field should answer this question for you.
5. You’ll also want to take note of the IPv4 addresses for the SAN A interfaces. We will be using them to verify connectivity in the final step.
6. I highly recommend keeping track these instance and interface IDs because we will use them later. Here’s what my configuration looks like at this point.  Note that the IP address for the LAN-Public interfaces are Elastic 1 and Elastic 2, we will actually assign them next.

Create and Assign Elastic IPs

If I was only using a single network interface per instance, I could have had a public IP Address assigned to each Interface.  However, because this functionality is not supported when you have more than one network interface on an instance, I’ve found the easiest way to get access to my instances is to use Elastic IPs.  Unfortunately, Elastic IPs are not free, so please be advised.

1. Under “Network & Security” select Elastic IPs and then click on Allocate Elastic IP address
2. Click Allocate and you will be returned to the Elastic IP Address screen. Take note of the Allocated IPv4 address.  In my case it is 3.230.foo.bar
3. Select the IP Address (little check box to the left of the IP), click Actions and then Associate Elastic IP address
4. Under resource type, choose Network interface and then click inside of the Network Interface box and select the Public-LAN interface of Host 1. BTW, you need to use the resource ID of the interface and this is why it is so important to keep an updated diagram of your virtual infrastructure as you create it.  In my case the last four digits of the interface on Host 1 that is connected to Public-LAN is 20d1.  Clicked on the correct interface ID and then click Associate.
5. Because you need two Elastic IP addresses, click on Allocate Elastic IP address a second time and repeat the above steps for the Public-LAN interface on Host 2. My second Elastic IP was 34.234.you.too

Connect to your Instances

To complete this next step you’re going to need the SANdboxBlog.ppk file that was downloaded to your system when you were creating your instances.  You’re also going to need PuTTY installed on your system as well. 

1. Open PuTTY
2. In the Host Name (or IP Address) field enter the Elastic IP Address associated with Host 1 (e.g., 3.230.foo.bar)
3. Under connection -> Data enter the default user name for ubuntu in AWS (ubuntu)
4. Under connection -> SSH -> Auth, browse to the location of the SANdboxBlog.ppk file.
5. Click Open
6. Accept the warning about the Host Key
7. And you should be logged into your Host 1 instance.
8. Ping the IP Address of the SAN A interface on Host 2, in my case, this IP Address is 10.0.134.136 and verify that it was successful

Next steps

So far we’ve created a Free AWS account, configured MFA, setup billing alerts, created a VPC with three networks, created 3 security groups and verified basic connectivity to and within our VPC using a pair of free tier servers. 

Next time we’re going to deploy 3 larger EC2 instances, and install:

1. nvme-stas on our Host EC2 instance,
2. SmartFabric Storage Software on our Centralized Discovery Controller (CDC) EC2 instance, and
3. SPDK on our Storge EC2 instance.

Thanks for reading!

Let’s play in an AWS based NVMe/TCP SANdbox!

One of the great things about NVMe/TCP is its flexibility.

Not only can you use the NVMe/TCP protocol to connect hosts to storage systems over an arbitrary IP network topology; as originally described here, you can also run it in a surprising number of operating environments, including:

• traditional enterprise environments that utilize array-based storage solutions (e.g., PowerStore and PowerMax),
• next generation operating environments that utilize Software Defined Storage (e.g., PowerFlex),
• completely new use cases, such allowing applications running on the Edge to access centralized block storage resources, and most recently,
• deploying your NVMe/TCP solution in the Cloud (e.g., AWS) and using the resulting "Virtual Lab" to simply get familiar with an NVMe IP-Based SAN, or more importantly test and validate your infrastructure automation software before you use it in production.

The last item, especially the part about using AWS as a Virtual Lab, needs a bit more explanation.  Along these lines, this blog post series will:

1. Provide an overview of the automation tools being made available in the SANdbox Github repository,
2. Provide the step-by-step instructions required to configure a AWS based NVMe/TCP test environment, and then
3. Demonstrate how you can get started with automating your storage provisioning tasks by using the PowerShell, Python or Ansible scripts that we will provide in SANdbox.

Introducing SANdbox!

SANdbox is a Github repo that provides access to the resources you will need when automating your NVMe IP-Based SAN. This includes:

Documentation

Start here if you’re just coming up to speed on the concept of an NVMe IP-Based SAN.  You’ll find links to an IP-Based SAN introductory level blog post as well as links to two detailed SNIA presentations describing how Discovery Automation works at a protocol level. 

There’s also a link to Dell’s SmartFabric Storage Software (SFSS) deployment guide. It covers not only how to install SFSS, but also the basic network topologies that are currently supported as well as step by step configuration examples.

Centralized Discovery Controller (CDC) Downloads

If you’ve been following the NVMe IP-Based SAN space for a little while, you probably already know that a group of companies identified a scalability problem related to the management of IP-Based SANs and decided to work together to solve it.  The result of this work can be found in a couple of NVM Express Technical Proposals (i.e., TP8009 and TP8010). 

• TP8009 describes how hosts can automatically discover subsystems using mDNS or DNS.
• TP8010 defines the concept of a Centralized Discovery Controller (CDC) that limits the number of storage subsystem interfaces visible to each host. As most people who have dealt with FC SANs can attest, this functionality is very important as environments scale up.

To help potential IP-Based SAN users get familiar with the usage of a CDC, Dell has made our implementation of a CDC, “SmartFabric Storage Software (SFSS)” available for download for trial purposes. Please note, this version is for trial and test purposes only and no support is available. If you want the latest, fully featured and fully supported version of SFSS, you need to purchase a license.

If you don’t want to download and install SFSS, either on your own infrastructure or on AWS, you can always give our Interactive Demo a try.     

Toolkit

Once you’ve installed SFSS and understand the basics of interacting with it, you’ll probably be interested in learning how to automate the configuration steps and interact with your infrastructure programmatically. To this end, Dell has provided sample scripts that will help you learn the basics.  Currently we’re providing PowerShell and Python examples and will be adding Ansible soon.

Both the PowerShell and Python scripts will allow you to:

1. connect to the SFSS via the REST API,
2. retrieve a copy of the name server database,
3. add all of the NS entries to a single zone, and then
4. activate the zoning configuration.

The experienced SAN admins amongst you will immediately recognize that zoning all of the ports together is NOT something that would typically be done! However, the example will give you enough information to do something a bit more realistic, such as allowing hosts to access a subset of the available subsystem interfaces.

Virtual Lab

If you’re interested in experimenting with an NVMe IP-Based SAN but you don’t have the HW infrastructure to do so, you can run all of it in AWS and experiment with it there on your own! The configuration I’m experimenting with is shown below.

In my AWS configuration I've created:

1. Three EC2 instances (Host 1, SFSS, Storage) all running Ubuntu 20.04
   1. The Host instance utilizes the Dell maintained open source nvme discovery client “nvme-stas”
   2. The SFSS instance is running our SFSS application and it handles all aspects of discovery automation
   3. The Storage instance is running either nvmet or our “Dell End Point Simulator” that allows you to simulate pull and push registration from the command line.  
2. A public facing network that I am using to access the instances from my desktop. This allows me to run my PowerShell or python scripts locally and configure my SFSS instance running in AWS.
3. Two private networks (i.e., SAN A and SAN B) that will allow you test the impact that things like zoning changes, or loss of connectivity to the CDC will have on your hosts ability to access block storage.

The Virtual Lab directory will eventually contain all of the information and configuration steps you’ll need to setup an AWS based virtual lab of your own that is capable of using NVMe/TCP to connect from your host to your subsystem. Much more will be coming in this area.

Take a look at SANdbox today and take what you need or feel free to contribute!

The next post in this series will provide the step-by-step instructions to setup your AWS based NVMe/TCP lab... Stay tuned.

#iworkfordell

Getting the most from your SAN with Dell EMC PowerMax

PowerMax Feature Prevents Congestion Spreading 

The big news in our new white paper is data that proves how a new PowerMax feature namely, initiator bandwidth limits, will allow you to prevent congestion spreading due to oversubscription from happening, even when you need to access your 32G storage from a 4G host.  We believe this feature is an industry first and it allows you to take full advantage of your 32G storage port bandwidth without needing to worry about congestion spreading due to oversubscription. 

Dealing with SAN Congestion - How did we get here?

A few years ago, a large Dell EMC customer approached us with a problem that appeared to be related to a SAN performance problem known as “slow drain”.  As we investigated, it became apparent that we were dealing with a much larger issue; a problem that would eventually become known as “Congestion Spreading due to bandwidth mismatch”.  The key takeaway from our investigation was:

Any time a Host is attached to the SAN at a speed that is slower than the storage it accesses (e.g., a 4G FC HBA accessing 8G storage, 10G FCoE CNA accessing 16G storage, etc), the Host will start behaving as a mild slow drain and cause congestion spreading if it uses large block READs.

Further complicating matters, predicting which hosts would end up using large block reads proved to be impractical.  As it turns out, this behavior is driven by the users of the applications running on each host.  As a result, we realized we were dealing with an unpredictable behavior that could impact the performance of an entire SAN.

After the implications of this performance issue started to sink in, we immediately reached out to our switch partners to start working on solutions.  We eventually settled on a 3-pronged strategy focused on Detection, Prevention and Remediation.

What does all of this have to do with 32G storage?

Well, to avoid the congestion spreading issue I described above, rather than just adding your 32G storage ports to an existing 8G or 16G SAN, you would ideally upgrade the entire SAN infrastructure to 32G.  The problem is, for most customers this kind of fork-lift upgrade would be too disruptive and cost prohibitive. 

Now, this isn’t to say that upgrading a portion of your SAN to 32G is non-essential, you’ll of course want to connect your 32G Storage ports to 32G SAN switches.  But beyond this, determining what other parts of your infrastructure will need to be upgraded can be a bit daunting.  With this in mind, we wanted to make sure that we provided our customers with the information they needed to upgrade exactly what they needed to upgrade and nothing more.  To this end, the Customer Solution Center (CSC) and Connectrix teams decided to collaborate on a new white paper “Getting the most out of your 32G storage”.  In it, we explore several different topologies and provide actual test results to demonstrate the impact of upgrading different parts of SAN to 32G.  From this work we were able to define a spectrum of different solutions that will help our customers to better understand the cost and benefit of the different options that are available to them. 

Download the whitepaper today! 

https://www.dellemc.com/resources/en-us/asset/white-papers/products/storage/h18140-getting-most-out-of-32gfc-storage.pdf

Connectrix in CloudIQ!

Background

Just about 2 years ago I got my first look at CloudIQ and was immediately impressed. 

I mean, who wouldn’t be impressed by a Cloud Native Application that enables our customers to view the Health, Configuration, Capacity and Performance information of their entire inventory of Unity, VMAX, SC, and XtremIO systems from a single pane of glass?

As I explored the early CloudIQ user interface, I remember thinking how amazing it would be if we could view SAN and host information from this interface as well.  I believed this type of ”end-to-end” information would make it much easier to troubleshoot and remediate things like the slow drain and congestion spreading events that have been experienced by many of our customers. 

Little did I know, less than a year later my team would be given the opportunity to add support for Connectrix products into CloudIQ and help make this end-to-end view of infrastructure a reality.

In this post I’ll provide a very high level overview of the data collection process used to support the Connectrix products as well as a couple of the important features that I believe you will find very useful when Connectrix in CloudIQ becomes available at the end of May.

Connectrix in CloudIQ overview

Rather than immediately dive into the different features and views, I’d like to start with overview of the process that we use to collect data.

 

Starting from the bottom of the diagram, you’ll note that we plan to support both B-Series and MDS Series Connectrix products.  I should point out, that we can collect data from any FC switch running FOS 8.2.1a or above or NX-OS 8.3(2) and above (including non-Connectrix models).  However, in order for the switches to be visible in CloudIQ, the product will need to be under an active service contract with Dell EMC.  We validate the contract status using the product serial number.

The actual data collection process is periodically performed by the CloudIQ Collector (~once every 5 minutes).  Each time a collection is performed, the CloudIQ Collector will gather the required health, configuration, capacity and performance data from every switch in the environment via their REST API.  This information is then pushed back to Dell EMC via Secure Remote Services (SRS) and then makes its way to our CloudIQ backend where is it processed and securely stored. 

The CloudIQ Collector itself is fairly lightweight (<700MB of disk space is required) and is based on Dell EMC Storage Resource Manager (SRM).  The collector also has the ability to collect VM information from vCenter.

One final point to note, the switches also have the capability to send alert information directly to our services organization (i.e., call home) and this capability does not require the CloudIQ collector, only SRS. 

Important features

When we first started this project, we wanted to make sure that the information we provided was always accurate, relevant and actionable.  To this end, we solicited a lot of input not only from our Customers, but also from our services team (since they are troubleshooting SAN issues all day) and have a really healthy understanding of what’s relevant when there’s a problem.  Because of this input, you’ll notice a number of features in our first release that are intended to help you troubleshoot in a more efficient manner. 

I’ll review a couple of those features next.

Inventory of all Connectrix products including a product specific Health Score

Once you've started collecting data from your Connectrix products and logged into CloudIQ, you can immediately get a sense of the health of your Connectrix products by selecting “Inventory” from the navigation tree on the left side of the CloudIQ UI and then clicking on the “SAN” tab.  As shown below, each SAN system (switch) will be displayed as a tile that includes the health score associated with that product.

Starting from the top, each tile includes:

• A health score that represents the HW status of the switch. In the short term, you can expect the health score calculation to include whether or not the switch is experiencing congestion. 
• The name of the product (e.g., Production SAN Extension). This is the user-friendly name you assigned to the product in your environment.
• The model (e.g., Connectrix ED-DCX6-4B) and serial number (e.g., EAF300M001).
• The Firmware Version (e.g, v8.2.1a)
• Last Contact Time – The last time this product sent information back to CloudIQ.
• As well as the Location and Site ID where the product was physically installed.

Note: There is also a list view available that provides the same type of information but in a table format.

If you want more detailed configuration information about the switch, you can always click on its name and you’ll be brought to a detailed configuration view as shown below.

You’ll notice that this screen consists of:

• A Summary Header that provides much of the same information as the previous view did as well as the Switch uptime, Management IP Address and the WWNs associated with the product.
• Five tabs that each provide detailed information about the Fabrics, Partitions (VFs or VSANs), Zones, Attached Devices and Components (e.g, FRUs) for this switch.

Finally, if you want even more detailed information about your switch or you want to perform active management of a product (e.g., add a zone), you can click on the launch Element Manager link at the top right of the screen.  This will launch a browser pointed to the IP Address of the management interface of the product.  Here you can access the native switch management tool and do whatever you need.  BTW, I LOVE THIS FEATURE!  It’s so simple, yet so convenient.

Capacity

If you’re looking for port level information about your products, start from the left-hand navigation tree and select Capacity -> System Capacity and then click the SAN tab.  For each product you’re managing in CloudIQ, you’ll see a tile similar to what is being shown below:

The donut graph gives a visual representation of the number of ports on the switch as well as the number of ports that have been consumed (e.g., are online) or are in need of attention (e.g., in an Error state).

If you’d like additional detail, click on the switch name and the Detailed Capacity view will be displayed as shown below.

The upper left side of the user interface provides the same information as the previous “System Capacity” view did.  However, depending on what type of interface you have selected (i.e., Online), the other views to the right as well as the table below the donut graph will be filtered accordingly.  This filtering can be combined so that you can limit what is displayed in the table view (at the bottom) to show only Online, F_Ports, that have registered as a storage port (for example).

Performance

If you’re looking for performance information about your products, start from the left-hand navigation tree and select Performance -> System Performance and then click the SAN tab.  For each product you’re managing in CloudIQ, you’ll see a tile similar to what is being shown below:

This view contains:

• System Bandwidth – This represents the average amount of data being transmitted by the switch over the past 24 hours. You’ll also note the graph at the bottom of the dialog provides a historical view of the amount being transmitted during each sample interval.  Note, we’ve already been asked to update this to include a metric that represents what percentage of total system bandwidth is being consumed and we plan to do this in an upcoming relase.
• Utilization >= 80% - Number of switch interfaces that were running at greater than or equal to 80% utilization when the last collection was taken from the switch.
• Congested - Number of switch interfaces that were congested (had a congestion ratio greater than or equal to 0.2) when the last collection was taken from the switch. NOTE: This is a REALLY COOL FEATURE.  If you don’t know how to calculate the congestion ratio or why it’s important, see our recent white paper on Congestion Spreading and how to avoid it.
• Errors - Number of switch interfaces that met one of the following criteria when the last collection was taken from the switch:
  • A change in an interface error counter of greater than or equal to 100 occurred between the current sample and a previous sample.
  • Greater than or equal to 50% of the samples from a given interface over the past hour have incremented an error counter.
  • Greater than or equal to 50% of the hour intervals from a given interface over the past 24 hours have met either of the above criteria.

• Link Resets – Similar to Errors (above) but with specific criteria related to the reception of Link Reset primitives.

After looking through all of this information, you’d like to see more detailed performance information, you can click on the switch name and a number of graphs showing detailed performance information will be displayed.  One of these graphs related to congestion is shown below.

Currently, this graph displays a number of data points each representing the sum of “time spent at zero transmit credit” across all interfaces during each sampling interval.  This can be useful when trying to understand if you’ve had a SAN event that could have impacted overall performance.

What’s next?

While we believe we’ve been able to provide some useful functionality, we also know that there’s a lot more we could do going forward.  I expect we’ll be working in three areas:

1. Customer originated enhancement requests. We built this tool with our end users in mind, so if you think we’re missing something or have a great idea for a feature, let us know! 
2. Predictive Analytics related to optic health. Shortly after we GA, we will start looking into how to predict optic failure based on TX and RX optical power, bit errors and other link events.
3. More granular performance information. We would like to make it possible for you to zoom into a chart (e.g., the Congestion chart above) and allow you to easily view the top contributors. 

All that said, we’re just super excited to reach this milestone and we look forward to helping you get the most from your SAN.

Let us know how we can help!

Thanks for reading!

 

NVMe over Fabrics’ Discovery problem

TL;DR – Today, Fibre Channel provides an NVMe over Fabrics Discovery mechanism, Ethernet does not.  

This blog post describes an investigation that was performed to determine if an enhancement to the NVMe over Fabrics Discovery protocol was needed in order for it to support a Network Centric connectivity model as described below.  The investigation determined that a multi-layer discovery technique was called for and that FC already provided the required Discovery mechanisms; Ethernet currently does not.

Before I continue I need to thank Dave Peterson (Broadcom) for his assistance with the FC Discovery portion of this investigation.   

Management models

In order for NVMe over Fabrics (NVMe-oF) to be widely adopted, I believe it must integrate into existing workflows and management tools (e.g., provisioning and monitoring) that existing enterprises use when managing external (e.g., Array based) storage.  As of today, there are three primary approaches that can be utilized for this purpose:

1. A Centralized Infrastructure as a Service (IaaS) Management and Orchestration (M&O) layer. OpenStack is one example of an open source project that provides this functionality, but some users are opting to write their own custom M&O Layer by making use of workflow orchestration frameworks (e.g., Stackstorm), while leveraging tools (e.g., Ansible) and/or extensions to languages (e.g., Pyvmomi).  This type of approach is typically used by larger customers with extensive development resources.  This type of approach can be used with both the “End-node Centric” and “Network Centric” provisioning models described below.  I should also point out that the IaaS Missing Link is something that complicates this kind of approach.   
2. The “End-node Centric” management model. This approach requires every host and storage resource to be individually configured and is primarily used in Small and Medium Business (SMB) environments to manage access to storage when NFS/CIFS or iSCSI are being used.  The benefit to this approach is “simplicity” and the fact that a centralized discovery service does not need to be installed nor configured.  The downside to this model is that it doesn’t scale very well; as the number of nodes increases the administrative burden increases non-linearly.
3. The “Network Centric” management model. This approach utilizes a centralized Discovery Service to enable Hosts to discover the Storage ports that are available to it.  This type of approach is used in Medium and Large Enterprise environments to manage connectivity in their Fibre Channel (FC) Storage Area Networks (SANs).  The benefit to this approach is that it scales fairly well (10k+ ports with FC today) and supports State Change Notifications.  The downside is a centralized service must be installed and in some cases configured.  Another limitation is the requirement that all host and storage subsystem ports register with the centralized Discovery Service.

For more information about the “Network Centric” and End-node Centric” management models as well as the scalability attributes of each, please refer to FC and FCoE versus iSCSI – “Network-centric” versus “End-Node-centric” provisioning

Since NVMe over Fabrics will need to scale (e.g., beyond a single rack) I believe it makes sense for the Transports to at least support a “Network Centric” management model.

In addition, since NVMe over Fabrics will need to co-exist with SCSI and SCSI-FCP, it makes sense to follow a layered discovery approach that is similar to what has been done for discovery with those protocols.   

The Network Centric management model

In order to support the “Network Centric” management model, this blog post breaks discovery into two different layers; NVMe and Transport Specific.  This separation will allow for the existing NVMe Discovery process to remain intact while allowing each Transport to leverage existing transport specific approaches for the discovery of Transport Addresses that support a specific feature (i.e., a Discovery Service).  This concept is illustrated in the diagram below. 

The Network Centric approach and how the layered approach will impact each transport is described below.

Transport Specific Discovery

In order to automate the discovery of end devices (e.g., Arrays) that support NVMe over Fabrics, I believe a “centralized”, Transport Specific, Discovery Service is needed.  Furthermore, this Discovery Service will need to be able to provide a list of Transport Specific addresses that provide an NMVe Discovery Controller (e.g., that reside on each NVMe Subsystem).

Side note: Transport Specific Discovery is totally optional.  Outside of expected behavior, the implementation details regarding a particular Transport Specific Discovery Service will probably not be defined in the NVMe over Fabrics specification.

Although this may seem obvious to some of you, it’s important to explicitly state that each of the protocols would need to handle Transport Specific Discovery in a Transport independent way, for example:

Fibre Channel

The Transport Specific Discovery Service for FC is the Name Server and is defined in the Fibre Channel standard FC-GS.  The Discovery Service capability is returned to the Host port in response to “Get N_Port Identifier for the specified FC-4 Feature” (GID_FF) with the FC-4 feature of 28h specified and the FC-4 Discovery Service Feature bit set to 1.  When the FC Transport Discovery process is complete, the Host port will have a list of all FC WWPNs and N_Port IDs that provide an NVMe Discovery Controller.  This requires that each VN_Port properly register its FC-4 features.

IP

Transport Specific Discovery for IP could be provided in a number of ways, including:

• manual configuration,
• via a configuration management tool (e.g., Ansible, Puppet), or
• a centralized service (as described below) could be deployed and then discovered via DHCP:

1. The location of the Transport Discovery Service for IP (e.g., iSNS could be enhanced and used) can be discovered via DHCP.
2. The Hosts and Discovery Controllers will need to register their NQN, Transport Address and Discovery Service capability with the Transport Discovery Service. These registration commands still need to be defined.
3. The Hosts and Discovery Controllers will optionally be allowed to register a Discovery Domain (Similar to what is defined for iSNS) with the Transport Discovery Service. The purpose of the Discovery Domain is to limit the number of Discovery Services that each host can access and this serves two purposes:
   1. It helps avoid scalability issues related to “Discovery and RSCN Storms”
   2. It provides a coarse grained access control mechanism similar to the role that zoning plays in a FC SAN.
4. The Hosts and Discovery Controllers will be allowed to query the Transport Discovery Service to determine the transport addresses that are available and of those which of them support a Discovery Service. The commands used to query the Transport Discovery Service will also need to be defined. 

IB

For IB I’m going to let Mellanox figure this one out.  :-)

NVMe Registration and Discovery

After the Transport Specific Discovery process has completed, the Host Software can use the Fabric Connect command and specify the well-known Discovery Service NQN as the destination.  For each Transport Specific address that registered as a Discovery Service and was discovered via the Transport Specific Discovery service, the host software will obtain a Discovery Log page as defined in section 5 of the NVMe over Fabrics 1.0 specification.

As shown in the diagram above, this Discovery Service can either be physically located in the same enclosure as the Controller used for IO (e.g., within the same Array enclosure) or it could be centrally located and accessible over an out-of-band network connection (e.g., to support a rack of JBOF).  In order to support both of these cases, the information that is registered with the NVMe Discovery Service must allow for a Discovery Log page to be fully populated.  Since this is the case, the Register Discovery Log Page command should contain one or more Discovery log pages. 

The Register Discovery Log Page command can be used to directly register a log page but other methods not defined in the NVMe over Fabrics specification may also be used (i.e., when the Discovery Service is located within a storage array).

The format of the Register Discovery Log Page command is also TBD.     

State Change Notification

With FC, State Change Notification has proven to be amazingly important for operational concerns.  Essentially, you need to know when a device goes away ASAP and RSCN allows for this. 

Similar to the approach used with Discovery, the mechanisms used to provide State Change Notification should also be layered into Transport specific and NVMe specific.

Transport Specific

Each Transport may support a mechanism to indicate that a change has occurred in the transport that may affect connectivity.

Fibre Channel – Fibre Channel provides Asynchronous state change notification via Registered State Change Notification (RSCN)

IP – I am honestly not sure what could be done here short of adding this functionality to an iSNS like centralized controller. 

IB – Again, I’ll defer to Mellanox.

NVMe Specific

State change notification can also be provided between NVMe host software and each controller through the use of the following approaches:

• Asynchronous Notification for optional events as defined in NVMe over Fabrics 1.0.
• Keepalive based monitoring between the NVMe host software and each controller.
• A combination of keepalive based monitoring between the NVMe host software/Controller/Discovery Controller and the Transport Discovery Service; with the Transport Discovery Service transmitting an asynchronous event when a keepalive timeout occurs

All three of these approaches will need to be defined.

Conclusion

As we’ve said before, NVMe over Fabrics needs a bit more work before it will be ready for use by the majority of our Enterprise Customers.  Along these lines, I believe that Discovery and State Change Notification are features that will be critically important to you as you start to evaluate different Transports. 

Thanks for reading!