Brass Tacks

Let’s play in an AWS based SANdbox – Part 2 Setting up your AWS VPC and networks

In my previous post, I introduced the SANdbox Github repo and described the work that had been done to create an Virtual Lab that is capable of supporting NVMe/TCP traffic.

In this post I’ll walk you through the steps to create your own SANdbox VPC in AWS that will eventually look similar to the following.

Before I get started, I need to acknowledge two sources of information that were invaluable to me as I was getting started with AWS:

1. The AWS documentation – I found an article that seemed to describe exactly what I wanted to do but I didn’t have enough context/knowledge of AWS at the time to be able to follow it successfully.
2. This “HOW TO Build a Home Lab in AWS for FREE” video by Jon Good. It was incredibly helpful and I highly recommend reviewing it before continuing with the following steps.

Create a Free AWS account

1. Click here to create a Free AWS account.
2. Provide an email address
3. Provide an AWS account name
4. Verify the email Address you provided by taking note of the Verification code they provide you an then entering it into the browser when prompted.
5. Create your Root user password.
6. Provide your contact information
7. Provide credit card information
8. Confirm your identity (I chose to use SMS)
9. Select a support plan (I chose Basic support – Free)
10. Click Complete sign up
11. Click Go to the AWS console
12. Enter or ensure that the correct email address is provided and click Next.
13. Enter the password associated with the email address provided and click Next.

Secure your account with MFA

1. In AWS, under Console Home, click IAM
   1. Note: If you ever lose track of a particular view or service, you can always click the Services icon at the top of the page or search for the service name you want to access and get to it that way. I do this all the time for accessing my “EC2” instances.
2. In the IAM dashboard, click Add MFA.
3. Under Your Security Credentials > Multi-factor authentication (MFA), click activate MFA.
4. Choose the type of MFA device to assign (I chose “Virtual MFA device”) and then click Continue
5. In the “Set Up virtual MFA device” dialog, you will be prompted to:
   1. Install a compatible app on your mobile device or computer (I chose the App “Authenticator” because it was supported on IOS and had good reviews online)
   2. Use the virtual MFA app and your device’s camera to scan the QR code shown within AWS.
   3. Enter two consecutive tokens from the Authenticator APP, then click OK, and you should see “You have successfully assigned a virtual MFA. Click Close
   4. Click on “Dashboard” (left side of the screen) and ensure there is a green check mark next to “Root user has MFA”.
6. Click on the user name at the top right of the screen (e.g., SANdboxBlog) and then click Sign out

Setup a Billing Alert

Note: Jon Good recommended this and I cannot stress how good of an idea it is.  You’d be surprised how quickly you can blow through $10 once you start using non-free instances and Elastic IP Addresses. My only regret is I did not also create a $100 alert :-) … do not make the same mistake I did!  The steps for doing this are very straight-forward and can be found here.

Here’s my alarm configuration after setup.

Create a Virtual Private Cloud (VPC)

The VPC is the top level “container” into which all of your Virtual Resources will be deployed. 

1. To create a VPC, click on Services (at the top left of AWS console screen) and click on (or search for) VPC. Click Create VPC.
2. Under VPC settings, I chose VPC and more.
3. Under Name tag auto-generation, I provided a tag of SANdboxBlog.
4. I decided to use a single Availability Zone as my lab configuration does not require HA.
5. I increased the number of private subnets from 1 to 2.
6. All other settings were left at defaults.
7. The VPC preview should look similar to the following
8. Click Create VPC and the VPC workflow will run. Click View VPC after the workflow completes.

Create and associate Security Groups

Creating and managing security groups is a tedious but necessary task.  It allows you to define the sources (e.g., IP Addresses) that can be used to access the resources inside of your VPC.  I created three Security Groups, one for my LAN (public facing) network and two for my SANs (e.g., SAN A and SAN B). I’ve summarized the SGs I created below for ease of reference.

LAN-Public

Name	IP ver	Type	Protocol	Port Range	Source
SSH	IPv4	SSH	TCP	22	0.0.0.0/0
HTTPS	IPv4	HTTPS	TCP	443	0.0.0.0/0

SAN A and SAN B

Name	IP ver	Type	Protocol	Port Range	Source
mDNS	-	Custom UDP	UDP	5353	0.0.0.0/0
NVMe/TCP Discovery	IPv4	Custom TCP	TCP	8009	0.0.0.0/0
NVMe/TCP IO	IPv4	Custom TCP	TCP	4420	0.0.0.0/0
PING	-	All ICMP – IPv4	ICMP	ALL	0.0.0.0/0

Create the “LAN-Public” Security Group

1. From “Services”, click on EC2. (Not really required but I like to start from the EC2 dashboard for everything)
2. On the left side of the console scroll down to “Network & Security” and click on Security Groups and then Create Security Group.
3. Add a Security Group Name (e.g., LAN-Public)
4. Add a Description (e.g., Allows SSH and HTTP access to instances)
5. VERY IMPORTANT Ensure the correct VPC is selected. The default VPC listed in the VPC field is probably not what you want. Click on the ‘X’ in the right side of the VPC field and ensure you have selected the correct VPC. In my case I selected the SANdboxBlog-vpc. If you haven’t already done this, take note of the last four digits of your VPCs instance ID.  In my case it’s “a245”, you will use this later on. Also, I highly recommend creating a diagram of your environment to keep track of all of the instance IDs. A completed example diagram will be provided at the end of this post. 
6. Create Inbound rules – As described in the LAN-Public table above, we need to create rules that will allow for both SSH and HTTPS access to our instances.

 SSH

1. Under Inbound rules, click Add Rule
2. From the Type pulldown, select SSH. This will automatically populate the Protocol and Port Range fields for you.
3. From the Source pulldown select Anywhere-IPv4. The IP address of 0.0.0.0/0 will automatically be added to your list of sources.

NOTE: Using Anywhere-IPv4 is NOT a best practice. Ideally, you would specify the IP Address of the system you will be using to access your VPC.

HTTPS

1. Click Add Rule
2. From the Type pulldown, select HTTPS. This will automatically populate the Protocol and Port Range fields for you.
3. From the Source pulldown select Anywhere-IPv4. The IP address of 0.0.0.0/0 will automatically be added to your list of sources.

Confirm that your security group looks similar to what is shown below and then click Create Security Group.

Once the SG was created I took note of the Instance number for the LAN-Public Security Group (i.e., 4872) and added this to my topology diagram.

Create the “SAN A” Security Group

1. On the left side of the console click on Security Groups and then Create Security Group.
2. Add a Security Group Name (e.g., SAN A)
3. Add a Description (e.g., SAN A is a private network for NVMe/TCP Discovery and IO traffic.)
4. VERY IMPORTANT Ensure the correct VPC is selected. The default VPC listed in the VPC field is probably not what you want. In my case I selected the SANdboxBlog-vpc and it has an instance ID of “a245”. 
5. Create Inbound rules – As described in the SAN A and SAN B table above, we need to create rules that will allow for mDNS, NVMe/TCP Discovery, NVMe/TCP IO and Ping to be used between the instances in our VPC.

 mDNS

1. Under Inbound rules, click Add Rule
2. From the Type pulldown, select Custom UDP. This will automatically populate the Protocol field for you.
3. In the Port range field enter 5353
4. From the Source pulldown select Anywhere-IPv4. The IP address of 0.0.0.0/0 will automatically be added to your list of sources.
5. Provide a description (e.g., mDNS)

NVMe/TCP Discovery

1. Click Add Rule
2. From the Type pulldown, select Custom TCP. This will automatically populate the Protocol field for you.
3. In the Port range field enter 8009
4. From the Source pulldown select Anywhere-IPv4. The IP address of 0.0.0.0/0 will automatically be added to your list of sources.
5. Provide a description (e.g., NVMe/TCP Discovery)

NVMe/TCP IO

1. Click Add Rule
2. From the Type pulldown, select Custom TCP. This will automatically populate the Protocol field for you.
3. In the Port range field enter 4420
4. From the Source pulldown select Anywhere-IPv4. The IP address of 0.0.0.0/0 will automatically be added to your list of sources.
5. Provide a description (e.g., NVMe/TCP IO)

PING (this is not technically necessary, but you will thank me later).

1. Click Add Rule
2. From the Type pulldown, select All ICMP – IPv4. This will automatically populate the Protocol and Port range fields for you.
3. From the Source pulldown select Anywhere-IPv4. The IP address of 0.0.0.0/0 will automatically be added to your list of sources.
4. Provide a description (e.g., PING)

Confirm that your security group looks similar to what is shown below and then click Create Security Group.

Create the “SAN B” Security Group

Using the information from the SAN A and SAN B table above, repeat the steps outlined in “Create the SAN A Security Group” and create another Security Group for SAN B.

OR... You can save a ton of time and do the following

1. Under Actions, at the top of the “SAN A Security Group” configuration screen, you can chose “Copy to a new security group”.

VERY IMPORTANT After providing a “Security group name” (e.g., SAN B) and a Description, you will still need to ensure the correct VPC is selected. The default VPC listed in the VPC field is probably not what you want.  In my case I selected the SANdboxBlog-vpc and it has an instance ID of “a245”.

2. Click Create Security Group

Create your first EC2 Instances

Amazon Elastic Compute Cloud (EC2) instances can be thought of as virtual servers.  For the purposes of the SANdbox configuration, I’ve chosen to use EC2 instances that are running Ubuntu 20.04.  When I was first getting started and needed to work my way through the process of setting up the environment and familiarizing myself with AWS in general, I chose EC2 instances that were available in the free tier. This means there is no charge up to a certain amount of usage.  I will walk you through doing the same for two instances and will get us to the point where you can connect to each instance with SSH and ping across the network interfaces used for SAN A.  We will not be setting up SAN B right now, just validating connectivity across one SAN is enough (for now). 

1. From the “EC2 Dashboard”, click Launch Instance
2. Provide a “Name” for the instance (e.g., Host1)
3. Under “Application and OS Images (Amazon Machine Image)”, click on ubuntu and then select a particular version. I chose “Ubuntu Server 20.04 LTS (HVM), SSD Volume Type” because it was “Free Tier Eligible”, when we actually create the instances we will use for NVMe/TCP testing, we will make different choices that will have a different cost profile. Leave Architecture set to 64-bit (x86)
4. Under “Instance type”, select micro again because it is Free tier eligible.
5. Under “key pair (login), select Create new key pair
   1. In the Create key pair dialog, provide a Key Pair name, I chose to use SANdboxBlog
   2. Select a “Key pair type” of RSA
   3. Select a “Private key file format” of .ppk
   4. Click Create key pair and a file (e.g., SANdboxBlog.ppk) will be downloaded to your system. You will need this file when we connect to the instance via PuTTY in a subsequent step.
   5. Ensure that your key name (e.g., SANdboxBlog) is shown in the Key pair name – required
6. Under “Network Settings” click on Edit
   1. Under “VPC”, ensure the correct VPC is selected (e.g., SANdboxBlog-vpc)
   2. Under “Subnet”, ensure your public network is selected (e.g., SANdboxBlog-subnet-public1-us-east-1a)
   3. Under “Auto-assign public IP”, ensure it is set to Disable
   4. Under “Firewall (security groups)”, click on “Select existing security group”
   5. Under “Common security groups”, leave it blank
   6. Click Advanced network configuration
   7. Under “Network Interface 1”
      1. Set the “Description” to Public-LAN
      2. Under the “Security groups” pull down menu, choose LAN-Public
      3. Leave everything else at the default setting
   8. Click Add network interface
      1. Set the “Description” to SAN A
      2. Under the “Subnet” pull down menu, select the SAN A subnet
      3. Under the “Security groups” pull down menu, choose SAN A
      4. Leave everything else at the default setting
   9. Note, this instance type only supports two network interfaces, so we will only configure SAN A for now.

7. Under the Summary tab (on the right side of the console)
   1. For Number of instances, specify 2
   2. Then click Launch instance
   3. Click View all instances when the button becomes available.

Take an inventory of your Virtual Infrastructure

1. Now that your EC2 instances are up, ensure that you name them appropriately, I chose Host 1 and Host 2. You can rename them in the EC2 Dashboard by clicking in the Name field.
2. For each Host, take note of at least the last four digits of the instance ID.
3. For each Host, click on the Instance ID and select the Networking tab
4. Scroll down to the Network Interfaces section and take note of the last four digits of each Interface ID (you may need to expand the Interface ID column to see the last four digits). You’ll also want to understand what network each interface is associated with, and the Description field should answer this question for you.
5. You’ll also want to take note of the IPv4 addresses for the SAN A interfaces. We will be using them to verify connectivity in the final step.
6. I highly recommend keeping track these instance and interface IDs because we will use them later. Here’s what my configuration looks like at this point.  Note that the IP address for the LAN-Public interfaces are Elastic 1 and Elastic 2, we will actually assign them next.

Create and Assign Elastic IPs

If I was only using a single network interface per instance, I could have had a public IP Address assigned to each Interface.  However, because this functionality is not supported when you have more than one network interface on an instance, I’ve found the easiest way to get access to my instances is to use Elastic IPs.  Unfortunately, Elastic IPs are not free, so please be advised.

1. Under “Network & Security” select Elastic IPs and then click on Allocate Elastic IP address
2. Click Allocate and you will be returned to the Elastic IP Address screen. Take note of the Allocated IPv4 address.  In my case it is 3.230.foo.bar
3. Select the IP Address (little check box to the left of the IP), click Actions and then Associate Elastic IP address
4. Under resource type, choose Network interface and then click inside of the Network Interface box and select the Public-LAN interface of Host 1. BTW, you need to use the resource ID of the interface and this is why it is so important to keep an updated diagram of your virtual infrastructure as you create it.  In my case the last four digits of the interface on Host 1 that is connected to Public-LAN is 20d1.  Clicked on the correct interface ID and then click Associate.
5. Because you need two Elastic IP addresses, click on Allocate Elastic IP address a second time and repeat the above steps for the Public-LAN interface on Host 2. My second Elastic IP was 34.234.you.too

Connect to your Instances

To complete this next step you’re going to need the SANdboxBlog.ppk file that was downloaded to your system when you were creating your instances.  You’re also going to need PuTTY installed on your system as well. 

1. Open PuTTY
2. In the Host Name (or IP Address) field enter the Elastic IP Address associated with Host 1 (e.g., 3.230.foo.bar)
3. Under connection -> Data enter the default user name for ubuntu in AWS (ubuntu)
4. Under connection -> SSH -> Auth, browse to the location of the SANdboxBlog.ppk file.
5. Click Open
6. Accept the warning about the Host Key
7. And you should be logged into your Host 1 instance.
8. Ping the IP Address of the SAN A interface on Host 2, in my case, this IP Address is 10.0.134.136 and verify that it was successful

Next steps

So far we’ve created a Free AWS account, configured MFA, setup billing alerts, created a VPC with three networks, created 3 security groups and verified basic connectivity to and within our VPC using a pair of free tier servers. 

Next time we’re going to deploy 3 larger EC2 instances, and install:

1. nvme-stas on our Host EC2 instance,
2. SmartFabric Storage Software on our Centralized Discovery Controller (CDC) EC2 instance, and
3. SPDK on our Storge EC2 instance.

Thanks for reading!

Let’s play in an AWS based NVMe/TCP SANdbox!

One of the great things about NVMe/TCP is its flexibility.

Not only can you use the NVMe/TCP protocol to connect hosts to storage systems over an arbitrary IP network topology; as originally described here, you can also run it in a surprising number of operating environments, including:

• traditional enterprise environments that utilize array-based storage solutions (e.g., PowerStore and PowerMax),
• next generation operating environments that utilize Software Defined Storage (e.g., PowerFlex),
• completely new use cases, such allowing applications running on the Edge to access centralized block storage resources, and most recently,
• deploying your NVMe/TCP solution in the Cloud (e.g., AWS) and using the resulting "Virtual Lab" to simply get familiar with an NVMe IP-Based SAN, or more importantly test and validate your infrastructure automation software before you use it in production.

The last item, especially the part about using AWS as a Virtual Lab, needs a bit more explanation.  Along these lines, this blog post series will:

1. Provide an overview of the automation tools being made available in the SANdbox Github repository,
2. Provide the step-by-step instructions required to configure a AWS based NVMe/TCP test environment, and then
3. Demonstrate how you can get started with automating your storage provisioning tasks by using the PowerShell, Python or Ansible scripts that we will provide in SANdbox.

Introducing SANdbox!

SANdbox is a Github repo that provides access to the resources you will need when automating your NVMe IP-Based SAN. This includes:

Documentation

Start here if you’re just coming up to speed on the concept of an NVMe IP-Based SAN.  You’ll find links to an IP-Based SAN introductory level blog post as well as links to two detailed SNIA presentations describing how Discovery Automation works at a protocol level. 

There’s also a link to Dell’s SmartFabric Storage Software (SFSS) deployment guide. It covers not only how to install SFSS, but also the basic network topologies that are currently supported as well as step by step configuration examples.

Centralized Discovery Controller (CDC) Downloads

If you’ve been following the NVMe IP-Based SAN space for a little while, you probably already know that a group of companies identified a scalability problem related to the management of IP-Based SANs and decided to work together to solve it.  The result of this work can be found in a couple of NVM Express Technical Proposals (i.e., TP8009 and TP8010). 

• TP8009 describes how hosts can automatically discover subsystems using mDNS or DNS.
• TP8010 defines the concept of a Centralized Discovery Controller (CDC) that limits the number of storage subsystem interfaces visible to each host. As most people who have dealt with FC SANs can attest, this functionality is very important as environments scale up.

To help potential IP-Based SAN users get familiar with the usage of a CDC, Dell has made our implementation of a CDC, “SmartFabric Storage Software (SFSS)” available for download for trial purposes. Please note, this version is for trial and test purposes only and no support is available. If you want the latest, fully featured and fully supported version of SFSS, you need to purchase a license.

If you don’t want to download and install SFSS, either on your own infrastructure or on AWS, you can always give our Interactive Demo a try.     

Toolkit

Once you’ve installed SFSS and understand the basics of interacting with it, you’ll probably be interested in learning how to automate the configuration steps and interact with your infrastructure programmatically. To this end, Dell has provided sample scripts that will help you learn the basics.  Currently we’re providing PowerShell and Python examples and will be adding Ansible soon.

Both the PowerShell and Python scripts will allow you to:

1. connect to the SFSS via the REST API,
2. retrieve a copy of the name server database,
3. add all of the NS entries to a single zone, and then
4. activate the zoning configuration.

The experienced SAN admins amongst you will immediately recognize that zoning all of the ports together is NOT something that would typically be done! However, the example will give you enough information to do something a bit more realistic, such as allowing hosts to access a subset of the available subsystem interfaces.

Virtual Lab

If you’re interested in experimenting with an NVMe IP-Based SAN but you don’t have the HW infrastructure to do so, you can run all of it in AWS and experiment with it there on your own! The configuration I’m experimenting with is shown below.

In my AWS configuration I've created:

1. Three EC2 instances (Host 1, SFSS, Storage) all running Ubuntu 20.04
   1. The Host instance utilizes the Dell maintained open source nvme discovery client “nvme-stas”
   2. The SFSS instance is running our SFSS application and it handles all aspects of discovery automation
   3. The Storage instance is running either nvmet or our “Dell End Point Simulator” that allows you to simulate pull and push registration from the command line.  
2. A public facing network that I am using to access the instances from my desktop. This allows me to run my PowerShell or python scripts locally and configure my SFSS instance running in AWS.
3. Two private networks (i.e., SAN A and SAN B) that will allow you test the impact that things like zoning changes, or loss of connectivity to the CDC will have on your hosts ability to access block storage.

The Virtual Lab directory will eventually contain all of the information and configuration steps you’ll need to setup an AWS based virtual lab of your own that is capable of using NVMe/TCP to connect from your host to your subsystem. Much more will be coming in this area.

Take a look at SANdbox today and take what you need or feel free to contribute!

The next post in this series will provide the step-by-step instructions to setup your AWS based NVMe/TCP lab... Stay tuned.

#iworkfordell