Server storage is a complex subject, storage fabrics, arrays, InfiniBand, SAS, Storage Area Networks SAN, Network Attached Storage NAS, Fibre Channel, connectors, cables and so much more. We will quickly review why storage networks focus on bandwidth, reliability, redundancy, and latency much more than data networks. We will look at disk media: SATA, SAS, NVMe, and Ethernet SSDs. Review server storage practiced architecture and avoid single points of failure.
Hello, my name is Lowell Vanderpool and this channel is dedicated to IT students,
IT professionals, and anyone who enjoys learning technical subjects.
This presentation is going to cover the complex world of server storage.
There are some important takeaways as we delve into the subject of server storage.
One, the more advanced the storage, the more complex it gets. There is no cheap
server storage solutions. They're all expensive. InfiniBand with NVMe is the king of speed
concerning server storage. Latency is the enemy of storage. So, let's say I have a 10-gigabit
connection to a storage solution and I have another 10-gigabit connection to a server storage.
This has one millisecond of latency. This has 100 milliseconds of latency. This application
is going to run much, much better. This one, even though they're both 10-gig connections,
the latency rules the day. NVMe is radically forcing changes in enterprise and data center
storage. HP and Dell and a lot of small storage vendors offer some incredible storage options,
but at mind-numbing prices. Storage technologies have a complex set of connectors,
cables, terminology, making it a very tough subject to learn.
If you want to learn a lot about storage, go to cina.org. Two important drivers in storage speed
are, one, we're decoupling servers from the storage they need. We've got massive servers
with virtual machines and containers on them, but their storage is somewhere else. And two,
enterprise apps are demanding more and more performance. That is demanding more and more
storage speed. Enterprise applications are driving faster, but more expanding storage. ERP
applications, databases, CRM applications, real-time analytics, productivity applications,
and collaboration apps. The two massive problems facing server admins are massive storage data
needs, petabytes, and faster storage hardware subsystems. We would like everything to be like
direct-attached storage or DAS. This year alone, over 56% of IT organizations have over one petabyte
of data versus only 23% in 2017. When it comes to server storage, get ready to spend some serious
cash. When we look at PCs, laptops, and servers, we break all their hardware down into subsystems.
Now, the only thing that I don't have showing here is GPU, which is very important in laptops
and desktops. It's growing and it's important in servers, but just for now, we'll ignore GPU.
We have the network subsystem. We have our CPU, memory. We have storage. Each of these play a very,
very important role in the performance of your hardware. Now, I said storage is expensive,
but this is about as cheap a storage appliance as you will ever look upon. This is a one-use
storage rack. It holds about 34 3.5-inch up to 14-terabyte hard drives. That would give you a
total capacity of about 476 terabytes. This type of storage unit is often used by Google for your
Gmail, for your Google Drive, for your Google Photos. Facebook uses these type of storage
appliances. And Microsoft, for their ConsumerOutlook.com, they use these types of storage
appliances. This is as cheap as you're going to go. You can go to almost any server vendor and
customize a 2U rack server. And with a few clicks of a mouse, you can put two AMD EPYC CPUs, each
with 64 cores in each processor, 128 gigs of RAM, a SAS RAID controller, 24 1.8-terabyte hard drives,
and you can end up with a $48,000 Dragster engine. This is an enormous amount of compute horsepower.
Now, many servers allow what's called Expanding DAS or Direct Attached Storage. Take a look at
the picture in the center. I've got a server down below, but it's got a limited amount of storage
drives. But with this particular model, I can add an enclosure that I can stack on top up to 96
hard drives. I can put them on the top or below the server. And if you'll look on the right-hand
side, you can see how I can take a SAS external connector and cables and daisy chain those servers
and storage enclosures together. Some vendors support up to 240 hard drives. That's a lot of
storage. They're very fast, and they have very low latency. Problem with this is it's got plenty
of storage, it's fast, low latency, but it's a single point of failure. Anything goes wrong in
this chain, the application comes down. So, you've got over $100,000 invested in this server
storage scenario, and you still have a single point of failure. This makes any server administrator
not sleep well at night. Now, take a look at this architecture. This is a classic application
design. I've got two virtual servers that are running, say, an Exchange application. I also
have two passive servers that are running an Exchange, but they are passive right now. Between
them is a network connection that's basically a heartbeat connection. Then below, I've got on each
site a shared pool of storage, a network connection between those shared pools of storage that
replicates and keeps the data consistent and the same. If my active Exchange servers die,
the heartbeat fires up the two passive servers, and my application never goes down. This provides
me shared storage, failover, and redundancy. With this design, your network administrator can go to
sleep at night. The simplest definition of a storage area network is a shared or pooled set
of storage that's used by more than one server. When you remove your storage away from your
servers, you have to design the network between them very carefully. For example, a standard
Ethernet switch can add too much latency between your server and your storage. If you're connecting
a network card to your server that's going to connect to the storage, you need a 25 to 50
gigabyte network card to connect you to your storage. Network bandwidth is very important,
especially as we get into NVMe storage arrays. Take a look at my chart on the left. If I have
one NVMe hard drive and I'm transferring data across the network, I need at least three gigabits
of bandwidth to transfer that data. If I've got two NVMes that are transferring data across the
network, I need at least six gigs. If I've got a storage array and I've got four NVMe hard drives
transferring data across the network, I need at least 12 gigs of bandwidth to have that happen.
If I've got a 10 gigabit NIC on my right, I'm going to have a bottleneck. When I move my storage
remotely, I have to consider my network cards, my cables, my switches. Everything between the
server and the storage has to be thoughtfully designed. As you start looking at storage,
you begin to see the term HBA. What is an HBA? Now, HBA stands for Host Bus Adapter. Now,
if you have InfiniBand, it's called HCA or Host Channel Adapter. HBA is simply a storage
controller. Most PCs and laptops, you don't add HBAs because it's built into the south bridge
of every desktop and laptop. You have a storage controller already. But in servers, they often
only come with, say, a SATA controller, which no one uses. And you have to buy these HBAs or
storage controllers for whatever scenario you're going to use for your storage. Now,
if you have a server that you're going to use direct-attached storage, you may purchase an
optional SAS controller. And you'll almost always want RAID functionality with that
SAS controller, such as this Broadcom MegaRAID 9500 series. This rents about $1,000 for this
HBA or storage controller with RAID. You can run probably 95 to 240 hard drives off that
controller. Now, if your server is going to connect to a fiber channel storage area network,
you're going to need an FC storage controller, such as this Dell Sandblade. It has two ports,
runs about $1,400. Now, if you've got a very powerful server, you've got virtual machines,
you've got containers, and you're going to connect to remote storage, you may have to purchase two
of these, maybe even three of these HBAs to get the bandwidth that you need to your storage.
If your server is going to connect to an InfiniBand storage area network, you're going to
have an HCA controller, such as this Mellanox 200 gigabit PCI 4.0 16-lane. This runs about $1,652.
Again, depending on what you have running on the server and what you need for storage,
you may need two of these controllers to get the bandwidth you need to your storage.
Now, if your server is connected to an iSCSI storage, you're going to need an iOS iSCSI
controller adapter. There's a Dell 10 gig iSCSI controller and it runs about $3,000.
Now, this is interesting. Windows 10 and 11 have a free iSCSI controller built in. It's
called an Initiator. Windows Server also contains this same software, but a lot of times that's
going to take CPU time in order to do that same feature via software. So, many people choose to
add the hardware controller to offload their CPU, let the controller do the iSCSI work,
free up their CPU. Now, if your server is connecting to a NAS storage, network-attached
storage, you don't need a controller. You just need a good NIC because the NAS device will have
a storage controller built into the NAS device. Now, if I want to launch my iSCSI I0 controller
that's software on my Windows 11 or Windows 10, I go to my start menu. At the search bar,
I'm going to type in iSCSI. Notice it pops up and shows me the iSCSI Initiator app. So,
I'm going to open that. And this application is going to allow me to connect with the proper
information, the IP address of my storage array, my iSCSI target. They call that a target. That's
got all my hard drives on it. So, I can put in the IP address and it will connect this Windows 11
that remote storage array. So, when we start removing our storage away from our servers,
one of the nice features of that is that it's more expandable. We can choose different kinds
of networks to put between our server and our storage, such as Fibre Channel, Ethernet,
or InfiniBand. The most simplest definition of a storage area network, which is remote storage,
is that it is storage that can be used by more than one server and applications. It's very
resilient to failure. We have to be careful about adding latency in that network, make sure that we
have plenty of bandwidth, and it is much more complex to manage that storage. And it's very
expensive. When you put your servers and your storage apart remotely, you're going to choose
a couple of networks. You're going to use InfiniBand network, Fibre Channel network,
or Ethernet. Storage admins call these networks fabrics. So, here's an example of HP's Ethernet
storage fabric. They've got their servers here. They've got their storage over here. They choose
very carefully NICs, switches. Everything is chosen carefully to get the highest bandwidth,
lowest latency. Storage fabrics also include protocols. Here's the OSI layer of InfiniBand,
and it looks very different than what is happening in traditional Ethernet. Here is an InfiniBand
fabric where we have our servers, our storage, our switches, and the cables, connectors that
make InfiniBand. InfiniBand is the choice for very high bandwidth and very, very low latency.
Fibre Channel is one of the most popular networks to put between your servers and your storage.
Fibre Channel is well-established in the enterprise. It's fast. It does require more skills
for the network administrator. It's complex. It requires special switches that build redundancy.
Typically, most storage area networks use Fibre Channel. Here's a great picture that shows who
is supporting Fibre Channel. We see the various operating systems, various server vendors,
what companies build the host adapters, Fibre Channel switching, and the disk flash storage
array systems for Fibre Channel. Now, we can also choose to connect our servers to storage using
Ethernet. Ethernet is growing in its popularity between servers and storage. You have to watch
out for latency with Ethernet. It's fast. iSCSI uses Ethernet. Fibre Channel can be run on top
of Ethernet. There's a protocol called NVMe over TCP. There's also remote DMA, which we'll talk
about a little bit later, over converged Ethernet. It's definitely cheaper to build a network with
Ethernet, but you have to watch out using standard switches because of latency. If you use iSCSI,
iSCSI commands are encapsulated in an Ethernet frame. They're actually routable. You can send
them across to WAN. Fibre Channel over Ethernet, Fibre Channel frame is encapsulated in the payload
of the Ethernet. It's not routable. It works well on a local area network. You can also use CAN
network adapters, converged network adapters that have both Fibre Channel driver and an Ethernet
driver. Common types of remote storage are NAS. Typically, they're slower and less expandable.
They typically depend on Linux file systems and RAID. Then we have the storage area network.
They're complex. They're failover. They're resilient. They're expensive. They're expandable.
And then you also have storage as a service. You can go to Google for cloud storage, AWS,
Azure, and many more. Remember, in the world of storage, speed is the key. And one of the
technologies that's happening in server storage is RDMA, the ability of an application on a server
to transfer data to an application on another server without the use of the CPU or the operating
system. This is called RDMA. When I can take data from one application and push it across
the network to another application without the CPU and without intervention of my operating system,
I get speed. So RDMA is very attractive to many developer and applications.
So let's take a look at server storage media. First, SATA drives. They're still used in servers.
They're cheap. They're slow. They have huge storage capacity, which is why we're still
using them. They're very good for archival storage. They need either hardware RAID or software RAID.
3.5-inch is the preferred form factor on server storage. Think of SATA moving data on and off
as a two-lane highway. It doesn't have the best transfer rate. SAS drives are very popular on
servers. They have medium performance. They're very expensive. They only have medium storage
capacity, nowhere near what you can do with SATA. They can run as high as 15,000 RPM. They have a lot
of problems with heat and power dissipation. They either need hardware RAID or software RAID,
and their preferred form factor is 2.5, not 3.5. Think of SAS as like a four-lane highway.
If you haven't been up on RAID, I've got three videos that I'm showing you right now. You can
take a look on my channel. They will bring you up to speed on a lot of basic RAID concepts.
Of course, another storage media is NVMe. They're expensive. They're fast. They have relatively low
storage capacity, very low power and heat dissipation. RAID can boost their performance
almost to the point of RAM speeds. Many enterprise form factors are used. M.2 is not used for servers.
It's a consumer form factor. Think of NVMe as a 12-lane highway. This is a nice little chart that
allows you to see the different kinds of media and their latency at the high end, their latency
at the low end, and then IOPS, which is Input Output Operations Per Second, here on the very
right-hand column. You can look at hard drives, solid-state hard drives, NVMe, and there at the
bottom is one that's called PM or Persistent Memory. When flash gets as fast as DIMM RAM,
which is what Intel is doing with their Optane Flash Memory, you begin to see the latency down
at the very bottom. You see the PM row, Persistent Memory. You can begin to see latency and our Input
Output Operations Per Second. Now, when enterprise uses NVMe, they typically use many other types of
form factors. If you look on the bottom, we see the M.2, which is typically what we put in our
desktops and laptops. Enterprises don't want to use those because they need more heat dissipation,
more memory capacity, so they have a tendency to use these other form factors in their storage
arrays. Now, the last storage media that we're going to talk about is something that's recent.
It's called Ethernet SSDs. And basically, we're going to take an SSD and directly connect it to
a 10-gig switchport. We're going to use open-source storage management software. There's no single
point of failure. It's expandable, and we can get massive storage capability. This is an enclosure
for Ethernet SSDs, and it basically provides power and cooling and a direct connection to a 10-gig
Ethernet switchport. Every drive gets a 10-gig port. We talked about how enterprise applications
want fast storage. As storage latency goes down and the speed of storage goes up, we can begin
to substitute it for memory. And that's exactly what Intel is doing with their Optane flash memory
is they're actually taking it and building what's known as persistent memory. Whenever you can get
flash storage to reach near the speeds of DRAM chips, you can begin to take that storage and
make it look and act like RAM, which is exactly what Intel is doing with their Optane flash memory.
Another important concept in storing data in today's server environment is we can store that
data in blocks, in files, or in objects. Now, block storage is simply, as it says,
we're going to store data in blocks. Databases do this and typically storage area networks.
File storage is very familiar to most of us. That's typically what we use on our operating
systems in Windows and Linux. It's hierarchical in structure. We have folders. Network-attached
storage typically uses a file storage file system to store data on its storage arrays.
Another way to store data is objects. It's a growing trend with container applications
to use object storage. Now, Azure calls it blobs. Go figure. Developers can use an HTTP API with any
kind of client operating system or programming language. The cost structure means you only pay
for what you use. Doing it this way, the developer does not have to worry about maintaining hard
drives or RAID arrays. It's simply handled by the cloud provider, such as Google, AWS, or Azure.
So, what is Microsoft doing to address the issue of super expensive storage area networks and
network-attached storage arrays? Well, one, they're approaching it with Storage Spaces Direct.
It's shipped with Server 2016. This type of storage is highly available. It's scalable.
It's software-defined storage, and it's at a fraction of the cost of a typical SAN or NAS array.
So, how does this work? Well, first, we're going to run a PowerShell to cluster these two servers
together. Then there's a number of ways you can do it. You can take all the storage, except maybe
the two hard drives that you have the operating system running on it, or divvy up your storage
however you want. But take whatever amount of storage you want on each server and dedicate that
to Storage Space Direct. Now, you can do that either through GUI, or you can do it all through
PowerShell. And you basically pool all that local storage together so that both servers can see it.
Now, once you have pooled all that storage together, you can start carving up volumes,
and you can put virtual machines in those volumes, or you can do whatever you want.
The key to any of this type of Storage Space Direct system, especially with standalone servers,
is very, very fast Ethernet connectivity and low latency between the servers. Once you've
created your volumes, you've assigned them to VMs or to extra storage, however you've done
your storage. If you lose a disk, it doesn't matter. If you lose a server, it will still run.
If you want to grow this, you can simply expand your storage, your local storage,
if that server will support that, or throw in more servers into the cluster.
Now, how complex and difficult is this to do? I watched Microsoft do this with three servers. It
took them five minutes to build, very similar to what I just shared. I'll put a link below
to the five-minute video that Microsoft demonstrates Storage Space Direct. Pretty impressive.