Understanding Server RAID requires a deep dive into concepts and terms. IOPS, latency, queuing, queue depth, kernel or controller-based RAID are some of the concepts to be mastered.
Benchmarking storage arrays leads us to a deeper understanding of RAID types, failures tolerated, and why Storage Spaces and ReFS are different. ZFS a Linux alternative to controller-based RAID is growing. Understand the importance of “rebuild time”. Learn the importance of disk, controller, and operating systems caching. Why do controllers make sense for many RAID configurations? A look at nested RAIDs like RAID 10 and 60.
Hello, my name is Will Vanderpool and this channel is dedicated to IT students, IT professionals,
and anyone who enjoys learning technical subjects.
Most of us who have had IT training have had to pass an A-plus exam, an F-plus exam, some
basic certification exams, and in those exams they teach you a little bit about RAID or
they test you on your basic knowledge of RAID.
But the minute you walk from training into a production environment, your knowledge about
storage has to totally change.
You've got to start understanding the fundamental concepts, really understanding the terms and
what they mean in order to build fault-tolerant, efficient, and effective storage for your
servers.
We're going to begin that journey today.
So one of the first questions we want to ask is, why RAID?
Well, it's all about keeping that application on that server running all the time.
Your application uptime is the most critical thing that you have responsibility for.
A typical SQL database can run a 48-disk array knowing that a hard drive is your major point
of failure.
Your service level agreements with your business units, you have to meet those, and data loss
is so expensive.
Disk arrays are growing in size.
It is very common to find an application that's running on a 192-terabyte storage array.
If you've been responsible for a production environment and you have experienced losing
data, especially your co-workers' data, you will come to understand that data loss is
really expensive.
It impacts employee productivity, loss of client confidence, and if it involves external
customers, it's a brand image impact.
Fundamentally, you can implement RAID on your servers two ways.
You can bring in a controller.
That's considered hardware RAID.
Some motherboards actually have the controller built in.
Or you can actually use the operating system.
Linux kernel supports a variety of RAIDs right with Linux.
You don't have to buy a controller.
Windows Server supports a variety of RAIDs without buying a controller.
Those are legitimate options.
This is a brand new controller from Broadcom.
It's a 12-gigabit SAS, SATA, the brand new PCI 4.0, runs about $1,000, can do RAID arrays
internally and externally.
Now, Windows Server and Linux both can do software RAID.
It's very inexpensive and believe it or not, it's very fast.
There is constant improvement on the software RAID in both Windows and Linux.
They're very good alternatives if it makes sense for your application that you're running.
Both Windows Server and Linux support RAID 0, RAID 1, RAID 5.
Now, Windows requires that all your disks be converted to dynamic disks, which is not
a big issue, but if you're going to implement software RAID with Windows, you do have to
convert all your disks from basic to dynamic.
I had a great friend who ran his entire enterprise environment on software RAID, so it can be
done.
As we look at Windows Storage Spaces, keep in mind the difference between the resilient
file system, which only can be used with Windows Storage Spaces.
Make sure you separate the two.
Windows Storage Spaces provides an alternative type of expandable fault tolerance system.
It supports RAID 1, three-way mirroring, and parity.
You can also run the resilient file system with Storage Spaces.
Resilient file system is about data integrity and data corruption.
It has nothing to do with fault tolerance.
That's the job of Windows Storage Spaces.
So keep those two apart, even though they work wonderfully well together.
Now, Linux has an alternative to Windows Storage Spaces called ZFS, Zettabyte File System.
Very respected, used by a ton of storage administrators.
So there is a Linux alternative to Windows Storage Spaces.
A lot of consumer motherboards also offer RAID.
Be careful.
This is known by storage administrators as fake RAID.
Typically, if you run Windows on a motherboard that says it has RAID, you're going to download
a driver from the motherboard manufacturer, and that's going to be the key component to RAID.
Be really wary of that.
Now, if you run Linux on a motherboard that says it supports RAID, you're basically going
to use the Linux kernel.
Be aware that these are not good options.
If you're going to use RAID on your consumer Windows machine or consumer Linux, get a controller,
do it right.
Because operating systems running on server hardware, servers have massive CPU capacity
and memory capacity.
They have fast networks, and they lend themselves well to operating system-based RAIDs or things
like Storage Spaces or ZFS.
All right, Mr. Vanderpool, why do I need a controller?
Well, let's talk about that, because there's a lot of reasons why you need to add a RAID
controller.
Let me name off a couple issues that you may want to consider why you need a controller.
Write back cache, we'll get into that, dedicated processing, avoiding OS boot errors, the need
for more complex RAID, high performance, you really are looking for a higher performance
disk array.
Your hard drive does two major things.
It reads data and writes data.
Of those two things, writing data is the most time-consuming.
In order to give applications a boost in performance, controllers do what is known as write back
cache.
This caching of those write requests are actually saved in RAM.
There's RAM, usually a DIMM that's inserted into a special adapter card on your controller,
and that RAM stores those write requests.
Now, the controller lies to the application, says they're done.
They are not done.
They're sitting in that RAM.
If you unexpectedly lose power, the write commands in RAM are gone because it's volatile
RAM.
We always add battery backup to that RAM or a super capacitor, but this write back cache
significantly improves writing information to the disk.
Now, all logical drives comprising of nested arrays like RAID 10, RAID 50, RAID 60, those
logical devices must have the same write cache setting.
Some applications and some configured servers have so much utilization just running the
application that it makes sense to bring an array controller with a dedicated processor
to offload that expensive RAID calculation, and especially during a failed disk and we're
doing a rebuild of the RAID.
So there are definitely a reason why you would want to bring a controller in that scenario.
If you use Windows or Linux to do software RAID, one of the ugly issues is if you have
a boot problem, an operating system boot issue, because now you have not only a boot problem
with the operating system, but you can't mount your RAID array.
Sometimes it's better to separate the RAID array from the operating system so you're
not trying to troubleshoot both of those issues at the same time.
And by the way, those problems usually exist on Monday morning.
When we get into complex scenarios where we want spare disks for a RAID array, we want
stripe size options for our RAID.
We want to adjust read cache settings.
We want to have rebuild priority options.
We're going to build very large arrays, 24 to 60 disks, or we want the ability to migrate
from one RAID type to another or things like run consistency checks is definitely where
we're going to have to have a RAID controller to do those kinds of things.
This is a block diagram of a basic RAID controller.
Some popular RAID server types is RAID 1, mirroring.
It's common for operating system protection.
This is a very, very effective way of protecting C drive and your operating system.
RAID 5 is one parity stripe.
Typically for a three drive data set, it's really a deprecated RAID system.
Try to avoid RAID 5.
RAID 6 is the king of the hill, really.
Two parity stripes protect you up to two hard drive failures, has pretty good performance.
RAID 6 is a really good RAID system for most applications.
RAID 10 or RAID, what they call RAID 1 plus 0 is expensive, allows two or more hard drive
failures and I'll explain why you really, really need to understand this carefully.
Good performance depending on the application.
So if an application needs high fault tolerance and really good performance, you may go RAID
10.
They're pretty expensive in terms of disks.
Here's an example of a RAID controller and you can see it supports most of those RAIDs
that I just talked about.
Just a quick review of RAID.
RAID 1 is mirroring.
We keep basically a copy of our data on both hard drives.
RAID 0 is just basically striping where there's no fault tolerance.
Mr. Vanderpool loves RAID 0.
I love storage speed.
So I use it on my desktops, I use it on my lab environments, but I never use it in my
production environments.
So it's great on my desktop.
It's great on my lab setup.
Never, never use RAID 0 for production.
RAID 5 gives you one parity stripe.
It allows for one disk failure.
You can use up to 16 disks in a RAID 5, but it's not recommended.
Stay away from RAID 5.
Move to RAID 6.
RAID 6 allows two disks failures, two parity stripes across your disks, up to 60 disks
maximum on a RAID 6 array.
Very popular, a good value for storage.
All right, I need you to listen really, really careful because RAID 10, it's also known as
RAID 1 plus 0, allows for two hard drive failures in this RAID system.
But I want you to look at my diagram here.
I've got a diagram of four disks.
Two, disk 1 and 2, make up a RAID 1 array.
Disk 3 and 4 make up a RAID 1 array.
They are striped.
I can lose up to two disks.
But I have to understand that I can lose disk 1 in my first array, and I could lose disk 3
in my second array, and I'm good.
But if I lose disk 1 and 2 in my first array, I have lost data.
Very important that you understand how the drive failures protect you or not.
This is a nested RAID where I have one RAID working on top of another RAID.
This is expensive because it costs you a lot of disks to get this system.
It does provide higher performance.
So if you're dealing with the need for very high fault tolerance, very high performance,
you may look, and money is not an issue, you may look at RAID 10.
So RAID 60 is a very popular RAID system.
Again, let's look at this very, very carefully.
Look at my disk.
I've got 0 to 7 on my disks in my diagram.
I can lose two disks in the first RAID 6 array and be okay.
I can lose two disks in the second RAID 6 array.
I can't lose three disks in either one of those RAID 6 arrays.
Very important you understand what disk failure you can absorb in your fault tolerance.
This is a nested RAID type often used for critical databases.
Now RAID is all about protecting your application, keeping your server running,
should you lose a hard drive.
What you have to understand is rebuild time.
I've got a bad disk.
I have to pull it out, put a new disk in.
How long does it take the RAID system to get back to a stable state?
That's known as rebuild time.
Each RAID type has different rebuild times.
As disks grow in size to 16 terabytes, this becomes really critical.
So for example, RAID 6 rebuild with 16, 160, 146 gigabyte, 15K RPM,
takes about 90 minutes to rebuild a new drive.
A failed 14 terabyte disk could take as long as several days to become back to a stable state.
So recovery time is very important.
Now some controllers actually allow you to give more processing time to the rebuild
than the production application so that you can rebuild your array quickly.
So just be aware of that.
Rebuild time is very important.
A critical concept in measuring the performance of our servers is using benchmarking tools.
Those benchmarking tools are greatly impacted by caching.
Now caching is multi-layered.
When you have a Mac or a Linux box or Windows, you have multiple layers of caching.
You as a storage administrator and IT professional have to understand all of them
in order to understand how to measure the performance of your storage.
So first of all, what is cache?
It's memory and predictive algorithms.
Those predictive algorithms try to rethink ahead of you,
get what you need before you need it, and have it in fast memory.
So caching is always making something slow look faster than it really is.
All hard disks have caching.
They have predictive algorithms, and they have memory on your hard drive
that is dedicated to looking and pulling data off the platter,
putting it into fast RAM before you need it,
so that when you do ask for it, you can shoot it to the host,
the operating system, very, very quickly.
I want you to look at our diagram here of a hard drive.
The worst situation is number one.
You see a red line going from the platter of the hard drive up to DRAM and up to the host.
This is the worst scenario.
You don't get any caching here.
You're reading it off the platter.
A better scenario is reading ahead on the platter,
getting it off the platter onto DRAM,
moving it to the read cache so that I can quickly send it to the host.
That's the best scenario.
This is very complicated.
The algorithms are very carefully engineered by software and hardware engineers.
There is a lot of effort in designing effective cache.
So here's another layer of caching,
and remember we talked about write-back caching on the RAID controller?
So here you see a controller.
I've got a photo of my server controller,
and that card that you see inserted into that slot is actually a DIMM module.
It's specially designed to handle those write-back cache commands.
It's battery backed up.
In fact, this one has got a supercapacitor that backs up that DIMM.
So in case I lose power, the same thing is true.
I'm lying to the application telling it everything is done,
but they're sitting in here waiting for the controller to send those write commands to the disks.
All right, Mr. Ram, surely we're done with all caching.
No, we just begun.
The operating system also does its own caching.
This is very true.
In fact, your operating system,
if you turn off your operating system file system cache,
you're going to cry at your desktop.
You're going to be hated on your server.
This particular caching file system is run in kernel mode.
It's transparent to your applications.
It uses a demand-based algorithm.
These are very complex systems.
Here is actually where you go in your registry.
I've got my registry opened.
My registry editor opened.
HKEYLOCALMACHINE, SYSTEM, CURRENTCONTROLSET, CONTROL, SESSIONMANAGER, MEMORYMANAGEMENT.
And if you look down here, you can click on the memory management and prefetch parameters.
Now, it is not recommended to go in and tweak your file system cache parameters.
Occasionally on the server side, it might be required to do so.
But this is where those caching systems are set up.
As you deal with storage and building arrays, it is very important.
You've got two big enemies.
One, heat.
The other is vibration.
Vibration with hard drives, spindle hard drives, can dramatically impact a hard drive performance.
This really becomes an issue as we stack lots and lots of hard drives in a single chassis.
The vibration from one hard drive can bleed into another, impacting all of the hard drives.
SATA drives are more sensitive to this than SAS drives.
But engineers have measured up to 10% performance
degradation caused by vibration in large disk arrays.
There are lots of arrays that provide these kinds of anti-vibration mounts.
Take a look when you're looking at arrays.
Make sure they're considering vibration anti-vibration systems.
Google actually has a hard drive anti-vibration patent
so that they can put lots of hard drives into an array.
This device, actually right out of their patent,
has a polymer materials that actually enhances vibration absorption and acoustic noise.
Temperature is a killer of hard drives, so know your temperatures.
Overheating can dramatically impact hard drive performance.
Seagate claims from 5 degrees to 50 degrees Celsius is their operating temperature.
That's 41 degrees Fahrenheit to 122 degrees Fahrenheit.
I would not go to 120 degrees Fahrenheit.
Most agree that hard drives running at 45 degrees Celsius or higher
have a much higher failure rate.
Keep your hard drives cool.
If you're not monitoring your hard drives, you need to go to cpuid.com.
It's a very respected group of engineers who write software for hardware.
They have a hardware monitor that's free.
They also have a professional one that you license.
These are two great utilities for watching the temperature
of all your components on your motherboards.
This software works both on servers and consumer products.
Let's keep moving on.
We're going to dive deeper into the terminology of the storage world.
We're going to be looking at terms like IO, IOPS, latency, sequential access, random access,
queue, and many more terms that are important to understand storage.
Let's talk about latency.
When an application or the operating system requests a read operation,
the time it takes for that operation to be done is called latency.
It's always measured in seconds.
It could be milliseconds, microseconds, whatever, but it's always measured in time.
This becomes real important as we take disks and we start putting them to NAS arrays,
into a SAN, or we're across the network, or we're talking to those disks across the network.
Latency becomes a very important issue that we understand carefully.
When we talk about I operations, input output operations per second, it's known as IOPS.
Be careful, IOPS are often quoted by salespeople,
but make sure they're always in combination with latency.
If someone tells me my storage has so many IOPS, I also want to know what the latency is.
IOPS without latency is meaningless.
Latency is what kills USB hard drives.
You can take a good hard drive, put it into a USB enclosure, and it's just horrible.
The reason is latency.
Look at my picture showing bandwidth and thorough put.
USB, we'll use USB 3.1, has a bandwidth of five gigabits per second.
But the thorough put of a USB hard drive is awful.
It's terrible.
So it isn't that my hard drive quit working as well because I put it in a USB enclosure.
It's my bandwidth is five gigabits, but my thorough put in USB is always pitiful,
which makes USB hard drives awful.
Latency has a big impact on external storage.
So if I've got an external array that's connected to a 3.15 gigabits bandwidth USB bus,
but my thorough put is who knows what, pretty bad.
That disk is going to have latency.
Those IOPS operations are going to be combined with latency, giving you poor performance.
If my storage array is on a 10 gigabit LAN, maybe pretty decent.
If I move to USB 3.2 Gen 2, that's 10 gigabits bandwidth,
but still my thorough put through USB is not good.
I might get better than 3.1, but it's not going to be exciting.
One of the best buses out there is Thunderbolt 3, 40 gigabits per second.
Outstanding.
Latency is very, very low.
You can put those disks external and they're just gorgeous.
The problem with Thunderbolt 3 is security.
So Microsoft has not incorporated Thunderbolt 3 into the Windows operating system
because of security concerns.
40 gigabit LANs are even better for external storages,
reducing that latency, giving you great storage performance.
Operating systems by default, when they need to store something on the hard drive,
they look at the platter and they find a sector that's unused.
The file system says, we got a spot here, you can use it.
So it just puts the data, your Word document, anywhere it can find an open spot.
This is called fragmentation, forcing when you want to read that Word document
that the head and platter have to seek to find that those sectors
where you put that Word document at, that is known as random data.
When I measure performance, I want to make sure this is a more realistic way
of measuring disk performance is using random data access.
Now, sequential data access, the best analogy that I can think of is a vinyl record.
Some of you have no idea what I'm talking about.
You put it on a turntable, you put the needle on the outside of the vinyl record
and you listen to five songs as it goes from the outside of the LP record
to the inside track and lifts and sets down.
That's sequential.
You're listening to music sequentially on the vinyl record.
Now, in terms of data, we see sequential data in things like imaging a hard drive,
where it just starts reading from one of the sectors
tenuously until it copies the hard drive.
Most backups are pretty much sequential data,
where you're reading one sector after the other contiguously.
This is a way you can also measure data on a hard drive.
Queuing, very important.
Don't get queuing confused with caching.
Queuing is a buffer.
As we send rewrite operations to our disk,
there is a queue that will take those requests and kind of stack them up
because it maybe can't do them all at one time.
So each command that you send to a hard drive is put into a queue.
The amount of commands that can be stacked up is called queue depth.
SATA drives have a single queue, and they have typically a queue depth of 32.
32 commands can be put in that queue.
SAS drives have one queue, and they can go up to 256.
Now, there are enterprise SATA drives that have a queue of 256 also,
but most of the consumer drives that you buy, spindle drives,
are going to have a queue depth of 32.
So it gets more complex.
So I've got all these commands in my stack,
and the hard drive is waiting one at a time to execute those commands.
This is called first in, first out.
So look at my hard drive with the red lines,
and this is a first in, first out, non-intelligent hard drive
that is executing commands in the queue.
I'm going to read a sector on 4, 3, 2, and 1,
and look how many times the hard drive has to rotate
in order to execute four commands.
It has to rotate almost two full rotations.
Now, over here on the hard drive that has the green lines,
this is known as native command queuing.
They put intelligence in the queue.
It looks at the commands in the stack and says,
hey, I think I can do a better job and rotate the disk less frequently
and still get more done with less rotations.
SATA use native command queuing.
Let's look at a real-life example and the problem with queue.
I've got a SATA drive, one queue, a queue depth of 32,
and on my desktop, I'm running an e-commerce application.
You can buy something, you can pay with your credit card,
and we'll send you delivery and tracking information.
So my e-commerce application, you can see in the block diagrams,
it has user authentication, shopping cart management,
purchase and payment, tracking and delivery.
All of those are processes and microservices that are running,
and they're all hammering that four little hard drive
with one queue, a queue depth of 32.
If I realistically tried to do that on a PC with one hard drive,
you could see it would take you a half hour
to buy anything from my e-commerce application.
Let's take a look at what happens when we add a controller.
Now I'm back to the same e-commerce application,
microservices all needing access to the hard drives.
Now I've got five or six SATA drives hanging off a controller.
The controller has its own queues and its own queue depth.
Now as all these application processes,
microservices are hammering the controller,
the controller can very easily pass on IO requests to the various disks,
giving this application five seconds to finish your purchase,
and you're a happy customer.
You can see the importance of understanding queuing.
It's important to understand that queuing is done by both disk,
controller, and yes, your operating system.
Now, queue depth is a feature of the disk,
the controller, and the operating system.
Windows does not give you the ability to adjust
your queue depth on the operating system,
but some controllers do allow you to manage your queue depth.
So here I've launched Task Manager,
and I'm looking under the performance tab,
and I can look at my disks,
and over here I can see IO read speeds and write speeds,
and these could be in kilobytes or megabytes per second,
but this gives me some idea of IO demand on my disks.
That doesn't help me a lot because I need more information than that.
So I may go down here, notice Open Resource Monitor.
Let's go take a look at Resource Monitor.
Now this type of tool gives me a lot more information.
I have a disk tab.
I have an overview, CPU memory, and disk tab.
Allows me to analyze processes on my operating system
that may be excessively reading or excessively writing.
Maybe I have a poorly written application.
This will allow me to analyze and focus in
on what process is actually causing me trouble,
or what process is actually demanding a lot of disk time.
I need these tools to be able to understand
what's going on in the operating system.
What are they doing to my disk subsystem?