Storage Spaces Explained. Microsoft's new file system ReFS leverages Storage Spaces to provide resiliency and data protection. Uses the B+ trees, integrity-streams, and 64-bit checksums. Compared to ZFS, BtrFS for Linux. We will look at the advantages and disadvantages of ReFS. Learn how to use Storage Spaces and create inexpensive backup or storage arrays using USB, Thunderbolt, or Firewire DAS enclosures. I will demo how to set up and configure Storage Spaces using Disk arrays, used Enterprise SATA drives. I use Storage Spaces for my backup array.
Hello and my name is Lowell Vanderpool and this channel is dedicated to IT professionals,
IT students, and anyone who's interested in technical subjects.
Windows 10 Storage Spaces is an incredible technology put inside of a client operating
system. So we're going to look at data center storage for the common man.
One of the goals of this video is to give you a very practical guidance and a demonstration
on how to set up enterprise-grade storage system that's resilient and inexpensive.
Even though I'm going to show you an enterprise-grade storage system that most people
can afford and set up, you still have to have a carefully designed and tested backup plan.
Windows 10 includes Windows 7 Backup, which can comprehensively backup your data
and provide bare metal recovery. Don't forget, you have to have backups.
I can't tell you how many times I've seen small businesses and home businesses that depend
entirely upon, say, lots of 4K or 8K video content, photographers that use high-resolution
photos, semi-professional musicians who have tremendous investment in their audio recordings,
or any small business owner with critical business data.
Any one of these businesses, if they lose their data, they have lost their business,
period. And yet, many times, it's the last thing they consider is the fragility of their data.
One incident of ransomware, and they would be finished.
Windows 10 comes with four enterprise technologies. Hyper-V, which is a full
enterprise-grade virtualization product, and it's built into a client operating system. Amazing.
Storage Spaces, an enterprise-grade storage system.
REF File System, and the Linux Subsystem. These are full-blown enterprise technologies that are
found in your copies of Windows 10. Let's begin by looking at REFS.
This is a new file system. Its codename was Protagon.
The Resilient File System, or REFS, was released in 2012 with Server 2012.
The technology now is included in Windows 10, but it can only be used as part of Storage Spaces.
REFS is a resilient file system used for enterprise storage.
NTFS is a good, solid file system, but REFS has a high data protection from corruption.
It uses what is known as the B plus trees for all data disk structures.
It uses integrity streams so that all metadata has 64-bit checksums,
and these checksums can also be used for data. It does not need error-checking tools like CheckDisk.
It also uses a feature called Copy-on-Write, which protects your data
should you lose power or have a system crash. Other modern file systems like ZFS,
BTRFS for Linux, as well as the Apple File System, also use this Copy-on-Write feature.
So why do we use a B plus tree storage system? The primary value of B plus tree storage
structures is efficient retrieval, which is perfect for a file system.
Many popular file systems and databases use B plus tree. Linux Extended 4 uses it for indexing.
Microsoft SQL Database, Oracle, SQLite, the BE File System for BOS,
Journaled File System used by AIX and HPUX. It was described in 1972 by Rudolph Baer and
Edward McRae. REF has some advantages right out of the box over NTFS.
It scales better for large files and volumes. It can handle up to 35 petabytes
if you use 64-bit clusters and 16 petabytes if you use 4K clusters.
In almost all cases, you and I, the common man, will use 4K clusters. Max file size,
35 petabytes if you use 64-bit clusters, 16 petabytes if you use 4K clusters.
If you're using normal data, 4K clusters is preferred.
If you're using a lot of, say, 4K video or 8K video, then 64-bit clusters might be preferred.
The number of files stored in a folder, up to 18 trillion.
REFS is compatible with a lot of existing APIs and technologies. It still supports
access control lists, bitlocker, mount points, volume snapshots, and on and on in this list,
disk quotas, et cetera. It does support things like data dedupe.
REFS does not support compression, page file support, bootable disks, 8.3 file name support,
and you can't just format a hard drive. You need storage spaces.
And if you convert an NTFS to REF, you're going to lose the data.
REFS and NTFS are not the same. REFS uses 64-bit addressing, and the main structural
element in this file system is the directory. Again, cluster sizes range from 4K, which is
typical, to 64K. Now, don't get confused. Storage spaces has nothing to do with REFS.
Now, REFS is simply a file system, and it can and only can be used with storage spaces.
You still can use NTFS with storage spaces. Windows 10 provides two disk management systems,
storage spaces and the old disk management. Even with Windows 10, you can't avoid disk
management. It's still critical to manage bootable disks. It contains the NTFS partition
for the operating system, and it still needs to manage the page file. So disk management is not
going away anytime soon. So let's talk about the fundamentals of storage spaces. Number one,
the word pool. This simply means a set of physical disks. The next term is called a slab.
It's a chunk of disk space, 250 megabytes. All physical disks are sliced into what is known as
slabs. And the last concept is space. It's just basically a virtual disk that's created
from a set of slabs. Now, there's three characteristics of every storage space.
One, you can have simple layout, mirror, three-way mirror, or parity. You can also use a fixed or a
thinly provisioned type of space. And remember, when we say space, think of virtual disk. And for
that virtual disk or space, we can use a file system of REFS or NTFS. Some concepts to remember
about Windows 10 storage spaces is it's basically software-defined storage. It's expandable storage.
You simply add more drives and it grows. Now, you can have either two-way mirror, which acts like
a RAID 1. You have to have a minimum of two disks or parity, which is like RAID 5. You have to have
a minimum of three disks. You can have three-way mirror, which requires five disks, or you can have
simple. It's kind of like RAID 0 with a minimum of one disk. Remember, storage space is the only
place you can use REFS, but you still can use NTFS on storage spaces if you so desire.
Now, Windows 10 storage spaces is designed to use internal disks. So, it can use SATA spindle
disks that are internal, internal SSDs, M.2s, internal SAS disks. I'm going to teach you how
to use external. If we're going to use Windows 10 storage spaces for resilient, redundant,
internal storage, your limitation is how many SATA ports you have on your motherboard. It's easy to
implement. You plug in a cable and a hard drive and off you go. Expandable based on how many
SATA ports you have on the motherboard. So, think about that. And it is the fastest transfer rates
because you're taking full advantage of SATA, or if you have NBME, you're getting the NBME speeds.
Windows 10 storage spaces fully supports internal hard drives. If you want to use Windows 10 storage
spaces for external storage, you can do this also. You can buy a variety of external enclosures.
Most of them connect via network, which I encourage you don't use. USB 3.0 or 3.1
generation 2 is the fastest. You can set up all your disks as just a bunch of disks. Do not use
the RAID software built into the enclosure. We want to use storage spaces, not their RAID. This
is the best way of building a highly reliable, resilient backup system. External enclosures are
very, very desirable, especially with storage spaces, even though, by default, storage spaces
will not use them without some reconfiguration, which I'm going to show you. Make sure that your
enclosure has adequate power supply. Make sure it has hard drive cooling. These are great because
you can add hard drives easily and replace defective hard drives. And if you need to,
you can pull the hard drives out, put them in another enclosure. They usually provide massive
storage capacity and they make for fantastic backup systems. One thing that I really like
about external enclosures is that I can disconnect them, except when I do a backup, so I'm protected
against ransomware. Now, some of the disadvantages are it uses USB 3.0, so it's got a little bit
slower transfer rate. If you can get the USB 3.1 Gen 2, which is 10 megabits, it's pretty good.
Again, the easiest way to use storage spaces is internal drives, so they'll work with almost
all internal disks. Make sure your power supply on your PC is oversized to handle that spindle
drive startup. Make sure it's got good fan support so that you can cool those hard drives. Anything
over three terabytes, you need a lot of cooling on those disks. It makes it more difficult to
remove disks and add them and expand them, but it's doable. When you're purchasing disks for
storage space, always buy Enterprise SATA hard drives. Enterprise disks have a much more robust
mechanical system and better bearings. The mechanical systems just last longer. They're
more expensive. They typically have a five-year warranty. You can even buy Enterprise used hard
drives on Amazon with a three-year warranty. If your power supply is low and you can't upgrade
your power supply on your case, you can always go down to two-and-a-half-inch Enterprise hard
drives. They run cooler and require less power. As of the time that I did this video, there were
only a few companies that were using USB 3.1 Gen 2 USB. If you can get it, it's worth it. It's 10
gigabits. It really makes a difference in transfer rate. If you decide to go with SSDs for your
storage space, use them only internally so that you get all the maximum transfer rate that you can.
Shop carefully and purchase maximum capacity for your budget. The great site to check out disk
prices is the one listed in the URL. It's relatively painless to force storage spaces to work with USB
externals. I'm going to show you how. I'm going to walk you through the process of how to get
storage spaces to work with external hard drives, which typically it doesn't want to do. Now, this
is simply two two-and-a-half-inch spindle drives. One's a Toshiba. One's a Seagate. They're very
small, 500 gigs, and I'm just using them as a demonstration. I've got a little USB 2 port
toaster, and I can plug in three-and-a-half-inch hard disks, or two-and-a-half-inch hard disk,
or SATA drives also. This works really, really well. I've used them for production servers. I've
used it in my own use because I do a lot of video, a lot of photos, and I need a reliable backup and
storage, mass storage. So, I'm going to demonstrate how to get this to work and walk you through it
step-by-step. So, let's begin by opening up Computer Management Console, and you can see my
two USB disks, and right now they're unallocated. They're simply plugged in. I'm going to go to
Device Manager, and you can see I've got their volume names, and I've changed the volume names
of both disks, so it indicates to me they're USB. That way, when I'm working with disks,
I don't make mistakes. The volume helps me identify what that disk is. In this case,
I could use the word external in the case, or I could use USB disk or device. So, this just
gives me a heads-up because you can see I have a lot of disks here. I don't want to make a mistake
and blow away a good disk. I've changed the volume names of my USB drives so that I know which disk
I'm dealing with. So, that's something you want to remember. So, I look in Device Manager. They're
all there. Go in Disk Management, and I can see that my two USB disks are unallocated and unused.
If I launch Storage Spaces right now, and I try to use these two disks, I'm going to create a new
pool. Remember, creating a new pool is about gathering a couple disks together, and I'm going
to use, here I'm going to use disk five and disk six, and notice there's my two disks, and I'm
going to go ahead and create a pool. Now, Storage Spaces does not like to use USB external disk,
and you can see it failed. It's not going to do it. So, we have to do some things to these two
disks so that we can convince Storage Space to use them. So, let's begin with step one. So,
the first thing I'm going to do is come to my first USB disk, and I'm going to come over,
and in Disk Management, I'm going to click Properties, and I'm going to go to the Policy tab.
USB drives, USB flash drives, all want the ability to be able to quickly remove them.
You can just pull them out, plug them in, and that's the way we want. But one step that we're
going to have to take is to enable Write Cache in Windows. Now, when you do this, you can no
longer unplug this drive like you did in the past. You're going to have to always use the
Safely Remove Hardware feature in Windows whenever you want to remove these disks.
Now, I'm going to change this policy. This is step one. So, I'm going to do this for disk 5,
and I'm going to come down here and do this for disk 6. So, I'm going to do this for both disks.
So, that's the first thing I want to do is go into each disk and change this Quick Remove Policy
to Enable Write Cache, and we'll say OK. Now, I'm not done. That's my first step.
The next thing we want to do is we want to tell the USB hard drive that it is not a removable
disk. So, we're going to use a tool from Lexar, which is a flash drive manufacturer. It's a USB
format tool, and make sure you go to a reputable site to get this utility. In this case,
I'm going to PendriveApps.com, which has a wonderful selection of portable apps,
and if you've watched my videos, you know I love portable apps. And you can use this tool,
and the only thing we're going to use is the Flip Removable Bit feature. So,
we're going to use that feature. Now, I've downloaded the tool, and I've got it right
here. I'm just going to execute it. So, before I run the Flip Bit feature tool,
I'm going to come over here, create a simple volume, just walk through and give it a drive
letter, and we'll do the same for the other. We're just going to put it into disk management
so it will be seen. All right, it's done. Now, let's go to the utility, and notice that it sees
D and E as removable USB drives, or they could be flash drives, and I'm going to choose D first,
and I'm going to flip the bit, and it's going to change a structure on the disk that is now
going to tell Windows this is a non-removable drive. Then, I'm going to go to E. I'm going
to select E, and I'm going to flip that bit, and then I'm going to exit. I'm done. That's
all I need to do. So, now we finished that. Let's go back to Storage Spaces,
and I'm going to create a new pool, which basically is allowing us to organize some
disks, and I'm going to come down to disk five and six, and remember, before this wouldn't work,
we're going to create a pool, and voila, it worked. So, that's all I had to do to take those
USB externals and make them available to Storage Spaces. So, now we're ready to go.
So, let's take a look. I'm going to choose a drive letter that is not the next available
drive letter. I never use the next available drive letter. I'm going to use one like R.
Further down in the drive letter assignment, we're going to use R-E-F-S. You always want
to use R-E-F-S. The whole idea is you're gaining resiliency with the type of RAID capability that
Storage Space gives you. There wouldn't really be a value of just using NTFS when you can use a
better file system. You can see it. We can use two-way mirror. I've got two 500 gig, which gives
me about a terabyte, but if I mirror, I'm going to get about 500 gigs of space. Now, I can say
I'm going to, let's say I put, say, 20 terabytes. That doesn't mean I have 20 terabytes. It means
that I can grow. I can keep adding drives up to 20 terabytes. I can change this to simple.
I can change this to three-way, which I don't think I can because I don't have five drives.
In most cases, most people have recommended not to use the parity. It seems to slow things down,
especially if they're USB hard drives. Either the two-way mirror, which is probably the best
in my case, so I can go ahead and create a Storage Space, and I'm going to use a letter R,
R-E-F-S, and go ahead and create the Storage Space. It's going to format the drive, and notice
it puts it in my Explorer view, so I can see now R Drive, and its volume label is Storage Spaces.
If I come down here in Disk, Disk Management, it sees it as one disk, not two, and it is now
drive letter R, and it is R-E-F-S, so I can see all of that. So, let's take a look at the recovery
ability, the resilience to failure. So, I've got my Storage Spaces, R Drive, and I put some
material in there. You can see I've got about four gigs of material in this new Storage Space
volume, and I'm going to go ahead and click OK. So, here I've actually taken one of the hard drives
out of the array, shut down my computer, popped it out, and you can see in Storage Spaces, it does
see this problem. It sees we've definitely got a problem with the Storage Spaces array. One drive
is there, one drive is, in this case, totally gone, and yet, I can come up to my Explorer,
and there's my R Drive, and all my tools, all my files are there. It does provide us all the
necessary redundancy and mirroring the RAID capability that we want, even with completely
failed hard drives. This is the beauty of Storage Spaces. To help lower my cost for hard drives,
I went for used Enterprise SATA hard drives. I was able to substantially lower the cost of all
the hard drives that I use. Make sure that when you're dealing with Storage Spaces, purchase the
same make, size, and model drive. Keep all the drives the same, and it's wisdom to buy an extra
spare disk. While you're buying disks for your array, go ahead and buy a spare disk. Look at
your return policies. I would encourage you to avoid eBay. Pay attention on how the drives are
packed and shipped. If they're not shipped like new drives, I probably would be suspect. Make sure
you test the drives upon receiving with a tool like Crystal Disk Info. It's a great tool to
test the disk. If you can, burn at least 24-hour burn-in for all your used drives. I purchased four
used drives, and you can see I ran Crystal Disks on all of them, and based on the power on count,
they had never been turned on. So, they were brand new. They were sold as used, and I was
able to save probably 70% on the price. So, these are very nice Enterprise 3TB drives.
A couple points about the disk enclosures, if you decide to go that way. You can either get
DAS enclosures or NAS enclosures. Direct Attack Storage are enclosures in which you attach them
to your PC via USB, eSATA, FireWire, Thunderbolt. Those are all DAS or Direct Attach Storage.
You'll usually get better performance with a DAS enclosure. If you're going to connect via a network
cable, that's usually going to be a NAS device. There's benefits to NAS, and there's drawbacks
to NAS. Generally, the lower price NASs are very slow, so that's one drawback. DAS and NAS can have
RAID built in, but if you're going to use Storage Spaces, you don't need it. So, you can save some
dollars and just buy an enclosure that supports as many disks as you want, and Storage Spaces
is going to provide you the RAID. If you're going to use USB externals, make sure that you disable
the USB suspend setting, so that you're not going into power mode and turning off your USB external.
You want to go into each disk that you use for a USB external and make sure that you disable
the quick removal feature and turn on the write back caching. You also want to download the boot
it utility and flip the bit, so that it is no longer seen as an external device. You can see
once you have your Storage Spaces running, you can actually see your Storage Spaces in your device
manager also. In the video description, you'll have a link to download the notes for this video.
This will have extensive step-by-step on how to do everything that I'm showing in this video.
In the notes, I'm going to include a lot of PowerShell cmdlets that you can use to look at
your Storage Spaces and repair them. I have an entire section in the notes based on when things
go wrong with Storage Spaces, so I've got a lot of PowerShell and troubleshooting
ideas on how to fix your Storage Spaces should something go wrong.