This video series is a comprehensive study using powerful tools to lift the hood of the Windows Operating System and allow the tech to understand and troubleshoot applications and services.
We will cover how to use Process Explorer, Process Hacker, and many other Sysinternal tools but most importantly we will discover how Windows really works and how we can understand our applications and effectively troubleshoot problems. From your PC/laptop to servers this understanding is critical to every IT professional.
Hello, my name is Lowell Vanderpool and we welcome comments and feedbacks on every presentation
we do.
You can contact me anytime, mrvanderpool901 at gmail.com.
Effective Windows troubleshooting is no magic.
You must do the hard work of understanding Windows fundamentals very well and continue
to develop a deeper level of knowledge.
Second, you must troubleshoot.
You must continue to be willing to troubleshoot all kinds of problems daily or as often as
you can.
No book or video or course will make you an effective troubleshooter without the hands-on.
To better understand how to troubleshoot Windows, we're going to divide it in half.
This upper portion, which is called user mode, includes all of our system processes, services,
user applications.
All of these components are what most IT professionals spend most of their time troubleshooting.
On the bottom half is the kernel mode.
This is where our kernel lives.
Our howl.dll is here.
It gives us portability and abstracts us from the actual hardware.
We have our user and GDI components, which is our graphical modules, memory managers,
registry, and of course, device drivers.
Most of the tools that we're going to use to troubleshoot Windows and understand architecture
and fundamentals is from the Sysinternal suite of tools.
You can simply type in Google Sysinternals and you will get to this main home page.
You can download the entire suite of tools, which I recommend.
They will download as a zipped file.
You can extract them just like I've done here and double-click and you can see a whole list
of tools that we're going to use for troubleshooting and understanding Windows.
The Windows Sysinternals tools were developed by Mark Rusanovich and Bruce Cogwell.
They are presently being maintained by both Mark and system engineers within Microsoft.
Before we jump into the tools, I want us to look back at this diagram.
Notice in kernel mode, all the code and the items and objects that are running
in the kernel mode are basically relatively small in relationship to user mode.
And when you look up, you see system processes, services, applications.
In most desktops, that can be as large as six gigabytes, all of that running up there.
And yet down in the kernel mode, we're talking sometimes as small as 290 megabytes.
Big difference between kernel and the rest of Windows.
Also, in kernel mode, they run in the same memory space.
Everything you see in kernel mode is running in one memory space.
Now, everything in user mode is running in private space.
Every single item running in user mode is going to be given a unique space
in memory and they don't share it.
This prevents Chrome running in user mode from crashing PowerPoint and vice versa.
So, it provides stability in the processes that are running in user mode.
But down in the kernel mode, everybody's running in the same address space.
Kernel mode has to be very secure, very robust.
This is the responsibility of Windows.
There are a couple of wild children in the kernel mode and that's usually device drivers.
So, even though they have to be signed and verified, it's the drivers,
the third-party drivers for your video card, your network card,
they are the ones that cause the grief and heartache in the kernel mode.
I would say 99% of all kernel mode related problems are going to be device drivers.
Rarely will it be a Microsoft kernel issue.
The kernel, because it runs in one address space,
is known as a monolithic operating system.
This really comes from Unix.
Now, Mark says this is a Unix-style operating system and it is monolithic.
Wikipedia disagrees and says that Windows kernel is hybrid.
All right, back to user mode.
Go back up with me to user mode.
All of these modules of software which we are going to learn are called processes.
That includes systems, services, applications.
They cannot reach the kernel.
They're not allowed to reach the kernel.
They access it through calls.
And one of the very important files that makes that happen,
that allows them to go into the kernel and do request functions such as access to disk,
network, cryptological functions, hardware,
all of that has to be done through the ntdll.dll.
Now, I didn't make up the file name, so I know it's weird.
But I want you to look at that and remember that
because I'm going to show you that over and over as we look at the processes in user mode.
Protecting the kernel has become a very important part of Windows.
We've added Trusted Platform Module, TPM, Secure Boot.
Very important.
Virtualized Secure Mode.
This is if you have Windows Enterprise or Windows Education Edition.
Kernel mode drivers must be signed because they're down there in kernel
and they're running in executive privilege.
DMA drive-by protection.
This was very big with Thunderbolt.
A data execution prevention.
Address space layout randomization.
All of these very important changes are designed to protect kernel.
There's been over 28 kernel hardening changes since Windows 8.1.
And what they're protecting us from is unstable drivers, rootkits,
bootkits, hash-the-hash, credential stealing, and on and on.
Remember, anything that impacts a kernel is going to give you that lovely blue screen of death.
So the kernel is critical.
As I mentioned before, the kernel is small.
It's very complex, complex and compact.
I want you to remember a couple of the files in the kernel.
One is ntoskernel.exe.
You're going to see that a lot as we look at processes trying to access the kernel.
The howl.dll.
This is a very important file that gives us portability.
I'll get into that more.
And there's that file, ntdll.dll.
We're going to see those files a lot as we troubleshoot Microsoft Word, Chrome,
Edge, all those applications.
We're definitely going to see these kernel files.
All of today's modern operating systems, Windows, the Mac OS, Linux,
all leverage a feature in today's CPUs, the x86 architecture.
And they're called rings of protection.
All of today's x86 processors support four rings of privilege.
These rings are part of the CPU architecture,
and they allow the operating system in close cooperation with the CPU
to provide privileges or areas of protection for running software code.
Windows runs its kernel in ring zero at the highest privilege mode.
It has protection against memory regions,
IO ports, and has access to special instructions.
You guessed it.
Everything else in Windows runs in ring three.
That includes all processes, all applications, all services.
Everything runs in ring three,
separating it via the protection of the CPU from the kernel.
Starting in 2005, Intel and AMD added hardware virtualization assistant.
They added some new instructions called the VT-X instruction set.
This added a new ring of privilege.
It was called negative one.
In other words, it was below ring zero.
That gave the hypervisor more privilege than the kernel.
Everything except the kernel runs in ring three.
Today's operating systems do not use ring one or ring two in the x86 architecture.
Even if you log on with the local administrative rights,
you are still running in ring three.
Now, this continues on as Windows develops and recompiles Windows 10 for the ARM chip.
Right now, Qualcomm Snapdragon 835 is being used for the new Windows 10 on ARM.
It also has three rings of privilege or protection.
PL2 is the highest level of privilege, and that is where the hypervisor will be located.
PL1 is where the kernel will be run.
And PL0 is where all the Windows applications and services will run.
Notice its reverse order from the x86.
Good process as a container or box.
No, I'm not talking about Docker containers.
But think of it as an isolation container in which code that runs inside is isolated
from any application that is running Windows.
A process requires a virtual private address space,
an executable program that creates the process,
a list of open handles to various system resources.
And that would be things like registered keys, folders, files, graphical objects.
Always, a process has a security context.
And this is called an access token.
Each process will have a unique identifier called a PID or a process ID.
And every process has at least one thread of execution.
That's the code inside the process that does the work.
Go ahead and find your folder that has sysinternal suite in it
and scroll down till you get to the Ps.
And we will see two files.
This is Process Explorer for 32-bit and Process Explorer for 64-bit.
Process Explorer is like Task Manager, but really is a far and away better tool.
Now, remember, right mouse click, run it as an administrator.
And let's go ahead and launch this tool.
Almost everything that we see in Process Explorer is running in user mode.
Remember the ring three of CPU privilege.
There are a few exceptions.
At the very top of Process Explorer are a few items that are actually running in kernel.
One of them, I'm going to look at system.
If I right mouse click and go to the properties, you notice under threads,
I'm running NTOS kernel.
Remember that file I told you about.
Another item is this registry up here at the top.
I'm going to go to properties and you'll see NTOS kernel.
So this is actually running in kernel mode.
Now, the reason that Mark put these in is because these are going to be
very helpful to us for troubleshooting.
For example, if I want to troubleshoot kernel mode drivers,
I can right mouse click on this system.
And actually, as I scroll down under the threads,
many of these .sys are actually hardware drivers,
making it very helpful for me to find a misbehaving driver.
In this column called processes,
it shows us the files that actually are responsible for creating the process itself.
So here I've got svchost.exe.
If I right mouse click, go to properties,
you can actually see the path to the executable that actually was able to create a process.
Now think about this.
When you realize that in Windows, there are probably about 8,000 files
that make up Windows, all the files that you install.
And yet out of 8,000 files on your hard drive,
probably 120, 130 can actually create a process.
That makes these files very unique in Windows.
In this column of Process Explorer is the PID column.
And if I expand this out a little bit,
you can see that every process that runs is assigned by the operating system a unique PID.
Let's go back to svchost.exe, this process right here.
I'm going to go right mouse click, go to properties,
and I'm going to click the performance tab.
And we can see that this process request virtual memory address space.
And it's about two terabytes.
Now hang on to your horses.
We'll get into that in a minute.
Also, look down here.
We see handles.
Under our security tab, we can see the security context that this process has been run under.
So processes do not do a whole lot.
They are dependent on this tab called threads.
And here are the threads that run inside this process.
And here we see ntdll.dll.
So this particular process is reaching down into the kernel using ntdll.dll.
Putting applications and processes is nothing new.
Linux uses it, the Mac OS, iOS, Android,
all modern operating systems use the concept of processes.
It allows applications to be in separate containers that give us separation,
stability, and most important, rules of sharing data and communicating between each process.
This is called IPC, inter-process communication.
Let's go back to Process Explorer and discover what else we can learn about processes using this tool.
I'm looking at svchost, and I can look over here on the column for CPU,
and I can see its CPU usage.
Under private bytes, that tells me how much virtual memory that particular process is using.
Under working set, I can see how much physical memory that particular process is using.
Description tells me information about that process and company name.
I also see under svchost a process that is below it, dllhost.exe,
and I notice there's a parent-child relationship.
It's indented in.
That tells me that this parent, svchost.exe, was responsible for launching
all of the child processes below.
This shows me that parent-child relationship in processes.
Process Explorer also provides a nice color code system
that helps us to identify the role that process plays in the operating system
or the type of process that it is.
If I take my mouse up to options and come down to configure colors,
it gives me a color code chart.
One of the ones that is very interesting is green.
That means when I launch a process and it starts, it will show up green.
When I terminate a process, it will show up red.
So let's watch Process Explorer, and I'm going to launch Edge down here,
and we'll watch a lot of processes turn green.
That means they're being launched or started,
and if I come up and I terminate Edge,
you'll see a lot of these processes turning red.
They're being terminated.
Another important color is the blue.
It says Own Processes.
You can see here I've got smartscreen.exe, dllhost.exe.
These are processes that were started upon my logon.
So all of the processes that we see that are blue in this list
were all launched upon my logon.
The pink color indicates Windows services,
and you can see quite a few of them,
and notice they're all launched by svchost.exe.
What's nice, Mark has done a mouseover,
so I can hover over any one svchost.exe,
and I can see which services are being launched in each svchost.exe.
The next color that's very important is immersive processes,
and that's that turquoise color.
You can see I have a lot of immersive processes right here.
Remember, this is a new API that Microsoft is using
to develop all of its new apps.
Microsoft is moving very quickly
on moving all of its new products to immersive.
So the Start menu, the Microsoft Store,
and all the apps that you download,
Microsoft Edge, the Search Bar, Cortana are all immersive apps.
When you go to Settings on Windows,
all of the settings are now immersive apps,
and there's lots of reasons for Microsoft to go in this direction.
Another important color is suspended processes,
and that's gray.
This is always regarding immersive apps.
You can see a very long list of suspended processes.
Immersive apps allow Microsoft to suspend a process
if it's not being used.
It still resides in memory.
It's still a process, but it no longer impacts the CPU,
which means in mobile devices, it's saving battery.
Because processes, which are highlighted in green,
can start and stop, and stop is highlighted in red,
these two actions can start and stop so quickly,
you cannot see them.
So another important option needs to be changed.
Let's go up to Options and Difference Highlight Duration.
We want to set this to at least five seconds.
So if a process is terminating or being created at a very high rate,
so much so that you don't see it in Process Explorer,
by changing the highlight duration to five seconds,
it will keep that green or red up there for at least five seconds
so you can see it.
I have representative up here, three processes.
Let's bear with me and let's just say
these are representing all the processes in user mode.
Every one of these processes have threads in them.
The question is, who goes first to the CPU?
How do they decide what process gets CPU time?
Second of all, how much time do they get in the CPU?
Well, the answer to that question is all done
by an algorithm called the scheduler.
The scheduler is a complex mathematical algorithm
designed by Microsoft engineers and it resides in the Windows kernel.
It is designed to give all the processes and the kernel
and drivers efficient access to the CPU.
Because CPUs run at high clock speeds and the efficiency of the scheduler,
we get the illusion that processes are being executed almost simultaneously.
This is where the word multitasking comes from.
The scheduler is very efficient and effective
and can keep the CPU busy and take care of the needs of the operating system.
Preemptive multitasking simply means that Windows engineers
have given the kernel the right to control the scheduler.
In other words, it can preempt the algorithm.
For example, if Chrome is being put into the CPU by the scheduler,
the kernel can tell the scheduler,
no, take Chrome out, I want PowerPoint to run.
At first, preemptive multitasking sounds kind of bossy,
but all modern operating systems use preemptive multitasking.
Let's put all these concepts together.
In the diagram, I'm going to show you a process, a new process.
This is going to be WordPad.
And let's say I just launched it in my Windows 10.
It is what we call created.
It is now pulled up into memory and is now known as waiting.
It is waiting for the scheduler to allow it to run in the CPU.
Whenever a process is in the waiting state and it moves from waiting to running,
this is known as a context switch.
It tells me the process has been executed.
Once it's executed, it moves into possibly the blocked condition.
In other words, it's waiting an input.
The process is waiting keyboard strokes or a mouse movement or a printer.
Once it moves out of blocked condition, it moves back into waiting.
And then again, the process is in the scheduler to get executed again.
Any process that's in the blocked condition for too long a period of time,
the memory manager will simply move that process into the page file or virtual memory.
Also, if a process is in the waiting state for too long of time or you're not using that process,
again, the memory manager can move that into the page file.
And finally, a process can be terminated.
You end WordPad.
The process cleans up all the threads, all the memory spaces,
returns the memory back to the kernel and then exits.
And that process is now moved out of main memory.
Every process in Windows goes through these different states as it runs in the operating system.
Now, let's get real practical with everything that we've learned.
Let me take my mouse up to the top of Process Explorer where you see these columns.
And I'm going to come over to the top where the columns are at,
right mouse click, and I can select and add more columns to Process Explorer.
I'm going to start with process performance and I'm going to add handle count.
How many handles do each process contain?
Notice context switch.
This tells me how often the process went into the waiting state, moved to the running state,
how many times that process has been executed by the CPU.
So I want to see that.
I also want to see how many threads, that's the modules of software inside the process.
I'm also going to go to process memory tab.
Here I see private bytes.
This shows me how much of the code in the process was either in the waiting or block state
and was not being used and was moved to virtual memory or the page file.
The working set is going to show me how much of that process is actually in physical RAM.
Now, we have added three new columns, handles, threads, and context switches.
I have already went to the top of my column.
And if you click on this, it will sort that column from highest to lowest.
So I've already done that.
You can try that out with yours.
I want you to look at the three top processes that have been executed on the CPU.
Remember, context switch shows us how many times that process has been executed by the CPU.
Come over here and you can see one is interrupts, system idle process, and the system.
All three of these are in the kernel.
System idle process is a special process that Microsoft designs so that when nobody else
needs the CPU, the scheduler will move system idle process into the CPU and execute it.
So if nobody needs a CPU, system idle process is put in.
And you can look under CPU utilization.
And right now it's using the CPU the most.
But over in context switch, this shows us how many times these three processes have
actually used the CPU.
And you can see interrupts has way far away used the CPU more than anybody else.
Even the system.
And this, remember, is the kernel.
So the kernel should be, we would, that would be a good assumption to think that the kernel
would be a top process in the CPU.
And it is.
Let's go back to interrupts.
Remember, those are drivers.
Those represent drivers are really not a process.
If you notice, they don't have a PID.
But Mark put them in there so that we could see the impact of drivers on a CPU.
As drivers need CPU time, they request the kernel to preempt, to interrupt the scheduler
and say, whatever you're running in CPU, Chrome, WordPad, whatever it is, take it out and please
run my video driver or run my printer driver or my mouse driver or whatever.
They are the ones that are constantly going to the kernel and asking for preemption.
Stop the scheduler, whatever it's doing.
I need you to interrupt it and give my code CPU runtime.
Now we understand that if we go to CPU utilization and we sort, I click on this column and I
scroll back up, I can see the processes that are using the CPU right this minute, real
time the most.
I can see their utilization, who is utilizing the CPU at the highest rate.
And that gives me a real time view.
But the beauty of context switch, it allows me to sort and look at what process has used
the CPU the most since I powered on Windows.
I like to look at context switch and look at my top 20 processes, especially if I'm
dealing with third party applications.
Because if I have a third party application in the top 20, and really when I look at that
application, it just doesn't make sense that they're hammering the CPU as much as they
are, that might be pause to reconsider that application for my environment.
Under threads, if I go to the threads column and I sort the threads, I'm going to scroll
back up and look at the processes with the most threads.
Generally what a good rule of thumb is, the more threads that any process has, it gives
you some idea of the complexity of that process.
Here I see my kernel, which is system, has 209 threads, which I kind of expect the kernel
is complex.
I see that the next one is the application that I'm using to actually record this video,
and it has 186 threads.
That's a good way of looking at the complexity of the process.
It's not a hard rule, but it does allow the tech some insight into the complexity of the
application.
Our third column is handles.
I'm going to click on this and sort that one by handles.
Why handles?
Handles are very important.
If any process has too many handles or that handle count is too high, I've seen SQL databases
that had over 100,000 handles.
When it starts getting that many handles, you're probably going to look at a system
crash or a server hang.
Pay attention to the amount of handles your processes do.
Go ahead and sort them by handles.
Look at who's in the top 10 and pay attention to those.
Now I want to look at who's using virtual memory the most.
I'm going to click on private bytes column, sort by private bytes.
This tells me which processes are, their process is either in the waiting state or the block
state, and the memory manager is moving chunks of that code into the page file.
And I can see that Camtasia Studio.exe has got a ton of code in virtual memory.
I can also click on the working set and click on that.
And I can see that also it's Camtasia Studio is the second largest process in actual physical
memory.
Memory compression is the largest of the processes in physical memory, but Camtasia Studio is
next.
When you're looking at working set, these are the processes that are using your physical
memory.
So your top 20 processes that are on your working set, you want to look at those hard
because that's what's using up your RAM.
Remember, whatever's using your RAM is slowing your PC down.
To all the wonderful and talented people and companies who freely share their videos, photos,
presentation templates to the creative community, their generosity has impacted my productions
in an incredibly positive way.