Troubleshooting Essentials: IT Admins' Guide to Windows App Types and Processes: Day 2

Hello, my name is Lowell Vanderpool and we welcome your comments and feedback. You can connect with me at any time at mrvanderpool901 at gmail.com. If you haven't watched part one of this series, you can go to YouTube and just search for VanderL2796 or my name, Lowell Vanderpool, and you can find my channel. You can go to my channel and where the little arrow, red arrow points to the search, you can search my videos by using Windows Troubleshooting. Just type in Windows Troubleshooting and you should get results and in the results will be Windows Troubleshooting like a pro, what you never learned in class part one. And that will get you caught up before you start this series. Without a doubt, one of the most exciting trends in the Windows world is the recompiling of Windows 10 on the ARM platform. That's going to give us laptops and other form factors with LTE connectivity or 5G eventually, amazing battery life, instant on, and Windows 10 that can support both native running ARM apps as well as x86 and x64 applications. Qualcomm has launched their new 850 Snapdragon processor and this is designed specifically for Windows. The architecture for Windows on ARM, fortunately, is going to be very similar to what we've already learned for the x86 and x64. So we see user mode, which we're comfortable with. We see the shell and Cortana compiled for the ARM64, Edge, everybody's favorite browser, and of course, all of our services that are running in user mode. We also know that down below should be our kernel and drivers, and all of these will be compiled for the ARM64 platform. So let's dig a little bit deeper. Here's our Qualcomm Snapdragon 850, and you'll notice the architecture for Windows on ARM is, with great relief, very similar to what we saw already and learned already. We do see in user mode that Microsoft has already ported or recompiled a whole lot of Windows for the ARM so that it runs in a native mode. Even services like ntdll.dll, we've seen that in x86. Here it is again in our ARM platform. Even over here in our emulated processes where we have x86 code, and remember x86 represents both 32-bit and 64-bit applications. They will run with no change. You can install 32-bit and 64-bit applications just like you did in regular Windows. They will have an abstraction layer, the Windows on Windows, the WOW abstraction layer, and we also have the Chippy DLLs, which we'll get into later. There's an x86 to ARM CPU emulator, and then back down to the ntdll.dll, which will allow user processes to go into the kernel and access kernel resources. It is so critical that we have foundational concepts. In our first video, we talked about the Windows scheduler. Now let's go and take those same concepts into Hyper-V. Remember, we introduced scheduler as this complex mathematical algorithm that is in the Windows kernel, and it determines how much time and who has access to the CPU. We also know that it's preemptive multitasking, so the kernel can actually interrupt the scheduler whenever it wants to. That's what preemptive means. Because I know that guys and gals that watch this type of lecture are curious, the scheduler is actually written in assembly as many elements of the kernel. Most of Windows is written in C, but the scheduler is actually assembly language. It's done that for speed and performance. As we talk about Windows 10 and Hyper-V, the idea of the scheduler gets very complex, so this also pertains to Server 2016 and above. Let's do a quick review of Hyper-V and Windows 10 and Server 2016. So I've got my parent partition, which is often called the host OS, which in this case, if this was a Windows 10 running Hyper-V, it would be the Windows 10 platform. Then we have the child partition, which would be the virtual machines. Notice that we have ring, the user mode runs in ring three, that the kernels of both the parent and the child run in ring zero. And of course, our hypervisor runs in ring negative one. We talked about that. The newest CPUs have a feature called second level address translation or SLAT. That is a new set of instructions in our latest CPUs, and that will be in the hypervisor. In the diagram that you're seeing, I have Windows 10 Enterprise running two virtual machines. My host OS, which is Windows 10 Enterprise, is running in the parent partition in the architecture for Hyper-V. My two virtual machines are running in child partitions, and they're all sitting on top of the hypervisor. Let's begin by looking at the top left where you see my host OS, Windows 10. And you'll notice it has processes that need to be scheduled, need to go from waiting to running. And I have my scheduler, and I have a yellow diamond there, and that's representing my virtual CPU. Well, if you look at the next virtual machines, my Windows 10 virtual machine and my server 2016 virtual machine, they also are just like my host, but they also have a virtual CPU. How are these three virtual CPUs going to really access the real i9 core sitting on the motherboard? You guessed it. The hypervisor has a scheduler. This scheduler takes all those three virtual CPUs and, using a mathematical algorithm, gives them all sufficient access to that i9 core that's sitting on that motherboard. Just like my scheduler in my real Windows 10 gives all my processes access to CPU, the hypervisor scheduler does the same to the virtual CPUs. With the introduction of Windows 10 version 1803, Microsoft changed out the existing hypervisor scheduler for a new scheduler, and it's called the root scheduler. Now, let's take these concepts into server 2016 and above. So, here this diagram shows a server 2016 running two virtual machines, one Linux, one is server 2016, sitting on a hypervisor on top of a server motherboard running Xeon processors. Let's take a look. So, the same thing is true on our parent partition. We've got the actual server 2016, and it's got a scheduler and a virtual CPU. My two VMs have exactly the same thing. And down on the hypervisor for server 2016, we now have been introduced to three different schedulers in the hypervisor. If you're a server operator and you're running Hyper-V, you now have three schedulers you can choose for Hyper-V. You can go with the classic fair share scheduler, which has been the original scheduler for Hyper-V for years and years. You can also choose the core scheduler. This is new. So, depending on your loads and what you're running on your server, you may choose a core scheduler. And the root scheduler, which is new with Windows 10, that is the default scheduler for Windows 10. What's cool about this new feature of root scheduler in the hypervisor for Windows 10 is that it gives the parent partition more control over the hypervisor scheduler. This makes sense for a client operating system. Back to our process explorer and let's deep dive into the operating system. One of the processes that is shown in process explorer is a artificial process. Mark added it so that you could see how much time the system spends servicing hardware interrupts and deferred procedure calls or DPCs. High CPU consumption of these activities can indicate a hardware problem or a device driver bug. So, I've got my Windows 10 box here. I'm going to come up to process explorer and I see my interrupts. I'm just going to right mouse click and go to properties. And I can see this isn't really a process. When hardware drivers begin acting up, they have a tendency to consume more CPU. One of the things you can do is come up to columns and go into process performance and add to add context switch. This allows you to see, and I'm going to grab this and slide it over to CPU and put it right next to CPU utilization. And here I can see how often drivers or DPCs interrupted the system since I powered on. That can be very helpful. Great, Mr. Vanderpool, but I need to know which driver is causing my problem. Well, remember we have the system process. And inside the system process are a lot of your device drivers. So, I'm highlighting system or right mouse click and go to properties. Now, once I launch my system properties, I would choose the thread tab. And here I can sort all these threads. And remember in the system are not only the kernel drivers, kernel and TOS.exe, but also all the drivers. And those usually end with the extension SYS. So, I would go to my CPU tab in threads, click on that. And that sorts by highest to lowest, scroll this back up. And I could see all the different module running in system that could be impacting CPU. And generally, you're going to see one in there that's really hammering the CPU. Hopefully, it's an SYS file. And here I would choose an SYS driver that is hammering my CPU. And I could do a couple of things. One, I could hit the module button here and I could actually see its full name. And I can see the path on the hard drive where that file is found. But that really doesn't tell me what that driver does. So, I can launch system information. And I can go under software environment under system drivers. And alphabetically, I can find the actual what is that driver. So, if we have a driver issue, we're going to go to system process. We're going to right mouse click, go to system properties. We're going to look at the threads tab. And we're going to go to the CPU utilization on the threads. We're going to look for any SYS file that's running high CPU usage. Highlight it, go to module, look at the driver name. We can either use system information to find that driver and what it's for or device manager. Then go from there. The information in Process Explorer is updated and refreshed on the screen on a regular basis. If you go up to the options under view, pull the drop down menu down. You can see update speed. And the default that Mark is set for Process Explorer is one second. That means all the metrics are updated every second. Now, you can change that to half a second, two seconds, five seconds, ten seconds. You can even pause the update with the space bar. You can also hit F5 and force it to refresh. This is very important that you understand this. Because all Delta metrics are based on the last display update. So, right now I have Process Explorer running. And if I go to view, I can come down and look at my update speed. And you can see it's the default one second. Now, why is this so important? If I go over to select a column, Mark has many metrics that he allows you to observe. And notice I've got one of them is called context switch. And that means I'm looking at threads being executed. They're going from waiting to running. They're being executed by the CPU. And then I've got another one called context switch Delta. That means how many threads are being executed by the CPU in one second. The time that the display is updated. Let's go look at Process Explorer and compare these. So, over here you can see I've got context switch. Which shows me processes that have threads that are being executed. Going from waiting to running in the CPU since the process started. So, we see the count. They're getting larger and larger. Context switch Delta shows me how many threads in that same process are being executed based on the update speed of the Process Explorer. In this case, one second. So, here I can see Process Explorer is having threads executed at about 353 threads per second. Since the process started, it's been over 9 million. That's why it's so important to understand update speed. Now, if I go over here and change. And change my update speed to five seconds. You will see that my context switch Delta just sits there. And notice it's now based on five seconds update speed. Anytime you see a metrics with Delta. It's based on the Process Explorer update speed. I'm going to put this back to one second. And now I can come back. And I can see how many threads are being executed per second. Process Explorer has a right mouse click drop down menu option set that is rich in features. When you're running Process Explorer, you can go to any process in the list. And you can simply right mouse click. And you get a number of features such as set affinity, set priority, kill process, kill process tree, restart, suspend, create, dump, and others. We'll take a look at each of these features one at a time. If a process owns a visible window on the desktop. You can use the Windows submenu to bring it to the foreground. Restore it. Minimize it. Maximize it. Or close it. Now if the Windows submenu is disabled. That process doesn't own a visible window. This does not work with immersive applications. Here you see all my applications are minimized. And I'm going to go to Process Explorer. And I'm going to find Snagit. Or let's go to Snagit editor. And I can because it does have a visible window. I can bring it to the front. And it brings that application right in view. Here you can see when logon.exe is another process. But it doesn't have a visible window. And you can see the window option submenu is disabled. The set affinity option is a wonderful tool. I can't tell you how many desktops or servers. That I had a problem with one application that was running. If I had a server and I had two applications shared to the network. And one application got an update. And now that one app is just hammering the CPU. Dragging the performance of the entire server down. Now I've got two applications that are running like molasses. And of course I've got users calling. Complaining about the application performance. I could go to the one misbehaving application. Set affinity. Go down and say I want this process. And its threads to only have access. In this case CPU zero. I'm not going to give it access to any other CPUs. Dramatically lowering the CPU utilization of that process. Bringing up the other applications to decent performance. Until I can get with a developer and solve the problem. Great tool on the desktop or the server. So here I have a CPU stress utility. And I've got four active threads. You can see their status. They're running pretty hard. I've got one set at busy, busy, low and busy. So I'm really hammering the CPU. And you can see here's my CPU stress dot exe process. And my CPU utilization is already at 33% of my CPU. So this thing is really taking a hit on my overall performance. But I can right mouse click. Set affinity. And I can simply uncheck the CPU cores that this process has access to. A lot of times you can't fix an application right away. You got to get a hold of the developer. You got to call a third-party vendor. And see if they've got a fix or a patch. But you still got to run your application. So by doing this I'm going to say okay. And watch my CPU utilization. It went from real high to bearable now. I'm down to 14, 16% of my CPU. So affinity can be a real lifesaver in a pinch. Process Explorer also allows you to modify a process priority for the scheduler. So every process the developer sets the priority of that particular process. With Process Explorer you can actually right mouse click and change that priority. Having the same impact that you just saw. Much like affinity. If I had a process or an application that was really hammering the CPU again. I could use affinity or set priority. And I could actually force the process to have less priority in the scheduler. Or if it was performing too poorly. I could actually boost it and actually give it a higher priority in the scheduler. Changing a priority on a process sometimes is a quick fix. Until you can get the time to really look at the underlying problem. So let's say a user calls me. We'll just use this as an example. And their Chrome browser is just incredibly slow. You don't have a chance to look at the problem. You don't know if it's a network problem. Or something that they've added to the PC. But you can go to Chrome, right mouse click. And actually give it a higher priority to the scheduler. And sometimes that can boost the performance of that application. So you can get back and try to figure out what's really wrong. You can also do this with a server application. So this is a quick way to quickly address a problem. Giving you time to go back and really find out what the real issue is. Remember both affinity settings and priority settings are not permanent. If you reboot that PC it will go back to its default settings. A couple things you want to remember when you start killing processes. Or killing process trees. Forcibly terminating a process does not give that process an opportunity to shut down cleanly and cause data loss. Or system instability. So if you're dealing with a user's access database. I would be very careful about killing that access database. Unless you want you know for sure they're not going to lose data. In addition process explorer does not give you any extra warning. If you try to terminate a system critical process. In other words it expects intelligence behind the keyboard. Terminating a system critical process can result in a blue screen crash. Kill process. Remember you can lose data and this will force the process to terminate. Kill process tree. If the parent process. If you've selected a parent process. It will terminate both the parent and all the child processes. Restart. Remember terminates the process. And it will restart based on the command line arguments. Suspend leaves that process in memory. But removes it from the scheduler. There's no CPU cycles used. Remember don't get crazy with the kill option. And the kill process tree option. Also be aware that restart does terminate the process. I don't use the kill process. And the restart or the kill process tree on servers very often. I'm very careful. On users if they've got an application that's hung. I don't hesitate. The one that's really helpful especially on the server side is suspend. Back to the server with two applications running. I've got one application on the server that's impacting say 10 people. And I've got another application that's impacting 150 people. Well the one that's running that's being used by 10 people. Is the one that's causing me problems. I can suspend it. Leave it just hanging there. Allowing my 150 users to quit calling me. Quit texting me. Quit bothering me. Until I can figure out what the problem is with the other application. So there's a nice handy use of suspend. In the create dump option you have two. You have the create mini dump and the create full dump. This allows you to select any process and capture a mini dump or a full dump. Keep in mind that capturing a dump does not terminate the process. Mark added a graphical system information on the toolbar. You see the red arrow pointing to it. When you click on that you get the graphical system information. Under the summary tab you have CPU usage. And remember green is user mode. Red is kernel. System commit which is a good way of looking at your virtual memory. Then your physical memory. IO this is input output within the process. And within other processes. So this is IO within the inside the process. And also IO between the processes. You also have network and disk. With the CPU tab you can see all of your cores. If you check show one graph per CPU it'll show you all your cores. It allows you to see your handles threads and processes. All of this is based on your update speed. So all of this is based on one how many handles per second. How many context switch per second. How many interrupt delta or interrupts per second. So all this information is shown based on your update speed. Red is kernel mode. Green is user mode. If you take your system information dialog box and stretch it out. You can actually see more history on the graphs. This is a look at memory. So the official terminology or the official definition of commit charge. Is a total amount of private bytes committed across all processes plus the page pool. In essence is virtual memory. Commit limit is the amount of private bytes that can be committed. Without increasing your page file size. Now I'm going to get into memory and applications later. So I'm just going to stop at this point and keep on going. Here's your IO tab. Again you have IO network and disk. Under the GPU tab you get more information about your video card. So if you're gaming this is a great way of looking at your GPU. You also have a button for engines. And allows you to look at all the GPUs that you have on your card. This little computer icon on this system on the toolbar. Allow you to launch your graphical system information. And you've got all your tabs. Let's take a look at some of the options in process explorer. When you go up to the options in your drop down menu. You can see you got run on log on. And this allows you to configure a scheduled task. That triggers the running of process explorer when your user account logs on. Very handy to troubleshoot processes that are locking your computer once you log on. Always on top this puts process explorer on top of all windows. You can also set up a scroll to new processes. So whenever a new process is created it automatically goes to that process. You can come up to options. And you can see there's all a number of things that you can do. You can also do a confirmation kill. So that when you kill a process it's going to make sure that that's what you want to do. Allow one instance prevents two process explorers launching at the same time. It only allows one instance of process explorer. Another nice option is the view. If you drop down the view drop down menu. You will see the ability to show lower pane. Which you can see with the red arrow. And you can choose what is displayed. So you choose a process. And you can choose either show me the DLLs in that process. Or show me the handles in that process. The icon that you see that looks like a pair of binoculars is the process explorer search. And it will allow you to search for a handle or a DLL. Showing DLLs and processes using the lower pane. The beauty of process explorer is it allows me to look inside the process. With a click of the button selecting any process that I so desire. I can now open that process see what is inside. Many times not so that I'm going to go and redevelop or rewrite the application. But that I can communicate more intelligently with a software developer. Or a third-party vendor to help them resolve the issue. So looking at process explorer. I can come up with my mouse and come up to this tooltip. And it says show lower pane. By clicking it I can turn on the lower pane. By clicking it again it takes it away. If I go up to click on the lower pane. And I come up to view. And I pull my drop down menu. I can decide what is shown in the lower pane. I can show DLLs or I can show handles. I'm going to choose chrome. And I'm displaying all the DLLs inside the process chrome.exe. Remember DLLs are the heart and soul of a process. There are times in this video that I have said things like. These processes are being executed by the CPU. And I only said that because I want to keep the concept simple. But now that you're on video two. Let's get the facts. Processes are not executed by the CPU in reality. It's the contents inside. And that's the threads. And you can see the threads when you look at the DLLs. Now I can look at chrome's threads by going to chrome.exe. Right mouse click. Go to properties. And by clicking on the threads tab. Those are the threads that are running in chrome.exe. You'll notice though in my lower pane view. I've got a lot of DLLs. Every DLL in the process is not a thread. But most of the threads that are shown in the process properties. Are going to be DLLs. Application developers when they write to Windows. Are offered a library. A rich library of functions and services. How to access the network. How to access the file system. How to use a clipboard. All of those library of services and features. Are provided in the Windows operating system. To a developer via what is known as APIs. Those APIs are translated into files. That provide those functions. And in Windows are called DLLs. You notice I'm in system 32. And I've searched the system 32 folder. For any file in the system 32. That ends in DLL. And you'll notice I end up with 4,282 DLLs. Those are the libraries of services and features. That are pulled up by processes. Into their process. So they can function. All processes in Windows do that. So here I'm looking at Google. This is chrome. This is my main process. That's launching the chrome browser. And I've got my lower pane showed. And you can see all the DLLs. Most of these probably 90% of these. Are Microsoft DLLs. They are provided by the operating system. To developers whether it's Google. Or Adobe or whoever it is. Very few of these are actually written by Google. In fact you could see at the bottom. Where I actually go into files. That are actually written by Google. Not too many. Most of the DLLs that run chrome browser. Are actually Windows libraries. To make it function. This is true of all processes. If I want to determine what DLL is running a thread. Then I have to come to the process. Go to properties. Go to the threads tab. And I can click on any one of these DLLs. Go to module. And that actually shows me the DLL. That is running that particular thread. So you can see that the browser. Chrome browser. Has many threads in it. Those DLLs are actually running threads. All the DLLs in the lower pane. May or may not be threads. But they are being used by the process. Another very important thing to remember. Is you can't go to a DLL. And double click it and run it. The only thing that can execute a DLL. Is a process. So DLLs can many times be executables. They actually run code. But the only thing that can run a DLL. Is a process. The process explorer also allows us. In the lower pane. To see the handles. Held by the process. And the threads that are running. Inside the process. Here we're actually looking at. The lower pane of chrome.exe. And we're looking at handles. Remember handles represent. A number of things. As threads go out. And request registry keys. They open files. They lock folders. They produce graphical objects. All of those elements. Are considered handles. Many times when you want to delete a file. And it will tell you. This file is owned by another process. And it's locked. And you can't. That's a handle. In other words. Some process has said. This file is mine. And it won't let you delete it. Until you get that process. To release that particular file. That's why handles are very important. Mark allows us to use color. To better understand processor behavior. And processor types. There are a number of protected processes. They are basically. Their primary purpose. Is to defend critical system processes. Protected processes. Change access rules. So that even the system account. Which is basically root. And other administrators. Are blocked from any control. Or access to these sensitive processes. Windows designates certain processes. As protected. Based on special digital signatures. In the process image file. So in your options. You can pull. Look at your color selection. And check the box. For protected process. It's kind of a bright pink. And then say okay. And you will see. All of these processes. Highlighted with a bright pink color. These are all protected processes. Another aspect of processes. Is the concept of jobs. Jobs allow the control. Or grouping of many processes. So that they all have the same behavior. So I can group. A number of process together. And assign them certain behaviors. Under your color selection. Go to jobs. And check the box for jobs. And then hit okay. And you'll see a lot of. Primarily services. That will light up in the brown color. When you go to a process. That is highlighted brown. In other words. It's under the auspices. Or the extension of a job. You can go to its properties. And you'll see a tab called jobs. If you look at processes in the jobs. It will show you. All of the processes in that group. And then down under job limits. It will show you the restrictions. For example. A lot of processes in a job. Will say. If you terminate this process. Then you have to terminate. All the processes in that job. Or they can all. Every process in this job. Can only access. So much physical memory. So there's just things. That you can group. And control a lot of processes. And that's called a job. Another nice feature. Is you can do the dot net. So again. Under colors. Check the dot net. And hit okay. And you can see all the processes. That were written. Under the dot net frameworks. As we look at the security tab. On any process. We have a section. In the security tab. That focuses on access entries. These are users or groups. That are assigned to this process. Notice the ones that are denied. Process cannot access. The local administrator. Or the administrator group. Or the built-in administrators. Or anything. So that is a protection. Against that process. Under privileges. Down in the bottom. It's important to understand. That every privilege. In that list. Can be changed. By the process. Depending on how it's written. But if there's. Not a privilege shown. It cannot be used. So if the developer. Did not add the privilege. It can't change it. It can't. You cannot use. What is not in the box. So if there's no privilege. It cannot use the privilege.