Author Archives: Thomas Weller

WinDbg exception codes

People sometimes wonder what exception number is behind the event codes of WinDbg. Since I didn’t find a list in WinDbg’s help file, I post a list here. Many error codes are defined in NtStatus.h. Some of the codes are still missing; if you know it or find anything else missing, you can send an email to windbg.yyyy-mm-dd@lockerflockig.de where yyyy-mm-dd is the current date.

WinDbg event code Exception number
3c (port disconnected) 0xC0000037
If you want to break on 0x0000003C, specify 0x3c
aph (Application hang) 0xCFFFFFFF
asrt (Assertion failure) 0xC0000420
av (Access Violation) 0xC0000005
bpe (Break instruction) 0x80000003
bpec (Break instruction continue) N/A, just defines how to continue bpe
cce (Control-C / Control-Break exception) 0x40010008 / 0x40010005
WinDbg seems to have the same name for both.
If you want to break on 0x00000CCE, specify 0xcce
cc (Control-C / Control-Break continue) If you want to break on 0x000000CC, specify 0xcc
ch (Invalid handle) 0xC0000008
clr (.NET exception) 0xE0434F4D
This is “.COM” in ASCII.
clrn (CLR notification) 0xE0444143
This is “.DAC” in ASCII.
dm (Data misaligned) 0x80000002
dbce (Debugger command exception) 0x40010009
dz (Divide by zero) 0xC0000094
eh (C++ EH exception) 0xE06D7363
This is “.MSC” in ASCII
gp (Guard page violation) 0x80000001
hc (Handle continue) N/A, just defines how to continue ch
ii (Illegal instruction) 0xC000001D
iov (Integer overflow) 0xC0000095
isc (Invalid system call) 0xC000001C
lsq (Lock sequence invalid) 0xC000001E
rto (Runtime originate error) 0x40080201
rtt (Runtime transform error)
sbo (Stack buffer overrun) 0xC0000409
sov (Stack overflow) 0xC00000FD
Note that there is also another stack overflow exception,
0xE053534F (ASCII “.SSO”)
svh (Service hang)
sse (Single step execution) 0x80000004
ssec (Single step continue) N/A, just defines how to continue sse
vcpp (Visual C++) 0x406D1388
vs (Verifier stop) 0xC0000421
wkd (Wake debugger) 0x80000007
wob (WOW breakpoint) 0x4000001F
wos (WOW single step) 0x4000001E

The following events are not exceptions, so they have no exception number:

WinDbg event code Exception number
ct (Create thread) N/A
cpr (Create process) N/A
ct (Create thread) N/A
epr (Exit process) N/A
et (Exit thread) N/A
ibr (Initial breakpoint) N/A
iml (Initial module load) N/A
ld (Load module) N/A
out (Debug output) N/A
ser (System error) N/A
ud (Unload module) N/A

VMware Workstation Setup: invalid drive D:\

When I tried to update VMWare 10.0.3 to the latest 10.0.7 I got an error message telling me that drive D: is invalid. In the msi.install.log file, I could read

...
MSI (s) (F0:78) [23:54:34:958]: Note: 1: 1327 2: D:\ 
Action start 23:54:34: CostFinalize.
MSI (s) (F0:78) [23:54:34:958]: Product: VMware Workstation -- Error 1327. Invalid Drive: D:\
Error 1327. Invalid Drive: D:\
...

My Google search returned a VMWare webpage which explains that the problem is related to UserShellFolders – which in my case turned out to be completely legitimate values. Many other sources gave the same answer, which was not helpful for me.

So I tried to resolve it myself and restarted the installation with Process Monitor running. After the problem happened, I searched the log for D:\ and it turned out that there is a Registry key at

HKLM\SOFTWARE\Wow6432Node\VMware, Inc.\Installer\VMware Workstation

which contains a value “DATASTORE_PATH” that points to a directory for virtual machines. Correcting that path resulted in a working installation. If I knew that before, I could also have found out in the MSI log file, since the datastore path is mentioned:

Property(S): DATASTORE_PATH.9F27F70C_3D9B_4082_8801_88387445C782 = D:\Virtual PCs\VMWare\SharedVMs

Anyway, it was a straight-forward debugging session and I’m glad it worked.

Process Monitor Log Analyzer

Process Monitor Log Analyzer LogoProcess Monitor is a good tool to detect missing files of your application. However, the process to find the one which is causing the issue is tedious. Starting from the bottom, you go through all the files which have a “Path not found” or “Name not found” error message.

We have automated this process and are glad to give you a convenient tool that displays all files which had at least one error. The result is grouped by process and sorted by the number of  errors descending, so typically the culprit is listed on top, like in the screenshot:

Process Monitor Log Analyzer Screenshot

Process Monitor Log Analyzer Screenshot

We’re also merging DLL, EXE and SYS files into one category, since those typically belong together. This makes the result even better.

Steps to use Process Monitor Log Analyzer:

  1. Start SysInternals Process Monitor
  2. Set a filter for your process to get rid of other noise
  3. Reproduce the issue, typically just run your application
  4. Save the Process Monitor result as XML file
  5. Load the XML file in Log Analyzer

You can filter the events by right-clicking one of the grouping boxes and selecting “Filter Editor…”.

Download

Procmon Log Analyzer Setup 0.2.3.42 (6.5 MB)

License

Creative Commons License
This work is licensed under a Creative Commons Attribution-NoDerivatives 4.0 International License.

This means: you can use it commercially, redistribute it by naming the author (just link here, please) but are not allowed to modify it. Source Code will not be provided.

Bugs

You can report bugs to procmonanalyzer.yyyy-mm-dd@lockerflockig.de, where yyyy-mm-dd is the current date (prevents spam) or register an account at our Bug Tracker.

System requirements

  • .NET framework 4.5 (not included in the installer)
  • Windows Vista SP2 or higher (tested on Windows 7 SP1 x64, 8.1 x32 and 10 x64 Preview)

Live System Timeline Builder

Description

Live system timeline builder retrieves a lot of data from a running Windows machine and combines all data to a super timeline. At the moment it only uses NirSoft tools to achieve the task. The overall timeline can be exported into an Excel file to be filtered etc.

Live system timeline builder is similar to Nirsoft Mega Report, a batch file that basically collects the same information but stores it in HTML format and does not combine the information. However, Timeline Builder only runs tools which contribute to the timeline, i.e. have some kind of time-stamp information inside.

Timeline Builder Screenshot

The tool comes in two flavors: developer mode and forensic mode.

  • The developer mode might be useful for developers when trying to find out why an application does not work any more. It focuses on changes to the system, application crashes, driver installations and blue screens.
  • The forensic mode is more useful for administrators or people interested in forensic research. Note that it cannot perform a true forensic analysis, for which you would use a read-only copy of the hard disk. Forensic mode focuses on user activity such as browser history, chat logs and document changes.

Download

Download Live System Timeline Builder 0.8.2.0 Setup (2.2 MB)

System Requirements

Live System Timeline Builder needs .NET 4.0 and therefore Windows XP Service Pack 3.

Known issues

The following tools are not included yet:

  • SafariCacheView
  • SafariHistoryView
  • OperaCacheView

Some of the Nirsoft Tools do not export the data into valid XML. The program will try to correct some of those issues but cannot correct all of them. When the Nirsoft Tools are updated regarding this issue, you can either replace them in the installation directory or download a new version of Timeline Builder when available from this website. When an error was found, the tool in question is highlighted in red. A tooltip shows what the problem was. Candidates for issues are MozillaCookiesView and SkypeLogView.

InvalidXMLInfoTooltip

License

Since the tool mainly consists of Nirsoft tools, the Nirsoft license policy applies. This means that you can also use the tool commercially. From http://nirsoft.net/about_nirsoft_freeware.html (rev. 2014-10-22)

My utilities are completely freeware, without any catch.

And an example from one of the tool specific pages (http://www.nirsoft.net/utils/userassist_view.html (rev. 2014-10-22)) but similar for all tools:

This utility is released as freeware. You are allowed to freely distribute this utility via floppy disk, CD-ROM, Internet, or in any other way, as long as you don’t charge anything for this. If you distribute this utility, you must include all files in the distribution package, without any modification !

Windows Error Reporting activating LocalDumps in the short term

Developers have the situation that an application crashes and Windows already shows the Windows Error Reporting dialog. The previous article analyzed the influence of attaching a debugger while the dialog is shown. This article uses the Windows Registry to enable the LocalDumps feature just before the application is terminated.

Assume that the application crashed and is already showing the Windows Error Reporting dialog. While the system is waiting for your response, you can go to the Registry at the LocalDumps key (Windows Vista SP1 or higher) and make the necessary entries for saving crash dumps on hard disk.

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps

I tried this and in fact got a crash dump written to disk. This is what it tells:

0:001> !findstack ntdll!DbgBreakPoint
0:001> *** There's no debug breakpoint injected
0:001> kn
 # ChildEBP RetAddr  
00 01b8f904 770715f7 ntdll!NtWaitForMultipleObjects+0x15
01 01b8f9a0 75b319f8 KERNELBASE!WaitForMultipleObjectsEx+0x100
02 01b8f9e8 75b34200 kernel32!WaitForMultipleObjectsExImplementation+0xe0
03 01b8fa04 75b580a4 kernel32!WaitForMultipleObjects+0x18
04 01b8fa70 75b57f63 kernel32!WerpReportFaultInternal+0x186
05 01b8fa84 75b57858 kernel32!WerpReportFault+0x70
06 01b8fa94 75b577d7 kernel32!BasepReportFault+0x20
07 01b8fb20 776574ff kernel32!UnhandledExceptionFilter+0x1af
08 01b8fb28 776573dc ntdll!__RtlUserThreadStart+0x62
09 01b8fb3c 77657281 ntdll!_EH4_CallFilterFunc+0x12
0a 01b8fb64 7763b499 ntdll!_except_handler4+0x8e
0b 01b8fb88 7763b46b ntdll!ExecuteHandler2+0x26
0c 01b8fbac 7763b40e ntdll!ExecuteHandler+0x24
0d 01b8fc38 775f0133 ntdll!RtlDispatchException+0x127
0e 01b8fc38 6f8c20ce ntdll!KiUserExceptionDispatcher+0xf
0f 01b8ff88 75b3338a FirstSteps+0x20ce
10 01b8ff94 77619f72 kernel32!BaseThreadInitThunk+0xe
11 01b8ffd4 77619f45 ntdll!__RtlUserThreadStart+0x70
12 01b8ffec 00000000 ntdll!_RtlUserThreadStart+0x1b
0:001> *** Callstack looks like the bad WER callstack from previous example
0:001> .exr -1
ExceptionAddress: 6f8c20ce (FirstSteps+0x000020ce)
   ExceptionCode: c000001d (Illegal instruction)
  ExceptionFlags: 00000000
NumberParameters: 0
0:001> *** But the exception is good
0:001> .frame 0f
0f 01b8ff88 75b3338a FirstSteps+0x20ce
0:001> u
FirstSteps+0x20ce:
6f8c20ce 0f0b            ud2
0:001> *** And changing the frame disassembles to the correct instruction

I have not found official documentation about the timing of the LocalDumps Registry key, so I can’t guarantee that this will work for all times. At least it worked on my Windows 7 SP1 x64 machine at the time of writing.

Windows Error Reporting influence when attaching a debugger

Developers have the situation that an application crashes and Windows already shows the Windows Error Reporting dialog. Which steps should be taken to get a useful dump in this situation? This article describes what happens when a debugger is attached while the dialog is shown. Another article describes the situation when WER LocalDumps is activated short-term.

Assume that the application just crashed and is now showing the typical WER dialog. Obviously “Close program” would terminate the application, so a developer decides to attach the debugger now and create a dump before everything is gone.

When opening the dump later, the exception you will see is just the breakpoint which was triggered by the debugger itself, so that’s not very useful.

0:002> .exr -1
ExceptionAddress: 775f000c (ntdll!DbgBreakPoint)
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 1
   Parameter[0]: 00000000
0:002> k
ChildEBP RetAddr  
02e2ff58 7767f926 ntdll!DbgBreakPoint
02e2ff88 75b3338a ntdll!DbgUiRemoteBreakin+0x3c
02e2ff94 77619f72 kernel32!BaseThreadInitThunk+0xe
02e2ffd4 77619f45 ntdll!__RtlUserThreadStart+0x70
02e2ffec 00000000 ntdll!_RtlUserThreadStart+0x1b

Looking at other threads, you’ll find a callstack that was injected by Windows Error Reporting:

0:001> k
ChildEBP RetAddr  
01aff904 770715f7 ntdll!NtWaitForMultipleObjects+0x15
01aff9a0 75b319f8 KERNELBASE!WaitForMultipleObjectsEx+0x100
01aff9e8 75b34200 kernel32!WaitForMultipleObjectsExImplementation+0xe0
01affa04 75b580a4 kernel32!WaitForMultipleObjects+0x18
01affa70 75b57f63 kernel32!WerpReportFaultInternal+0x186
01affa84 75b57858 kernel32!WerpReportFault+0x70
01affa94 75b577d7 kernel32!BasepReportFault+0x20
01affb20 776574ff kernel32!UnhandledExceptionFilter+0x1af
01affb28 776573dc ntdll!__RtlUserThreadStart+0x62
01affb3c 77657281 ntdll!_EH4_CallFilterFunc+0x12
01affb64 7763b499 ntdll!_except_handler4+0x8e
01affb88 7763b46b ntdll!ExecuteHandler2+0x26
01affbac 7763b40e ntdll!ExecuteHandler+0x24
01affc38 775f0133 ntdll!RtlDispatchException+0x127
01affc38 6f8c20ce ntdll!KiUserExceptionDispatcher+0xf

A shortcut to check for the presence of such a callstack is

0:001> !findstack kernel32!WerpReportFault
Thread 001, 2 frame(s) match
        * 04 01affa70 75b57f63 kernel32!WerpReportFaultInternal+0x186
        * 05 01affa84 75b57858 kernel32!WerpReportFault+0x70

So, creating a crash dump while the WER dialog is shown does not produce good results. The better way to create a dump is to proceed like this:

  1. Enter g in the debugger to let the application continue
  2. Press the “Debug” button of the WER dialog
  3. Confirm the warning about the debugger which is already attached
  4. Click “No” when asked to start debugging using the selected debugger

After that the second chance exception is thrown again and caught by the already attached debugger so that you can take a nice crash dump. In my example the dump now evaluates to

0:001> .exr -1
ExceptionAddress: 6f8c20ce (FirstSteps+0x000020ce)
   ExceptionCode: c000001d (Illegal instruction)
  ExceptionFlags: 00000000
NumberParameters: 0
0:001> k
ChildEBP RetAddr  
WARNING: Stack unwind information not available. Following frames may be wrong.
01afff88 75b3338a FirstSteps+0x20ce
01afff94 77619f72 kernel32!BaseThreadInitThunk+0xe
01afffd4 77619f45 ntdll!__RtlUserThreadStart+0x70
01afffec 00000000 ntdll!_RtlUserThreadStart+0x1b

which is exactly the expected exception for my demonstration application.

ConvertStore Errors

When converting a 2-tier symbol store to a 3-tier symbol store, you might get some error codes without error messages. Therefore it is hard to find out what the problem is and how to solve it.

The first error message you might see is probably
C\...>convertstore
Failed initial checks.

This happens if you run convertstore without command line parameters. The correct syntax is
C\...>convertstore -s d:\debug\symbols
Failed initial checks.

If you still get the same error, this could mean that the symbol store is already a 3-tier symbol store.

The error
C\...>convertstore -s d:\debug\symbols
Locking Symbol Store to prevent transactions while this operation completes.
Failed to lock Symbol Store. Error 0x00000003.

occurs if your symbol store does not have a pingme.txt file or no 000Admin subfolder. Just create an empty (0 byte) pingme.txt file and an empty 000Admin folder and you’ll probably get rid of this error.

During the conversion you may also get
C\...>convertstore -s d:\debug\symbols
Locking Symbol Store to prevent transactions while this operation completes.
Iterating store: d:\debug\symbols
ERROR: Failed to move d:\debug\symbols\kernel32.pdb -> d:\debug\symbols\ke\kernel32.pdb. Error 0x00000005.
ERROR: Failed to move d:\debug\symbols\ntdll.pdb -> d:\debug\symbols\nt\ntdll.pdb. Error 0x00000005.
Symbol Store no longer locked.

In this case, the file is in use by something else. Close programs that are using the files, delete index2.txt from your symbol store and run the command again.

If you get

C\...>convertstore -s d:\debug\symbols
Locking Symbol Store to prevent transactions while this operation completes.
Iterating store: d:\debug\symbols
ERROR: Failed to move d:\debug\symbols\advapi32.pdb -> d:\debug\symbols\ad\advapi32.pdb. Error 0x000000B7.

then the destination file already exists. This could happen if you had a 2-tier store with some files not moved, then downloaded files from the Internet and you’re now trying to move the 2-tier files over the downloaded 3-tier files. In this case you can just delete the affected 2-tier folder.

Mscordacwks collector

Did you ever grab a crash dump from a customer’s machine and you were not able to analyze it in WinDbg, because you didn’t have the matching mscordacwks.dll and SOS.dll? Until you recognize the mistake lies in the missing files, the customer has installed an update or reset the machine?

Never have this sort of trouble again with the Mscordacwks / SOS collector application. It scans the PC for those DLLs, copies them into a directory of your choice and renames them according their version number.

The program will look in

  • .NET framework 2.0 folder
  • .NET framework 4.0 folder
  • SXS folders

System requirements: .NET 2.0
License: free for personal and commercial use

Download Mscordacwks Collector 1.1.4.0 (.exe.zip) (67 kB)

If you encounter any bugs, you can report to mscordacwks.yyyy-mm-dd@lockerflockig.de (where yyyy-mm-dd is the current date).

Version history

1.0.0.0 (2014-02-12)

  • Preliminary release

1.0.1.0 (2014-03-08)

  • Release
  • Hourglass cursor during file search
  • Display intermediate search results

1.0.2.0 (2014-06-19)

  • Usability improvements
  • Preview of full output directory
  • Support for environment variables in path
  • Selection of .NET framework path

1.1.4.0 (2014-09-28)

  • Output structure customizable
  • Add Logo
  • Started writing unit tests and build script

Source code

SVN URL: http://opensource.svn.wellisolutions.de/MscordacwksCollector/
Username: guest
Password: debugging

If you want to contribute, you can request access rights by email to mscordacwks.yyyy-mm-dd@lockerflockig.de (where yyyy-mm-dd is the current date).

WinDbg Versions

WinDbg, the probably most powerful debugger for Windows is part of the Microsoft Platform SDK or Microsoft Windows SDK. Recently, the version numbering changed and maybe confused some developers.

When searching the Internet, I could not find a complete list of WinDbg versions including download links and release dates. The probably most official website listing version numbers is available in MSDN.  As WinDbg is part of Microsoft Windows SDKs, some filter on the Microsoft download center provide links to SDKs for download. Of course, Wikipedia was also helpful. So I assembled my own table and I want to share it with you:

Version Part of this SDK Release Date
WinDbg 4.0.18 (FTP Download) 2001-12-17
WinDbg 6.1.0017.1 (FTP Download) ~2002-12-14
WinDbg 6.1.0017.2 Microsoft Platform SDK February 2003
(Download information)
2003-03-26
WinDbg 6.2.0013.1 ? ?
WinDbg 6.3.0005.? Beta ? ?
WinDbg 6.3.0017.0 Microsoft Platform SDK August 2004
(Download information)
2004-08-21
WinDbg 6.4.0004.4 ? ?
WinDbg 6.4.0007.0
(Setup says 6.4.0007.2)
Windows Server 2003 SP1 Platform SDK
(Download) and also
Windows Server 2003 R2 Platform SDK
(Download)
2005-05-02
and
2006-03-14
WinDbg 6.5.0003.8 ? 2005-04-24
WinDbg 6.6.0003.5 ? 2006-01-18
WinDbg 6.6.0007.5 Microsoft Windows SDK
for Windows Vista and .NET Framework 3.0
(Web installer)
2006-10-30
WinDbg 6.7.0005.0 WinDbg version with .NET support built-in according to Volker von Einem 2007-04-??
WinDbg 6.7.0005.1 .NET support was removed again according to Volker von Einem 2007-06-20
WinDbg 6.8.0004.0 ? 2007-09-27
WinDbg 6.9.0003.113 ? 2008-03-20
WinDbg 6.10.0003.233 ? 2008-09-08
WinDbg 6.11.0001.402 ? 2009-01-30
WinDbg 6.11.0001.404 Microsoft Windows SDK
for Windows 7 and .NET Framework 3.5 SP 1
(Download)
2009-02-25
WinDbg 6.12.0002.633 Microsoft Windows SDK
for Windows 7 and .NET Framework 4
(Download)
2010-05-21
WinDbg 6.13.0001.776 mentioned in Mark Russinovich’s blog <2011-01-29
WinDbg 6.13.0008.1108  ? ?
WinDbg 6.13.0009.1140 used in the book “Practical Reverse Engineering: x86, x64, ARM, Windows Kernel, Reversing”, page 188 ?
WinDbg 6.1.7600.16385 (?) ? 2009-07-24
WinDbg 6.2.8229.0 Windows 8 ? <2012-04-04
WinDbg 6.2.8400.0 Windows 8 ? <2012-06-23
WinDbg 6.2.9200.16384 Microsoft Windows SDK
for Windows 8 and .NET Framework 4.5
(Download)Last version that runs on Windows XP
2012-11-15
WinDbg 6.3.9600.16384 Microsoft Windows SDK
for Windows 8.1
(Download)
2013-10-17
WinDbg 6.3.9600.17200 Updated version of
Microsoft Windows SDK
for Windows 8.1
2014-06-06
WinDbg 6.3.9600.17298 Updated version of
Microsoft Windows SDK
for Windows 8.1
2014-10-25
WinDbg 10.0.10069.9 Used by Andrew Richards
in Defrag Tools Episode #136
<2015-04-29
WinDbg 10.0.10075.9 Windows 10, direct download from CodeMachine 2015-04-29
WinDbg 10.0.10586.15 A version I got when I installed WinDbg via Chocolatey 2015-11-20 (?)
WinDbg 10.0.10586.567 Windows 10, build 1511 (TH2), direct download from CodeMachine 2015-10-30
WinDbg 10.0.14321.1024 Windows 10, build 1607, comes with a console version of GFlags 2016-07-29
WinDbg 10.0.15003.1001 Windows 10 SDK preview 14951 2017-01-04
WinDbg 10.0.15063.137 Windows 10 SDK Creators Update, build 1703 2017-03-30
WinDbg 10.0.15063.400 Windows 10 SDK 2017-05-09

To keep updates simple, any corrections can be sent to windbg.yyyy-mm-dd@lockerflockig.de, where yyyy-mm-dd is the current date (to bypass my email spam filter).

Most times, the installer will detect the bitness of the operating system and then install the files for that bitness. So if you like both versions of WinDbg (32 bit and 64 bit), you have to install twice on different operating systems. Interestingly, the 2003 version still installs on newer PCs (Intel Core i5) but the 2004 version doesn’t.

Note that the installer will often not install WinDbg directly but instead place another MSI installer into the Setup folder of the target directory. If you can’t find WinDbg.exe, you probably need to run another installer.

Once you went through all the installation processes, you can make WinDbg portable by simply copying the whole directory to your USB stick (or wherever you want to have it available).

SysInternals Administrator’s Reference

Although I thought I am quite familiar with Sysinternals Tools because I use them almost every day, I bought the Windows Sysinternals Administrator’s Reference due to a recommendation of a colleague. Even if I would not have read the book, it would look great in my office and people could see what I’m familiar with.

After reading the first few pages, I realized that I was completely wrong. The book covers the tools really in depth and as you can see from the image, I have added lots of markers. There are about 80 blue markers which indicate things I didn’t know about Sysinternals tools before and which I’d like to remember. There are also about 10 yellow markers which indicate items I want to cover by a new tool. I used green markers for experiments which I want to repeat – I didn’t execute them when reading the book, because I mostly read it in bed in the evening.

A few orange items indicate mismatches between my current understanding of how things work and how they are described in Mark Russinovichs book. These are probably paragraphs which I will check against other resources, such as the Windows Internalsbook (Mark Russinovich), Inside Windows Debugging (Tarik Soulami) or Advanced Windows Debugging (Mario Hewardt).

The book starts with some core concepts, which is a really short summary of Windows Internals. The next chapters cover Process Explorer and Process Monitor in detail, followed by a description of Autoruns and PsTools. The subsequent chapters bundle process and diagnostic utilities, security tools as well as Active Directory stuff. File, disk and network utilities are covered before the book explains system information utilities and the miscellaneous tools.

Part three of the book depicts some real-word examples of troubleshooting cases which were solved using Sysinternals utilities. This part is split into three parts: analysis of error messages, debugging performance issues and malware removal.

All in all I was really impressed by the depth of this book. I believe that this book should not only be read by every person involved in debugging, but also by every developer. I can only recommend Sysinternals Administrator’s Reference.