Category Archives: WinDbg

Windows Error Reporting activating LocalDumps in the short term

Developers have the situation that an application crashes and Windows already shows the Windows Error Reporting dialog. The previous article analyzed the influence of attaching a debugger while the dialog is shown. This article uses the Windows Registry to enable the LocalDumps feature just before the application is terminated.

Assume that the application crashed and is already showing the Windows Error Reporting dialog. While the system is waiting for your response, you can go to the Registry at the LocalDumps key (Windows Vista SP1 or higher) and make the necessary entries for saving crash dumps on hard disk.

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps

I tried this and in fact got a crash dump written to disk. This is what it tells:

0:001> !findstack ntdll!DbgBreakPoint
0:001> *** There's no debug breakpoint injected
0:001> kn
 # ChildEBP RetAddr  
00 01b8f904 770715f7 ntdll!NtWaitForMultipleObjects+0x15
01 01b8f9a0 75b319f8 KERNELBASE!WaitForMultipleObjectsEx+0x100
02 01b8f9e8 75b34200 kernel32!WaitForMultipleObjectsExImplementation+0xe0
03 01b8fa04 75b580a4 kernel32!WaitForMultipleObjects+0x18
04 01b8fa70 75b57f63 kernel32!WerpReportFaultInternal+0x186
05 01b8fa84 75b57858 kernel32!WerpReportFault+0x70
06 01b8fa94 75b577d7 kernel32!BasepReportFault+0x20
07 01b8fb20 776574ff kernel32!UnhandledExceptionFilter+0x1af
08 01b8fb28 776573dc ntdll!__RtlUserThreadStart+0x62
09 01b8fb3c 77657281 ntdll!_EH4_CallFilterFunc+0x12
0a 01b8fb64 7763b499 ntdll!_except_handler4+0x8e
0b 01b8fb88 7763b46b ntdll!ExecuteHandler2+0x26
0c 01b8fbac 7763b40e ntdll!ExecuteHandler+0x24
0d 01b8fc38 775f0133 ntdll!RtlDispatchException+0x127
0e 01b8fc38 6f8c20ce ntdll!KiUserExceptionDispatcher+0xf
0f 01b8ff88 75b3338a FirstSteps+0x20ce
10 01b8ff94 77619f72 kernel32!BaseThreadInitThunk+0xe
11 01b8ffd4 77619f45 ntdll!__RtlUserThreadStart+0x70
12 01b8ffec 00000000 ntdll!_RtlUserThreadStart+0x1b
0:001> *** Callstack looks like the bad WER callstack from previous example
0:001> .exr -1
ExceptionAddress: 6f8c20ce (FirstSteps+0x000020ce)
   ExceptionCode: c000001d (Illegal instruction)
  ExceptionFlags: 00000000
NumberParameters: 0
0:001> *** But the exception is good
0:001> .frame 0f
0f 01b8ff88 75b3338a FirstSteps+0x20ce
0:001> u
FirstSteps+0x20ce:
6f8c20ce 0f0b            ud2
0:001> *** And changing the frame disassembles to the correct instruction

I have not found official documentation about the timing of the LocalDumps Registry key, so I can’t guarantee that this will work for all times. At least it worked on my Windows 7 SP1 x64 machine at the time of writing.

Windows Error Reporting influence when attaching a debugger

Developers have the situation that an application crashes and Windows already shows the Windows Error Reporting dialog. Which steps should be taken to get a useful dump in this situation? This article describes what happens when a debugger is attached while the dialog is shown. Another article describes the situation when WER LocalDumps is activated short-term.

Assume that the application just crashed and is now showing the typical WER dialog. Obviously “Close program” would terminate the application, so a developer decides to attach the debugger now and create a dump before everything is gone.

When opening the dump later, the exception you will see is just the breakpoint which was triggered by the debugger itself, so that’s not very useful.

0:002> .exr -1
ExceptionAddress: 775f000c (ntdll!DbgBreakPoint)
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 1
   Parameter[0]: 00000000
0:002> k
ChildEBP RetAddr  
02e2ff58 7767f926 ntdll!DbgBreakPoint
02e2ff88 75b3338a ntdll!DbgUiRemoteBreakin+0x3c
02e2ff94 77619f72 kernel32!BaseThreadInitThunk+0xe
02e2ffd4 77619f45 ntdll!__RtlUserThreadStart+0x70
02e2ffec 00000000 ntdll!_RtlUserThreadStart+0x1b

Looking at other threads, you’ll find a callstack that was injected by Windows Error Reporting:

0:001> k
ChildEBP RetAddr  
01aff904 770715f7 ntdll!NtWaitForMultipleObjects+0x15
01aff9a0 75b319f8 KERNELBASE!WaitForMultipleObjectsEx+0x100
01aff9e8 75b34200 kernel32!WaitForMultipleObjectsExImplementation+0xe0
01affa04 75b580a4 kernel32!WaitForMultipleObjects+0x18
01affa70 75b57f63 kernel32!WerpReportFaultInternal+0x186
01affa84 75b57858 kernel32!WerpReportFault+0x70
01affa94 75b577d7 kernel32!BasepReportFault+0x20
01affb20 776574ff kernel32!UnhandledExceptionFilter+0x1af
01affb28 776573dc ntdll!__RtlUserThreadStart+0x62
01affb3c 77657281 ntdll!_EH4_CallFilterFunc+0x12
01affb64 7763b499 ntdll!_except_handler4+0x8e
01affb88 7763b46b ntdll!ExecuteHandler2+0x26
01affbac 7763b40e ntdll!ExecuteHandler+0x24
01affc38 775f0133 ntdll!RtlDispatchException+0x127
01affc38 6f8c20ce ntdll!KiUserExceptionDispatcher+0xf

A shortcut to check for the presence of such a callstack is

0:001> !findstack kernel32!WerpReportFault
Thread 001, 2 frame(s) match
        * 04 01affa70 75b57f63 kernel32!WerpReportFaultInternal+0x186
        * 05 01affa84 75b57858 kernel32!WerpReportFault+0x70

So, creating a crash dump while the WER dialog is shown does not produce good results. The better way to create a dump is to proceed like this:

  1. Enter g in the debugger to let the application continue
  2. Press the “Debug” button of the WER dialog
  3. Confirm the warning about the debugger which is already attached
  4. Click “No” when asked to start debugging using the selected debugger

After that the second chance exception is thrown again and caught by the already attached debugger so that you can take a nice crash dump. In my example the dump now evaluates to

0:001> .exr -1
ExceptionAddress: 6f8c20ce (FirstSteps+0x000020ce)
   ExceptionCode: c000001d (Illegal instruction)
  ExceptionFlags: 00000000
NumberParameters: 0
0:001> k
ChildEBP RetAddr  
WARNING: Stack unwind information not available. Following frames may be wrong.
01afff88 75b3338a FirstSteps+0x20ce
01afff94 77619f72 kernel32!BaseThreadInitThunk+0xe
01afffd4 77619f45 ntdll!__RtlUserThreadStart+0x70
01afffec 00000000 ntdll!_RtlUserThreadStart+0x1b

which is exactly the expected exception for my demonstration application.

ConvertStore Errors

When converting a 2-tier symbol store to a 3-tier symbol store, you might get some error codes without error messages. Therefore it is hard to find out what the problem is and how to solve it.

The first error message you might see is probably
C\...>convertstore
Failed initial checks.

This happens if you run convertstore without command line parameters. The correct syntax is
C\...>convertstore -s d:\debug\symbols
Failed initial checks.

If you still get the same error, this could mean that the symbol store is already a 3-tier symbol store.

The error
C\...>convertstore -s d:\debug\symbols
Locking Symbol Store to prevent transactions while this operation completes.
Failed to lock Symbol Store. Error 0x00000003.

occurs if your symbol store does not have a pingme.txt file or no 000Admin subfolder. Just create an empty (0 byte) pingme.txt file and an empty 000Admin folder and you’ll probably get rid of this error.

During the conversion you may also get
C\...>convertstore -s d:\debug\symbols
Locking Symbol Store to prevent transactions while this operation completes.
Iterating store: d:\debug\symbols
ERROR: Failed to move d:\debug\symbols\kernel32.pdb -> d:\debug\symbols\ke\kernel32.pdb. Error 0x00000005.
ERROR: Failed to move d:\debug\symbols\ntdll.pdb -> d:\debug\symbols\nt\ntdll.pdb. Error 0x00000005.
Symbol Store no longer locked.

In this case, the file is in use by something else. Close programs that are using the files, delete index2.txt from your symbol store and run the command again.

If you get

C\...>convertstore -s d:\debug\symbols
Locking Symbol Store to prevent transactions while this operation completes.
Iterating store: d:\debug\symbols
ERROR: Failed to move d:\debug\symbols\advapi32.pdb -> d:\debug\symbols\ad\advapi32.pdb. Error 0x000000B7.

then the destination file already exists. This could happen if you had a 2-tier store with some files not moved, then downloaded files from the Internet and you’re now trying to move the 2-tier files over the downloaded 3-tier files. In this case you can just delete the affected 2-tier folder.

Mscordacwks collector

Did you ever grab a crash dump from a customer’s machine and you were not able to analyze it in WinDbg, because you didn’t have the matching mscordacwks.dll and SOS.dll? Until you recognize the mistake lies in the missing files, the customer has installed an update or reset the machine?

Never have this sort of trouble again with the Mscordacwks / SOS collector application. It scans the PC for those DLLs, copies them into a directory of your choice and renames them according their version number.

The program will look in

  • .NET framework 2.0 folder
  • .NET framework 4.0 folder
  • SXS folders

System requirements: .NET 2.0
License: free for personal and commercial use

Download Mscordacwks Collector 1.1.4.0 (.exe.zip) (67 kB)

If you encounter any bugs, you can report to mscordacwks.yyyy-mm-dd@lockerflockig.de (where yyyy-mm-dd is the current date).

Version history

1.0.0.0 (2014-02-12)

  • Preliminary release

1.0.1.0 (2014-03-08)

  • Release
  • Hourglass cursor during file search
  • Display intermediate search results

1.0.2.0 (2014-06-19)

  • Usability improvements
  • Preview of full output directory
  • Support for environment variables in path
  • Selection of .NET framework path

1.1.4.0 (2014-09-28)

  • Output structure customizable
  • Add Logo
  • Started writing unit tests and build script

Source code

SVN URL: http://opensource.svn.wellisolutions.de/MscordacwksCollector/
Username: guest
Password: debugging

If you want to contribute, you can request access rights by email to mscordacwks.yyyy-mm-dd@lockerflockig.de (where yyyy-mm-dd is the current date).

WinDbg Versions

WinDbg, the probably most powerful debugger for Windows is part of the Microsoft Platform SDK or Microsoft Windows SDK. Recently, the version numbering changed and maybe confused some developers.

When searching the Internet, I could not find a complete list of WinDbg versions including download links and release dates. The probably most official website listing version numbers is available in MSDN.  As WinDbg is part of Microsoft Windows SDKs, some filter on the Microsoft download center provide links to SDKs for download. Of course, Wikipedia was also helpful. So I assembled my own table and I want to share it with you:

Version Part of this SDK Release Date
WinDbg 4.0.18 (FTP Download) 2001-12-17
WinDbg 6.1.0017.1 (FTP Download) ~2002-12-14
WinDbg 6.1.0017.2 Microsoft Platform SDK February 2003
(Download information)
2003-03-26
WinDbg 6.2.0013.1 ? ?
WinDbg 6.3.0005.? Beta ? ?
WinDbg 6.3.0017.0 Microsoft Platform SDK August 2004
(Download information)
2004-08-21
WinDbg 6.4.0004.4 ? ?
WinDbg 6.4.0007.0
(Setup says 6.4.0007.2)
Windows Server 2003 SP1 Platform SDK
(Download) and also
Windows Server 2003 R2 Platform SDK
(Download)
2005-05-02
and
2006-03-14
WinDbg 6.5.0003.8 ? 2005-04-24
WinDbg 6.6.0003.5 ? 2006-01-18
WinDbg 6.6.0007.5 Microsoft Windows SDK
for Windows Vista and .NET Framework 3.0
(Web installer)
2006-10-30
WinDbg 6.7.0005.0 WinDbg version with .NET support built-in according to Volker von Einem 2007-04-??
WinDbg 6.7.0005.1 .NET support was removed again according to Volker von Einem 2007-06-20
WinDbg 6.8.0004.0 ? 2007-09-27
WinDbg 6.9.0003.113 ? 2008-03-20
WinDbg 6.10.0003.233 ? 2008-09-08
WinDbg 6.11.0001.402 ? 2009-01-30
WinDbg 6.11.0001.404 Microsoft Windows SDK
for Windows 7 and .NET Framework 3.5 SP 1
(Download)
2009-02-25
WinDbg 6.12.0002.633 Microsoft Windows SDK
for Windows 7 and .NET Framework 4
(Download)
2010-05-21
WinDbg 6.13.0001.776 mentioned in Mark Russinovich’s blog <2011-01-29
WinDbg 6.13.0008.1108  ? ?
WinDbg 6.13.0009.1140 used in the book “Practical Reverse Engineering: x86, x64, ARM, Windows Kernel, Reversing”, page 188 ?
WinDbg 6.1.7600.16385 (?) ? 2009-07-24
WinDbg 6.2.8229.0 Windows 8 ? <2012-04-04
WinDbg 6.2.8400.0 Windows 8 ? <2012-06-23
WinDbg 6.2.9200.16384 Microsoft Windows SDK
for Windows 8 and .NET Framework 4.5
(Download)Last version that runs on Windows XP
2012-11-15
WinDbg 6.3.9600.16384 Microsoft Windows SDK
for Windows 8.1
(Download)
2013-10-17
WinDbg 6.3.9600.17200 Updated version of
Microsoft Windows SDK
for Windows 8.1
2014-06-06
WinDbg 6.3.9600.17298 Updated version of
Microsoft Windows SDK
for Windows 8.1
2014-10-25
WinDbg 10.0.10069.9 Used by Andrew Richards
in Defrag Tools Episode #136
<2015-04-29
WinDbg 10.0.10075.9 Windows 10, direct download from CodeMachine 2015-04-29
WinDbg 10.0.10586.15 A version I got when I installed WinDbg via Chocolatey 2015-11-20 (?)
WinDbg 10.0.10586.567 Windows 10, build 1511 (TH2), direct download from CodeMachine 2015-10-30
WinDbg 10.0.14321.1024 Windows 10, build 1607, comes with a console version of GFlags 2016-07-29
WinDbg 10.0.15003.1001 Windows 10 SDK preview 14951 2017-01-04
WinDbg 10.0.15063.137 Windows 10 SDK Creators Update, build 1703 2017-03-30
WinDbg 10.0.15063.400 Windows 10 SDK 2017-05-09

To keep updates simple, any corrections can be sent to windbg.yyyy-mm-dd@lockerflockig.de, where yyyy-mm-dd is the current date (to bypass my email spam filter).

Most times, the installer will detect the bitness of the operating system and then install the files for that bitness. So if you like both versions of WinDbg (32 bit and 64 bit), you have to install twice on different operating systems. Interestingly, the 2003 version still installs on newer PCs (Intel Core i5) but the 2004 version doesn’t.

Note that the installer will often not install WinDbg directly but instead place another MSI installer into the Setup folder of the target directory. If you can’t find WinDbg.exe, you probably need to run another installer.

Once you went through all the installation processes, you can make WinDbg portable by simply copying the whole directory to your USB stick (or wherever you want to have it available).

WinDbg .cmdtree file format reverse engineered

The undocumented WinDbg command .cmdtree has become more and more popular. Various sources talk about it, e.g. Roberto Alexis Farah (rev. 2012-12-31), Tess Ferrandez (rev. 2012-12-31), John Robbins (rev 2013-01-02), Dmitri Vostokov (rev. 2012-12-31), Claudio Brotto (rev. 2012-12-31). Some of them have even been translated to Chinese (rev 2013-01-02), French (rev 2013-01-02), Czech (rev. 2013-01-02) or Brazil (rev. 2013-01-02).

Almost all of them have examples of an appropriate text file with useful commands for debugging. However, none of the posts I read until now (I tried the top 50 Google hits) was actually able to describe the file format in a comprehensive way. Volker von Einem (rev. 2013-01-02) even reported that he doesn’t know how to put commands into that file. In fact, I somehow destroyed the file myself several times, so it seems to be a little fragile, as stated by Asher (rev. 2013-01-02).

Some facts and usage

.cmdtree is supported since WinDbg 6.6 and is a WinDbg UI function, so you can’t use it from within a script file. To load a file during a debug session, simply type .cmdtree <filename> at the WinDbg command prompt.

The window layout will be saved as part of the workspace and open up next time, but it will not load the file contents. To load a specific command tree, you can specify the -c switch on WinDbg’s command line, e.g. windbg -c “.cmdtree d:\tree.txt”. Note that the window will open when attaching to a process, not immediately.

Unfortunately, using the -c switch as described will create additional windows every time. This was noticed by Volker von Einem already (rev. 2013-01-02, discussed in the comments). The workaround seems to be creating a workspace, docking the command tree window once, then closing it and saving the workspace. After this preparation, load the workspace with the -WF command line switch and execute the .cmdtree command with the -c switch at the same time. Make sure you never save that workspace with the window open – otherwise you’ll have it twice again.

Examples on the web

There are some examples out in the web, which can be found by searching for the magic term “windbg ANSI Command Tree 1.0”. Most of them are copies of the same sources as there are (hopefully giving correct credits):

People seem to adapt the files to their specific environment or for traveling to the customer as described by Romiko Derbynew (rev.2013-01-02). In most cases, it is not distinguished between the different types of debugging (i.e. kernel debugging vs. user mode debugging or post mortem debugging vs. live debugging). So, if you simply combine all the .cmdtree files to have all the tricks available, some of them seem not to work in your current debug session.

Basic information about the file format

Note: This is only a simplified description of the file format. Details follow in the next section.

The first line is windbg ANSI Command Tree 1.0.

The next line defines the title of the window. This is useful if you want to have several different command tree windows open. The title is introduced by the keyword title, followed by an opening sequence {“ and a closing sequence “}.

The header is separated from the content by the body keyword.

The content of the body is a tree structure. The first element specifies the caption, the second element defines the command to be executed when the item is double clicked. Elements are opened with the sequence {“ and ended with “}. Inside an element, you can use everything except “}. Quotes, braces, newlines and even the opening sequence and is allowed.

The tree structure is created by indentation, using spaces or tabs.

Details about the file structure

These details are reverse engineered from WinDbg version 6.12. It seems to me that earlier versions were more fragile than this one. Unfortunately, my old versions of WinDbg got lost and I can’t figure out more details for WinDbg 6.6 to 6.11.

All example files found in the web start with the term windbg ANSI Command Tree 1.0. However, if you look at the binary executable content (using the SysInternals Strings Utility (rev. 2013-01-02) here), you’ll also find windbg ANSI Command Tree 1.1.
C:\>strings -n 25 windbg.exe | find "windbg ANSI"
windbg ANSI Command Tree 1.1
windbg ANSI Command Tree 1.0

WinDbg does not load files with versions other than 1.0 or 1.1. The most noticeable difference between those versions is the order of the tree. Version 1.0 displays the tree as defined in the file, version 1.1 reverses the order.

As the first line implies, WinDbg does not load UTF-8 files with BOM at all and does not display special characters in UTF-8 files without BOM correctly.

Directly after those findings there’s the title and body keyword, but nothing else. This probably indicates that there aren’t many more secrets to be discovered.

Nevertheless, I wanted to figure out how fragile the file format actually is and I tried adding various things to the file. In the following, I call windbg ANSI Command Tree 1.0 the ANSI header{“xxx”} an element and from two subsequent elements, the first one is the caption and the second one the command.

What you can do:

  • There can be tabs and spaces at the end of the ANSI header (before the newline).
  • Unlike many other text editors, WinDbg handles newlines in all three ways (CR, LF, CR+LF)
  • There can be multiple tabs or space before and after the title keyword.
  • There can be multiple tabs or spaces after the title definition.
  • Newlines can be used in the title definition, but the title of the window may not be layouted correctly.
  • There can be multiple tabs or spaces before the body keyword.
  • There can be one (!) tab or space after the body keyword (before the newline)
  • There can be multiple tabs or spaces between a caption and a command.
  • Quotes, braces, newlines and even the opening sequence is allowed within a element without using any escape sequence.
  • Newlines can be used in captions, but they will have no effect.
  • Empty lines after the ANSI header are ignored.
  • Indentation can be followed by a newline. However, I encourage you not to do so, because it confuses any editors of your file. See also the “weird” section below.
  • Folders can have commands as well. However, I don’t recommend using it, because it confuses users of the file.
  • Leaves do not need commands. However, I recommend putting commands there.

What you can’t do:

  • Any line (empty or not) before the ANSI header will cause a failure.
  • If there’s a non-empty line between the ANSI header and title, the file will not be loaded.
  • WinDbg does not load the file, if there’s no element after title.
  • The file will also not be loaded if there’s no space or tab between title and the title element.
  • Any non-empty line between title and body will cause loading the file to fail.
  • There must be only zero or one space or tab after the body statement before the newline.
  • The file cannot be loaded if there is indentation on the first line of the content.
  • If there’s too much indentation on a line, the file will also not load.
  • Duplicate end sequences prevent the file from being loaded.
  • Missing end sequences also prevent the file from loading.
  • Non-empty lines which are not elements are not accepted by WinDbg.
  • There must be at least one element in the content.

Weird things:

  • While indentation can be on a separate line, only the first line counts. Additional lines before the caption are ignored. Indent characters on the same line as the caption are ignored as well then.
  • There can be multiple tabs or spaces after a command, except for the last line.
  • If there are more than two elements on one line, the odd ones are captions and the even ones are commands.
  • HTML entities &quot; &amp; &lt; and &gt; can be used, but not &apos;. If &apos; is used, none of the other HTML entities in the same command works any more. HTML entities only work in commands, not in captions. At least you can escape “} by &quot;}.
    This seems less a command tree feature but the WinDbg DML feature. If you try a .echo &quot;&quot; at the command prompt, WinDbg itself will also unescape the entity – and also not handle &apos; (as described in the DML documentation).
    Unfortunately it seems DML cannot be used any further.

Test files for download

I figured this out mostly by trial and error. If I were more l33t, I’d probably do a .dbgdbg and debug WinDbg itself to know how it operates with command tree files.

Nevertheless, I provide the 21 good cases (files which can be loaded) and 24 bad cases (files which cannot be loaded) as a reference for download.

Your comments

Did you figure out any other specialty of the .cmdtree format? Are any conclusions I made wrong? Do you have a file which does not load which I could analyze? Do you have older versions of WinDbg available I can use for testing? Any comments are welcome.