Fun High-Integrity 64-bit Debugging

I recently had the privilege of debugging something pretty painful.  The project is a WiX (http://wix.sourceforge.net) sample which includes a custom action for opening a firewall port upon installation (and closing it upon uninstall).  The crazy thing was that the install was succeeding on x86 and failing on x64.

It gets worse.  Running the 64-bit custom action code as an interactive user worked fine.  It was only in the context of MSI that was failing.  Initially, I wasn’t even sure it was the custom action causing it – it could have been any other piece of the MSI (for example, the install payload includes an NT service that could have been tanking). 

Digging into the MSI log file proved the custom action to be the culprit, though.  I learned that that information can be obtained by running the install from the command-line like this:  “msiexec.exe /i mypackage.msi /l*v verbose.log”.  The search the log for “Return value 3”, which is how MSI tells you that your custom action failed!

However, before I learned about that useful logging information, I was trying to figure out how to get the custom action to run in the debugger.  I first tried setting up msiexec.exe to always launch in ntsd.exe via Image File Execution Options in the registry.  No luck.  The first instance would start fine and kick off the install sequence.  But the second instance – the one that hosts the custom action – was failing the debugger attach with Access Denied.  Painful!

So then I learned about the log file stuff which confirmed the custom action as the 100% culprit.  That inspired me to focus on getting this thing in a debugger, one way or another.  I knew MSI was running my custom action as Local System, and the Access Denied made me suspect a privilege issue (even though I was running all of these tests as local admin).  Thus, my next approach was to configure my x64 (Vista) test machine with a kernel debugger. 

 I first tried setting a global breakpoint in the kd on nt!NtCreateProcess, but that didn’t work; it never fired.  I may not have been in the correct context for that to work.  So then I thought – what the heck? – add a hard DebugBreak to the custom action DLL entry point.  It’ll either take out the host process or cause it to break into my kd.  Fortunately it did the latter!

Anyway, once you’ve got the thing running in a proper debugging environment, you know it’s just a matter of time before the bug is squashed.  Turns out the line of code that sets a Group name (via the resource-based “indirect string” syntax) for the new firewall rule was throwing a COM exception.  Admittedly, I still don’t know exactly why that only fails in the 64-bit system scenario; I still need to try using a full path to the binary with the resource for the group name.  For the purpose of this sample, it was fine to just leave that property blank – problem solved.

So why was the user mode debugger attach failing?  Here’s my theory – let’s see what the kd can tell us about that when my DebugBreak fires:

kd> !process -1

PROCESS fffffa8002245860
SessionId: 1  Cid: 0a80    Peb: 7fffffd3000  ParentCid: 09e8
DirBase: 39588000  ObjectTable: fffff8800280cfa0  HandleCount: 106.
Image: msiexec.exe
… 
      THREAD fffffa8002e91bb0  Cid 0a80.0bc4  Teb: 000007fffffd6000 Win32Thread: fffff900c07d2ad0 RUNNING on processor 0
kd> !token -n
…

User: S-1-5-18 (Well Known Group: NT AUTHORITY\SYSTEM)

Groups:

00 S-1-16-16384 (no name mapped)

Doh!  That last SID represents the System Integrity Level (per SECURITY_MANDATORY_SYSTEM_RID in winnt.h).  Now, Image File Execution Options will cause a debugger to be launched at system, but apparently not with sufficient juice to debug a high-integrity process.  Anyone confirm this?

 

No Comments »

Leave a Reply