It gets worse. Running the 64-bit custom action code as an interactive user worked fine. It was only in the context of MSI that was failing. Initially, I wasn’t even sure it was the custom action causing it – it could have been any other piece of the MSI (for example, the install payload includes an NT service that could have been tanking).
However, before I learned about that useful logging information, I was trying to figure out how to get the custom action to run in the debugger. I first tried setting up msiexec.exe to always launch in ntsd.exe via Image File Execution Options in the registry. No luck. The first instance would start fine and kick off the install sequence. But the second instance – the one that hosts the custom action – was failing the debugger attach with Access Denied. Painful!
So then I learned about the log file stuff which confirmed the custom action as the 100% culprit. That inspired me to focus on getting this thing in a debugger, one way or another. I knew MSI was running my custom action as Local System, and the Access Denied made me suspect a privilege issue (even though I was running all of these tests as local admin). Thus, my next approach was to configure my x64 (Vista) test machine with a kernel debugger.
I first tried setting a global breakpoint in the kd on nt!NtCreateProcess, but that didn’t work; it never fired. I may not have been in the correct context for that to work. So then I thought – what the heck? – add a hard DebugBreak to the custom action DLL entry point. It’ll either take out the host process or cause it to break into my kd. Fortunately it did the latter!
Anyway, once you’ve got the thing running in a proper debugging environment, you know it’s just a matter of time before the bug is squashed. Turns out the line of code that sets a Group name (via the resource-based “indirect string” syntax) for the new firewall rule was throwing a COM exception. Admittedly, I still don’t know exactly why that only fails in the 64-bit system scenario; I still need to try using a full path to the binary with the resource for the group name. For the purpose of this sample, it was fine to just leave that property blank – problem solved.
kd> !process -1
User: S-1-5-18 (Well Known Group: NT AUTHORITY\SYSTEM)
00 S-1-16-16384 (no name mapped)
Doh! That last SID represents the System Integrity Level (per SECURITY_MANDATORY_SYSTEM_RID in winnt.h). Now, Image File Execution Options will cause a debugger to be launched at system, but apparently not with sufficient juice to debug a high-integrity process. Anyone confirm this?