Finding the /NXCOMPAT bit

I’ve recently been reading about the Visual C++ /NXCOMPAT linker option, which indicates to the loader that the image in question is compatible with the no-execute feature supported by most processors these days, and exposed by the operating system.

http://msdn2.microsoft.com/en-us/library/ms235442(VS.80).aspx

http://developer.amd.com/articlex.jsp?id=143

http://en.wikipedia.org/wiki/NX_bit

In summary, NX is an important security feature which allows memory pages to be marked as non-executable, and for that protection to be enforced via hardware. That way, for example, if a compromised process attempts to execute instructions written to the stack following a buffer overflow, an exception will be raised instead.

There are still two issues to be aware of, though. First, the exception handler may have been compromised. In that case, following the raise, control will be returned to hacker code anyway. The second issue is that a memory page can be marked executable programmatically, although that would generally be a much tougher attack to mount remotely.

The first thing I was wondering is how to tell if a PE binary has the NXCOMPAT bit set. I found the answer in the 3rd post of this thread – http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=163194&SiteID=1. To confirm, I built a simple x86 test program, first without the /NXCOMPT linker option, and did a dumpbin. Highlights:

>link.exe /dump /headers NxOn\Test.exe

OPTIONAL HEADER VALUES

0 DLL characteristics

Next, I built the program with the /NXCOMPAT linker option and did a dumpbin:

>link.exe /dump /headers NxOn\Test.exe

OPTIONAL HEADER VALUES

100 DLL characteristics

NX compatible

To fully satisfy my curiousity, I next tried the same thing on two x64 builds of the same test code. Got the same result; the 0×100 bit is set in the DLL characteristics field of the PE optional header values only if linked with /NXCOMPAT. Interestingly, in both x64 binaries, and unlike the x86 ones, the 0×8000 bit ("Terminal Server Aware"; http://msdn2.microsoft.com/en-us/library/01cfys9z(VS.80).aspx) is also set.

Leave a Reply