Dan Griffin's Blog
Comments on security, PKI, smart cards, cryptography, and entrepreneurship.
I wasn’t able to find documentation describing how to handle this exact scenario, so, for posterity, I’ll share what I learned. The problem: a native code dll export routine takes a pointer to a structure as its sole input. Suppose that the structure has only two fields. The first field is a string buffer allocated by the caller into which the callee will copy data. The second field is the size of the buffer, as allocated by the caller. We want to call this code from C#.
/* Native code */
typedef struct _PARAM
{
__inout LPWSTR wszOutput;
__in DWORD cchMaxOutput;
} PARAM, *PPARAM;
__declspec(dllexport)
long
WINAPI
ExportFunctionA(__inout PPARAM pParam);
The challenge: declare the managed version of the parameter and function in such a way that they will be correctly marshaled. Here’s the equivalent of what I did (note that I’m pasting and modifying snippets of code without confirming that they still build).
/* Managed code */
[StructLayout(LayoutKind.Sequential)]
class TestDllParam
{
[MarshalAs(UnmanagedType.LPWStr, SizeParamIndex=1)] public String output;
public UInt32 outputMaxLength;
}
class TestDll
{
[DllImport("testdll.dll", CharSet=CharSet.Unicode)]
public static extern Int32 ExportFunctionA(
[MarshalAs(UnmanagedType.LPStruct)] [In, Out] TestDllParam testDllParam);
}
static void Main(string[] args)
{
Int32 result = 0;
TestDll testDll = new TestDll();
TestDllParam testDllParam = new TestDllParam();
testDllParam.output = new String(’\0′, (int)bufferSize);
testDllParam.outputMaxLength = (UInt32)testDllParam.output.Length;
result = TestDll.ExportFunctionA(testDllParam);
…
}
That seems to work, at least in the context of the project from which I adapted it. As a bonus, here are some things that didn’t work.
- Omitting the [MarshalAs] attribute from class TestDllParam. When I did that, I only got one output character copied into my buffer, since the marshaler doesn’t know how long the native LPWSTR in the C-struct is without a length hint.
- Omitting the CharSet= attribute from the managed declaration of ExportFunctionA. When I did that, my native code overflowed the buffer provided by the marshaler, which caused an interesting runtime exception. I didn’t actually check how long the native input buffer was in the broken version, but the fix seemed to confirm the educated guess that I was being provided with a buffer large enough to fit outputMaxLength Ansi characters, rather than that many Unicode characters.
- Omitting the [In, Out] attribute from the managed declaration of ExportFunctionA’s testDllParam arg. When I did that, changes made by the callee to the input structure would not be reflected after the function returned. Incidentally, I confirmed that via a simpler example: add a UInt32 to the structure and have the callee modify it. Again, changes made by the callee are not seen by the caller.
- Adding a ref modifier to managed ExportFunctionA’s testDllParam arg. I’m not yet sure why this happens, but I get a runtime exception. Based on what I’m seeing in memory, my guess is that the parameter structure, or at least some subset of it, can only be marshaled correctly by value, rather than by reference, given its current declaration.
No Comments »
No comments yet.
RSS feed for comments on this post. TrackBack URL
