First, the version of the IPsecPing client code available above makes a call to WSASetSocketPeerTargetName with a Service Principal Name (SPN) based on the DNS name (specified via the command-line) of the target peer. Using an SPN forces the identity of the server to be verified by the underlying authentication protocol, e.g. Kerberos. However, between peers that lack trust (i.e. non-domain joined; or joined to domains that don’t share trust), Kerberos isn’t available. This was the case in my TechEd demo environment. Anticipating this, I configured shared-password-based IPsec policies. But I’d forgotten that the code was written to require checking the SPN.
Note – I’m told that certificate-based IPsec authentication also supports verification of an SPN (presumably via something resembling schannel name mapping), and that in that situation a Kerberos environment isn’t required. I haven’t confirmed that.
Anyway, to allow IPsecPing to work in my demo environment, I modified the code to only conditionally call WSASetSocketPeerTargetName (based on the presence of a command-line argument). Now I’m able to confirm IPsec-protected IPv4 connectivity between the un-trusted peers.
But IPsec-protected IPv6 connectivity wasn’t working! Why? Well, this one was easy – I was using an IPsec policy that only specified v4 addresses. Actually, I’d followed the directions on p. 1144 of the Vista Resource Kit book (http://download.microsoft.com/download/f/c/7/fc7d048b-b7a5-4add-be2c-baaee38091e3/9780735622838_WindowsVistaRK_ch27.pdf) and created a Connection Security rule via Windows Firewall. However, when creating an address-specific rule that way, it appears that the only way to provide an v6-compatible policy is to type in the v6 addresses. I’m just too lazy for that 🙂
So I disabled my WF Connection Security rule and instead created a new IPsec policy via the IP Security Policies snap-in. That approach is arguably more time-consuming than the above, but it allows policies to be specified via DNS name (which – note – can probably be spoofed). The resulting filter ruleset seems to apply to both v4 and v6 traffic. Running IPsecPing with the –6 switch confirmed this (I used "stcpserver.exe -v6", from the Vista SDK sample code, on the test peer in this case; note that the latter listens on port 27015).