How does NetTcpBinding(read WindowsStreamSecurityBindingElement) encrypt/sign messages?

The short answer is that, yes, with NTLM authentication the messages will still get signed and encrypted if you have set the Transport security ProtectionLevel to EncryptAndSign (the default).

Here's an outline of how it works:

• selecting Transport security configures a WindowsStreamSecurityBindingElement in the channel stack. This inserts a stream upgrade provider (see below)
• in the NetTcpBinding, message exchange between the client and service happens within the .NET Message Framing Protocol, which provides both message framing and a mechanism for client and service to negotiate stream upgrades, the principal use of which is to establish transport security. If there is a stream upgrade provider configured in the channel stack, this will be invoked during the Preamble stage of the Framing Protocol when the client opens the channel.
• the upgrade provider for WindowsStreamSecurityBindingElement invokes an SSPI handshake between the client and the server using the SPNEGO security package: in the NetTcpBinding this will normally result in Kerberos being selected as the underlying security provider if available, but will choose NTLM if not.
• if NTLM is the resulting authentication provider, the SSPI handshake will involve the three-leg NTLM challenge-response exchange of tokens described in the NTLM specification. This protocol includes a mechanism for exchanging keys for message signing and encryption. Once the SSPI handshake has generated an appropriate security context, thereafter all messages exchanged are signed and encrypted in the sending channel stack's stream upgrade provider, and decrypted and verified in the receiving channel stack's stream upgrade provider, in each case by using calls to the NTLM security provider via the abstracted SSPI message support functions.

This is a Microsoft propriety implementation and not properly documented and perhaps on purpose to prevent intruders to take advantage of it.

As far as I know, this usually happens at the TCP level with a special token is generated by the user's credentials and passed along with the request. This is intercepted by windows security channel and authenticated against the AD.

This token is used as a key (or as a basis for generating the key) for encrypting the communication.

I think if you look at the TCP packet, you must be able to see the token - although I have never seen it.

If you are doing this all in code then you can find out the options here (search for 'NetTcpBinding'). Transport security is via Windows builtin TLS.

The diagram here should be helpful for your scenario.