Getting the [DllImport] declaration wrong is a standard cause. The CallingConvention property is very important. Getting it wrong causes the stack to get imbalanced, ultimately causing SO if you call it often enough. There's an MDA for that, make sure you didn't turn off PinvokeStackImbalance. Using Debug + Windows + Registers is another way to diagnose it, the ESP register value must be the same before and after the call.
CallingConvention.Cdecl is often required for C or C++ code unless that code was explicitly written with the __stdcall keyword.
Okay, the Embarcadero link suggests another reason for the exception. Borland libraries traditionally enabled FPU exceptions. That's grossly incompatible with .NET code. Especially WPF because it heavily uses doubles for control sizes and positions. An FPU stack overflow is a bit odd, you more typically have trouble with NaN values.
If you don't have the source code of the library then you don't have many attractive options to fix the problem. One thing you can try is to throw and catch an exception after the first call to the library. The .NET exception handling plumbing resets the FPU control word. Like this:
bool ok = MSR_InitComm("COM1", 9600);
try {
throw new Exception("Fpu reset intended");
}
catch (Exception) {
}
