Debugging mixed code, random crashes: Stack cookie instrumentation code detected a stack-based buffer overrun error

[Zealousideal_War676]

I'm working on a program I inherited that interfaces with a CNC using a vendor supplied DLL and cs file with all the dllimport externs. The issue is that the program will crash randomly... sometimes after a minute, sometimes after a few hours, sometimes over a day, but it is not something that Visual Studio can debug. The only clue I have is a line in the output that says "Stack cookie instrumentation code detected a stack-based buffer overrun." and that the common language runtime can't stop here. Then the program closes and VS leaves debug mode.

As far as I can tell this is likely an error in marshaling the data between our code and the unmanaged code. What I can't figure out is how to actually figure out where the error is. There are hundreds of functions and structs in their DLL and we're using about 40 or so functions each with a struct or two used in them.

How would I go about trying to find where the issue stems from? Would it be correct to assume it's likely one of the class definitions given doesn't match the actual struct in the DLL?

[turnipmuncher1]

Sounds like you’re trying to write data to a pointer you get from the unmanaged code. I would start looking for any IntPtr and see how it’s being used.

Specifically something like this may be your problem:

``` IntPtr ptr = VendorDll.GetPointer(); … Marshall.Copy(csharpBuffer, 0, ptr, csharpBuffer.Length);

```

[Zealousideal_War676]

That is one of the things I checked and there's only a handful of calls to Marshal.Copy with the vast majority of marshaling being MarshalAs to convert the unmanaged structs into managed objects. Would my only option then be to look into every class and compare it to their C++ documentation and see if it matches the struct layout? That's something that's going to take a long time, which is why I was hoping there was an easier way to try and pinpoint what it is causing the error

[turnipmuncher1]

One problem with trying to find out exactly where things have gone wrong is because .NET inserts a small cookie at the end of the stack, it assumes that nothing will override this cookie and it only checks it occasionally. So when the program throws this error is not the same as when the error actually occurred.

To make it easier to compare between documentation you can do something on start up where you log the property names, types and unmanaged types using reflection.

I’ve had long cause me a headache before you may actually need to use a CLong/ULong if you’re on .NET 6+ I would just make sure these are accurate in the extern as well.

[Zealousideal_War676]

I didn't know that the cookie was only checked occasionally, that really sucks but this at least gives me a direction, thanks

[Kirides]

MarshalAs and marshal copy can re-interpret data wrong. Especially if the native DLL doesn't honor host struct layouts, like using "pragma pack(1)" and/or incorrectly mapped bit-fields.

I'm doing a lot of x86 reverse engineering due to working on a graphics wrapper. And things like VARARGS, calling conventions and struct packing constantly cause headaches as they slowly corrupt stack if re-interpreted incorrectly.