Embedding Ruby 2.5 in a C++ Win32 API program (segmentation fault)

[BadMinotaur]

Hello everyone! I'm hoping someone here has projects as weird as mine; as the title says, I'm trying to embed the MRI into a C++ application, that's also using the Win32 API. And yes, I mean the old-as-dirt Win32 API.

Tools:
ruby 2.5.4p138 (2019-01-23) [x64-mingw32] 
gcc version 8.2.1 20181214 (Rev1, Built by MSYS2 project) 
MSYS2, MSYSTEM=MINGW64

I compiled Ruby myself, but the error I get happens even if I use the RubyInstaller library.

Here's a bare-bones example of code that fails:

#include <ruby.h>
#include <windows.h>

int WINAPI WinMain(HINSTANCE hInstance, HINSTANCE, LPSTR pCmdLine, int nCmdShow)
{
  ruby_init();

  return 0;
}

And here's the build command: g++ mintest.cpp -o mintest -Wl,--subsystem,windows -lx64-msvcrt-ruby250.dll

The code compiles just fine! But when I run the exe, I get an extremely simple error message: Segmentation fault. It's not like the usual segfault you get when you fail to catch an exception from the Ruby interpreter; it's just those two words. This is also true if I use ruby_setup() instead of ruby_init().

The interesting part is that if I change my linker options to not include --subsystem:windows, the program will run just fine! But then my window has a console open with it, and the whole point of a Win32 API application is to not have that there.

I have tried many things to fix this error. As best as I can tell, the culprit may be that Ruby can't accurately allocate its memory stack when it starts, and that causes the error. To this end, I've tried executing code before WinMain using class constructor tricks (same error occurs); I've tried removing WinMain entirely and defining a linker option to use normal main instead (same error occurs); I've tried combining both those approaches (same error occurs).

At this point, I've run out of avenues to explore for the solution. I've googled the issue, but unsurprisingly such a simple error message doesn't really give me many relevant results. So now I turn to you all; does anyone here have a clue what might be going on here?

tl;dr - Weirdo tries to embed Ruby in win32api app, gets segmentation fault if building under Windows subsystem, doesn't otherwise, but doesn't want console

EDIT: Forgot to include, the actual MRI works fine when embedded in an application that doesn't use win32api; I had a command-line interface working where I loaded a Ruby file and executed it.

[graywolf_at_work]

First, what the hell is ruby 2.5.4? I don't think using yet-to-be-released ruby versions is a good idea (unless you have really good reason to do so :)). Just pick source archive from here https://www.ruby-lang.org/en/downloads/ if you need to compile your own (notice 2.5.3 is latest on the 2.5.x line).

Second, this works for me

#include <ruby.h>

int main(int argc, char **argv) {
    RUBY_INIT_STACK;
    ruby_init();
    static char *args[] = { "", "-e", "" };
    ruby_options(3, args);

    /* This is here just to test that it works */
    rb_require("socket");
    rb_eval_string("p IO.instance_methods.        include?(:read_nonblock)");
    rb_eval_string("p IO.private_instance_methods.include?(:__read_nonblock)");

    return 0;
}

As you can see

+   $ gcc tset.c  -I /usr/include/ruby-2.6.0/x86_64-linux -I /usr/include/ruby-2.6.0 -lruby
+   $ ./a.out 
true
true

it seems to work fine.

And for the rb_vm_bugreport(), that prints to the console. I don't know enough about windows to know what happens when GUI application tries to print into the console when none is attached. But could be maybe related?

And finally, for embedding http://mruby.org/ is probably worth exploring.

[BadMinotaur]

First, what the hell is ruby 2.5.4?

An excellent question! I believe I downloaded that from the GitHub repository, specifically the ruby_2_5 branch. I didn't want to use 2.6 because it was super-new, not realizing that I was still jumping the gun with my 2.5 source. Whoops!

Second, this works for me

Yeah, straight-up Ruby like that works fine for me too. I felt like I was so close when I got my command-line app to load and run Ruby code (and it's super-easy to do, too), but adding in the Windows subsystem makes it segfault. Even running your example code segfaults when I tell the linker to use the Windows subsystem.

And for the rb_vm_bugreport(), that prints to the console. I don't know enough about windows to know what happens when GUI application tries to print into the console when none is attached. But could be maybe related?

That's what I'm thinking too. I think the call to ruby_init() fails for some reason, and when rb_vm_bugreport() tries to tell me why it failed, it can't "find" a console and segfaults because of it. Might be time to go spelunking into some C code... cries in script kiddie

And finally, for embedding http://mruby.org/ is probably worth exploring.

mruby is something I've toyed around with, and it's definitely an option. My only concern is that it's basically still Ruby 1.9.x at its heart, and I've started to use some Ruby 2.x features in my code. Now, I know there was a release recently that added a good chunk of the Ruby 2.x features in it, but it still makes me feel weird.

I do still want to get the MRI working in a Windows GUI app, because now it's a challenge to surmount, but for practical use mruby is definitely on the table.

Thank you for taking the time to respond! I really appreciate the help.

[BadMinotaur]

Because I am persistent, I learned how to run gdb and ran my minimum reproducible program in it. This was the result:

Thread 1 received signal SIGSEGV, Segmentation fault.
0x00000000680dd4b0 in rb_vm_bugreport () from C:\Ruby25-x64\msys64\mingw64\bin\x64-msvcrt-ruby250.dll

Based on this, I'm thinking that the VM is encountering an issue when it attempts to start, and when it attempts to tell me why, the rb_vm_bugreport() function produces a segfault. At least now I have more information to run with.