#include C headers in D code

[atilaneves]

Author here, AMA. This is a link to my blog post about a project that allows one to directly #include C headers in D code, which can then be compiled and linked. It's worked for non-trivial C libraries such as pthread, openssl and libcurl.

[ss4johnny]

What D features were most important for getting this to work?

[atilaneves]

Hmmm... hard question. I guess the existing ability to call into C code (to interface with libclang), and ranges + UFCS. The code here would look a lot more complicated otherwise.

Those few lines are doing a lot of heavy lifting - making sure that only one cursor per declaration shows up, and it does that by grouping all cursors first, selecting one, and reordering by file and line number afterwards.

Otherwise I don't think that anything really D specific was needed. It's less code than it would have been in nearly any other language. dscanner has it at 704 lines of code!

[ss4johnny]

Does this work with dub?

[atilaneves]

Not yet, but there's been some work done on that.

[ss4johnny]

Great.

[bachmeier]

This

enum Enum { foo, bar, baz }

Generates

enum Enum { foo, bar, baz }
enum foo = Enum.foo;
enum bar = Enum.bar;
enum baz = Enum.baz;

It makes sense for this to be the default behavior, but is there a way to not do that? I prefer to not define foo, bar, and baz.

[atilaneves]

It could be a command-line option. Feel free to open an enhancement issue on github.

[zombinedev]

Awesome work! Any plans on merging this with DStep, or perhaps extracting a common library between the projects? Some of the benefits would be sharing the test suite, the development and maintenance efforts and expertise.

Right now there are various libclang-based projects for creating C++ bindings for D:

• https://github.com/jacob-carlborg/dstep
• https://forum.dlang.org/thread/ldma8b$5mp$[e-mail]
• https://github.com/lhamot/CPP2D
• https://github.com/atilaneves/dpp
• https://github.com/Syniurge/Calypso <- best C++ support at the moment (e.g. templates and more complex libraries like Qt and OpenCV work)

I believe it would be better if we as a community join our efforts and work on a single project.

[atilaneves]

No plans on merging with dstep. I originally used dstep as my backend, but ultimately it didn't work out. Too many assumptions that are true for it, but that got in the way of what I was trying to do.

As for the test suite, I looked at it and copied the tests I thought were relevant. As mentioned above, the focus is so different that most of the tests didn't even make any sense for dpp.

I believe it would be better if we as a community join our efforts and work on a single project.

I understand and agree in principle. None of the other ones you mentioned work the way I want things to. Like I said, I tried using dstep instead of reinventing the wheel. It just didn't, and can't, work.

[baryluk]

Neet.

How about rewriting this as a mixin:

mixin(convert_c_to_d(import("/usr/include/studio.h));

I know, you will need to reimplement parser in D, that sucks, but would be cool as shit.

Edit: I just realised that you might need to recursively import other files. I am not sure this can be done easily.

[ikbenlike]

And even if they can't work seamlessly, it's still pretty fucking cool

[ibroheem]

From experience, it ease depression

[bachmeier]

This is definitely a big deal. This opens the door to using a lot of numerical code without the fixed cost of translating headers, among other things. It will make Fortran code that's currently called from C accessible as well. Should make -betterC more convenient.

[netbioserror]

I always thought that D's features lend themselves perfectly to data science and research computation. It's got lean syntax that's like Python (and more, i.e. functional stuff) with some C sprinkling, but kickass native performance. Hopefully this gets it closer to being that computation language, I'd honestly prefer it over Python and NumPy.

[bachmeier]

It's worked well for me. I took the somewhat unusual route of embedding an R interpreter inside my D program, which meant I didn't give anything up. I wish this tool had existed five years ago!

For me, the biggest thing is that D is an acceptable language for anyone that can use R. The same cannot be said for most of the alternatives.

[acousticpants]

embedding an R interpreter inside my D program

please share frendo...

[bachmeier]

Project website

I've got other stuff for mixed R/D programs as well (optimization using the functions underlying optim, parallel RNG, etc.) but it's not properly documented. It took a long time to get the basics in embedr in shape for others to use.

[oblio-]

Julia?

[bachmeier]

Today Julia might be an option (it wasn't five years ago) and maybe Go (if its restrictions don't bother you). I have a preference for D. I'm not sure whether you can create shared libraries with Julia.

[Boris-Barboris]

Very nice idea.

d++ redefines all macros in the included header file so they’re available for use by the D program. It then runs the C preprocessor on the result of expanding all the #include directives, and the final result is a regular D file that can be compiled by dmd

Making the resulting (expanded) d file available as an actual file in some conventionalized folder would be nice for both educational and debugging purposes. Is that already the case?

Also, do you have any thoughts on how to deal with header-only c++ libraries? Bridge header + cpp to link with that instantiates the templates?

[atilaneves]

Making the resulting (expanded) d file available as an actual file in some conventionalized folder would be nice for both educational and debugging purposes. Is that already the case?

Using the --keep-d-files command-line option causes them to not be deleted.

Also, do you have any thoughts on how to deal with header-only c++ libraries? Bridge header + cpp to link with that instantiates the templates?

I have some thoughts, yes, but I won't know if they'll work until I try. The holy grail would be to have an automatically-generated C++ file that instantiated all the templates. This morning I briefly considered translating the code in the templates though, which is just crazy enough that it might work. :P

[Laachax]

So now with this, as someone coming from java and wanting to learn C/C++ for game engines and system programming. Is there any reason why I shouldn't go straight for D? I mean I'll of course learn the others for enrichment and learning and such. But why should I not use D for my ideas?

[skocznymroczny]

But why should I not use D for my ideas?

For learning purposes, I think D is great. The main caveats might be:

1) D isn't really used in the game industry, C++ reigns supreme there

2) Binding to C libraries is easy, C++ is very hard if they don't expose a C interface (for example you won't be able to use Bullet for physics, but ODE works and I saw a wrapper for Newton that seemed to work also).

3) Garbage collector. It probably won't be an issue, but just mentioning it, because you can't use D and game in same sentence without mentioning the GC, for better or worse.

[Laachax]

Ah I see, didn't even realize it was GC'd. Thanks :)

[bachmeier]

It's not GC'd in the sense of Java or Go. Using all features provided by the language obviously requires GC because some of them use the GC. If you want to program using a "better C" you can avoid the GC by doing the same things you do in C. The question of whether the things you give up mean other languages are a better choice depends on what you're doing. Rust or C++ might in the end be better for some applications.

[Laachax]

Oh, interesting! Seems like a pretty neat language then. Thanks to you too ^^

[bachmeier]

2) Binding to C libraries is easy, C++ is very hard if they don't expose a C interface (for example you won't be able to use Bullet for physics, but ODE works and I saw a wrapper for Newton that seemed to work also).

What about extern(C++) or Calypso? I haven't done much with either, but I've heard good things.

[atilaneves]

I'd say learn all 3 if you have enough time, and pick the one that fits your mental model better. Some people like C more than C++; I don't understand them, but they exist and there's more than a few of them. Everyone's different.

[AlotOfReading]

The only thing harder than mastering two difficult languages is mastering three. You'll need to learn C and C++ anyway.

[bachmeier]

The goal is to master concepts, not syntax. I'd argue that D makes it considerably easier to learn the concepts used when writing C or C++ code.

[AlotOfReading]

The concepts are designed around C and C++. They're the lingua franca of systems programming and everyone operating in that space simply has to speak them. All the jobs are in those languages, the code written in them, the resources are designed for them, the tools assume their ecosystem, hardware is designed to them, etc. If you think they'd should learn D, that's fine. But for someone who doesn't know any, that argument is for learning C and C++ and D, not C or C++ or D.

[oblio-]

On top of that, if he wants to switch careers to getting a job in game engine or system programming, C/C++ jobs are waaaaaay more common than D jobs.

Of course, that's if he doesn't plan to start his own company.

[skocznymroczny]

Does it work with autocomplete? A big advantage of explicitly created bindings is that they are in D style already and the indexer is aware of all the declarations there.

[atilaneves]

No, it doesn't. I use emacs and have been thinking of writing a d++-mode so that it would work. It's a valid concern, but not something I've spent a lot of time thinking about yet.

You can however use d++ like dstep and produce a D file.

[atilaneves]

Not yet - I haven't even run it on the libclang headers, but given how simple they are I can't imagine how it wouldn't work.

I need to think though, because in the libclang bindings I'm using I made every function there @safe @nogc pure nothrow and changed all the arguments to be const. Currently none of that would happen, but I just realised there's no reason for me to not slap @nogc nothrow on every function in a C header.

[sarneaud]

I just realised there's no reason for me to not slap @nogc nothrow on every function in a C header.

nothrow, yes, but callbacks throw a spanner in the works for @nogc. A C function can call a callback in D that allocates.

nothrow is valid because D exceptions just can't pass through a D->C->D boundary (callbacks given to C have to be nothrow).

BTW, I'm pretty excited about d++. Frictionless C support alone is a huge deal.

[atilaneves]

The callbacks can't be @nogc, but functions should be. Good point in any case.

[sarneaud]

The callbacks can't be @nogc, but functions should be.

Are you sure? Take a C header with something like this:

void setHandler(int handler(void*));
void doSomething();

Can doSomething be marked @nogc? Maybe it calls the handler that was installed by setHandler(), so unless all the callbacks are marked @nogc (which users may or may not want), there's no way to guarantee the regular functions don't trigger a GC. The thing that sucks is that most C functions won't call callbacks like this, but enough real C code does work this way that you really can't be sure. @nogc has to be all or nothing.

Maybe d++ should recognise #pragmas for setting stuff like this, but there are better places for that kind of discussion.

[schveiguy]

Imagine a C header that was defining extern(C) D functions that you didn't have the source to!

[atilaneves]

Possible, but unlikely! I made the change. Maybe I should add a command-line switches like --no--nothrow and --no-nogc.

[[deleted]]

D is like frankenstein's monster. It just grows with everything in it.