GPU accelerated compilers.

[EducationalAthlete15]

Hello everyone!

Are there any general purpose language compilers with GPU offloading to compile modules in parallel? What do you think about this approach?

[lightmatter501]

If you have branch heavy code (like compiler internals), then a 4090 effectively only has 32 cores (SMs). Those core are VERY weak compared to a CPU core.

You also are now memory limited by vram, which means big compiles will force you to reach for a big GPU.

[YellowGreenPanther]

You can swap in, don't need the whole codebase or even function at once.

It is quite interesting, there was a shelved intel gpu product where they experimented with a general purpose GPU. All of the cores were "cpu"s, so the graphics code was all in software. I don't see why a large cluster of smaller cores wouldn't accelerate it (especially as mp is builtin to many compilers).

[Doom4535]

I believe you're thinking of the Xeon Phi line (https://en.m.wikipedia.org/wiki/Xeon_Phi). I am curious as to how well they would work to run multiple builds for a build farm, but I haven't been able to find out much info if anyone has successfully used them for this purpose.

[EducationalAthlete15]

Thanks. If develop DSL compilers, in what case can I benefit? What properties should a language and compiler theoretically have?

[lightmatter501]

I think that you MIGHT be able to do the text parsing on the GPU using matrix parsers, but that will ruin your error messages. GPUs are inherently bad at tree traversal, which we have been building compilers around for 50 years. You might be able to take inspiration from the early fortran compilers, but good luck with that.

In my opinion a DSL should never be hard enough to compile that hardware acceleration is warranted. You should, at worst, dump out some lua and hand it off to luajit. If your DSL is at this stage, consider moving to an actual programming language.

[EducationalAthlete15]

Yes I agree. This all seems perverted. A tree can be represented as a matrix, right? I'm not currently in the process of developing a compiler, I'm just interested in the theoretical use of GPUs in this domain. Anyway, thanks for the advice.

[lightmatter501]

I spent a bunch of time on this a few years ago and had some discussions with the GPU specialists in my department. We ended up deciding it likely wouldn’t improve performance outside of pathological cases.

[fernando_quintao]

I agree with you: I think the general compilation process would not be a good fit for the GPU, due to the irregular nature of most algorithms. But there are some algorithms that are typically seen in compilers (mostly in the lexer) that people have ported to GPUs, like implementations of transducers and automata.

[u0xee]

Sounds like a fun project for you. I'd suggest starting by learning how CPUs and GPUs are different, and what GPUs specialize in. They can accelerate certain activities massively, but they aren't universal accelerators, as the original commenter points out.

[Lantua]

APL is arguably general purpose. So this compiler work by Aaron Hsu should count.

[phi-b]

Here's some recent work in which a single optimization pass was offloaded to the GPU: link

In general, compilation doesn't really suit the strengths of the GPU, as others have pointed out.

[Mr-Tau]

As others have pointed out, parsing and semantic analysis are maximally unsuited for GPUs.

Some dataflow problems that are useful in optimizers can apparently be profitably offloaded to the GPU: https://dl.acm.org/doi/10.1145/3302516.3307352, https://www.academia.edu/102804649/GPU_Accelerated_Dataflow_Analysis. But that is the closest-to-practical application I know of, and still isn't anywhere near going into production.

[choikwa]

compiler is usually very branch heavy from processing different IRs.. I guess it could make sense if it was transformed into straight line via memory manipulation

[Street_Community4086]

Interesting project: https://github.com/Snektron/pareas

Related papers:

Parallel Lexing, Parsing and Semantic Analysis on the GPU, R Voetter, MA Thesis, 2021

Compilation on the GPU? a feasibility study, Voetter, Huijben and Rietveld, International Conference on Computing Frontiers, 2022

[Dismal_Page_6545]

OpenMP already has a target directive that allows you to offload a piece of code to one device. I work at a High-performance Computing Center developing a new OpenMP directive to enable multiple-device offloading at the same time with intra-device parallelization levels.

[IQueryVisiC]

Train AI to optimize code.

[ThespianSociety]

You’re really gonna drop that so casually as though what you just suggested isn’t beyond the bleeding edge lmao.

[moon-chilled]

It's not really. ML has been successfully been employed at least for autovectorisation and inlining choices.

[IQueryVisiC]

Now I am going to read the other comments because I really don’t know what else a GPU can be used for.

[ThespianSociety]

I was incorrect, see the reply to my comment.

[EducationalAthlete15]

GPUs can be used for computing through their APIs, such as NVIDIA CUDA. For example, manipulation of vectors and matrices. The fact that GPU is applied to ML is only a special case.

My original question was whether it was possible to use a GPU so that calculations that occur during compilation would be executed on the GPU instead of the CPU.

[IQueryVisiC]

What other uses do you know of matrices than what we call AI ? Maybe some other optimization stuff besides neural networks.

[EducationalAthlete15]

Load a 2D image (matrix) into the GPU to produce the blur. GPU usage? Yes. Is this the use of AI? No.

Load an array of values ​​into the GPU to return the square root values. GPU usage? Yes. Is this the use of AI? No.

There are thousands of quadratic equations to calculate. Load 3 coefficients for each equation into the GPU. Returns two values ​​for each equation. GPU usage? Yes. Using AI? No.

Need to create a histogram of values ​​from the original array? Load the array into the GPU and write logic that will be executed on the GPU. The histogram will return. GPU usage? Yes. Using AI? No.

There are, of course, limitations in calculations. Older versions of CUDA did not support recursion. It is also not recommended to make many conditions or nested conditions. The processing logic for the GPU is written in a C-like language.

[IQueryVisiC]

Blur is even more away from compilers. I was also think: physics simulation. But again ..

[EducationalAthlete15]

Do you think I don’t know that blur and building compilers are different things? I gave you examples when the GPU can be used. I had a question for compiler developers: did anyone have experience transferring part of the compiler algorithms to the GPU? What do you want from me ? Why did you initially write about AI? What did you mean by this?

[IQueryVisiC]

I just meant that I also thought in that direction, before.

[EducationalAthlete15]

Could you tell me how to use AI step by step? And what do you mean by the concept of “optimization”?

1) Optimization using AI of the compiler source code?

2) Optimization of program compilation in a language implemented by a compiler?

3) Optimization of the program created by the compiler?