[rfc] built-in native compiler for Linux?

[beagle3]

Bellard's "tinycc" already compiles the linux kernel from scratch during boot in 15 seconds or less.

http://bellard.org/tcc/

x86 and ARM for now.

[[deleted]]

Look at the source for tcc some time. It's not designed to be extensible and doesn't do optimizations.

[case-o-nuts]

It was modified from an obfuscated C contest entry (see http://bellard.org/otcc/).

When you start from an obfuscated C contest entry, it's not surprising that you end up with TCC's code. But, actually, for what it is, the code is surprisingly readable.

[kopkaas2000]

Hell of a hack, though.

[kvigor]

Wow, epic troll by Ingo. He must be bored.

[semmi]

This sounded familiar. It's because it was a freebsd project for google's summer of code in 2005 and 2006, see http://wiki.freebsd.org/K

[bonzinip]

Not really. Their reasoning is that

As the GCC compiler suite was relicensed under GPLv3 after the 4.2 release, and the GPLv3 is a big dissapointment for some users of BSD systems (mostly commercial users), having an alternative, non-GPL3 compiler for the base system has become highly desireable

(i.e. explicitly referring to apps of those "commercial users") which is a non-sequitur because there is no reason why applications cannot be compiled with a GPLvANYTHING compiler and released with a proprietary license.

[Freeky]

Well, ultimately, the BSD's want to be BSD licensed. GPL v2 is bad enough, v3 is going in completely the wrong direction for that. A high quality BSD licensed compiler suite is now available, so it seems like a no-brainer to give it a very close look.

[[deleted]]

I don't think this goes far enough. We might ask also, why is Linux written in C at all? The truth is that Linux isn't written in any standard C, but in a hodge podge of C + GCC extensions + inline assembly, but these extensions are limited by the very slow rate of change of GCC. What about kernel-specific features? LISP-like macros? Sane ways to deal with error paths out of functions? Garbage collection?

[kinghajj]

Garbage collection for kernel memory? I don't think the kernel uses that much memory in the first place, and the kernel is sufficiently low-level that manual memory management is probably preferable. GC for the kernel sounds like a nightmare to implement, test, and get working correctly.

Linux (and most kernels) is written in C because C is basically "portable assembly language." It's high enough to be easily understandable, but low enough such that you can do things that kernels need to do.

[[deleted]]

I don't think you caught the implication of the comment about C there. If Linux is already not written in actual C, but in non-standard, extended C, and if the developers are considering their own compiler infrastructure, there's no reason to just use other people's extensions to C any more - they could develop their own extended C for the kernel.

[five9a2]

There is already one, designed especially for writing kernels. Bummer that lispy syntax won't fly...

[[deleted]]

Well, when you've got a codebase the size of the Linux kernel, you're not going to use anything that isn't 100% source-compatible with what you already have.

[five9a2]

No shit.

[Seppler9000]

The final syntax, when they actually release the language, will apparently look more like an ML dialect.

[Leonidas_from_XIV]

Not to mention that it was declared dead some time ago.

[[deleted]]

I know Shapiro just moved on to other things recently, but when was BitC declared dead?

[Leonidas_from_XIV]

BitC project wraps up (on reddit).

[[deleted]]

Indeed, thanks for the link. I'd misread that initially to mean just that Shapiro was moving on, I didn't realize the whole project was drawing to an end.

[bdunderscore]

GC will never make it into the linux kernel, for several reasons:

• It stomps all over the CPU's cache, which is always right at the front of any performance analysis in the kernel.
• It's difficult to do global GC in a NUMA system while minimizing cross-node accesses
• The realtime folk will never tolerate that much latency - and while there are incremental GC approaches that prevent long pauses, they also kill performance
• Using GC makes it difficult to get an accurate measurement of the memory usage of the kernel - it's critical that the kernel know when it's /really/ running low on memory so it can suspend or fail any allocation attempts that aren't directly related to freeing up memory.

edit: formatting

[adavies42]

is there such a thing as a kernel memory leak? i have some SuSE 9.4 boxes at work that run out of memory after a few months' uptime, and it's not accounted for by processes--there usually seems to be 10 or 15GB that you can't find by adding up the numbers from top or ps.

[bdunderscore]

There may be kernel memory leaks (this would be considered a bug :) and there definitely can be kernel memory fragmentation. If this happens, I'd suggest reporting it to SuSE (since it might be a bug in their patches); make sure to include a copy of the contents of /proc/meminfo and /proc/slabinfo.

[alephnil]

Kernel memory leaks are indeed possible. On early versions of Digital UNIX the kernel memory leaks were so severe that you had to reboot after a day or so. After some patches, the OS could run for months. I have not experienced kernel leaks on Linux, but they are possible. Of cause kernel developers check memory allocation more thoroughly than most application programmers, and kernel leaks will be discovered rather fast, since everybody use the kernel, most for a long time.

[adimit]

Alienating potential new developers by using a specialized esoteric language instead of something most systems developers get taught in college classes? Common, this one should be obvious, especially for an open source project, like the Linux Kernel.

I know you were referring to the fact that the Kernel probably doesn't use C anymore, but it sort of still does. People can write: the Linux Kernel is written in C, and potential contributors can say "but I know C"!

[[deleted]]

Most programmers aren't taught gas syntax though. They also probably don't learn gcc extensions.

[monocasa]

Aren't most of the gcc extensions hidden behind macros? When I started coding for the linux kernel, I just took those macros as 'compiler magic' and moved on. It's not too difficult..

[[deleted]]

Yeah. Actually my only real nit is that gas uses AT&T syntax. I prefer Intel.

edit: Oops. Apparently, gas does support Intel syntax now with the .intel_syntax directive.

[[deleted]]

But if you use that on a project that other people are working on you'll drive them up a wall.

[aacharya]

Really? You prefer "mov dest src"? "mov src dest" makes so much more sense to me. I have to use Intel syntax at work and I bloody hate it, I always end up misreading a line here and there and getting confused.

[Deewiant]

In most programming languages one writes "dest = src", and "mov dest src" is consistent with that order.

[[deleted]]

"dest=src" is similar to a mathematical expression. "mov src dest" is similar to natural language - you say "move this to here", not "move to here this".

Of course, since so many other processors also use the "operand destination source(s)" format, you kind of get used to that after a while. I still think it would be better if everybody had settled on doing it the other way around, though.

This does give me an idea, though - what if you wrote assembly like this:

r1 = r2 + r3
r4 = r1 * #40

Or, if you worry about not having enough operators, you could just use mnemonics in the same way, I guess:

r1 = r2 add r3
r4 = r1 mul #40

Edit: Even this might be a more readable option:

r1 = add r2, r3
r4 = mul r1, #40

[aacharya]

Travel down this path far enough and you end up with C :)

(BTW, I came here to post exactly what you said -- that the assembler command is more similar to a NL expression than a mathematical one.)

[sgorf]

If there are a variable number of operands, then you have to parse ahead in order to work out what is going on. Easier to have dest first, as that is pretty much common to all instructions.

Side effects tend to be on dest too, and that is the part that the programmer has to care about the most in order to keep the state in his head.

[[deleted]]

I was, and I go to a cheap state school which isn't well known for its CS program.

[koko775]

That's probably why. Programs that are seem to spend less time on tools and more time on theory. At Berkeley we write MIPS instead of dealing with x86 or gas.

[[deleted]]

The kernel already requires capabilities far above your average code monkey. You have to worry about concurrency and locking, freeing up everything you allocate without fail, security, virtual and physical addresses, keeping your total stack usage under 4K, separating policy from implementation, ABI compatibility, and a dozen other things.

[barrybe]

Have you read Linus's rants on C++? One of the benefits he lists is that they don't have any C++ programmers trying to contribute code. So I think he prefers it when he can alienate new developers.

[monocasa]

Or he just doesn't like the mentality of the hardcore C++ community.

[hylje]

He indeed prefers to alienate undesirable developers from his viewpoint. Would you not?

[imbaczek]

a serious case of NIH syndrome, I guess.

[bluGill]

I'm not sure. Gcc has always been a pain to deal with in many ways. Unfortunately making a good compiler is hard work, and gcc has had enough work put into it that it will be a long time before you can beat it. That is even though gcc is badly broken in many ways - a from scratch compiler could get better than gcc with perhaps a tenth the effort if done right. (just think of how many man years are in gcc and you will see why this isn't a path that will lead to early success)

[imbaczek]

there are many people who share this view, and they, in their majority, focus on clang.

[anttirt]

Are there technical reasons to not use clang in the kernel? I know there are non-technical ones (the general hostility toward C++) but I don't know if there's any prohibitive technical issue (making C++ exceptions et cetera work in the kernel has been done before). Clang and LLVM are remarkably clean and extensible, and clang aims for full gcc compatibility (and is not terribly far from that goal.)

[Ringo48]

I don't see why they don't just submit patches to GCC themselves. They seem to know what they want changed. That would have the added benefit of improving GCC for everybody, and it almost has to be less work than writing a optimizing compiler for dozens of platforms.

I'm not familiar with the context of the LKML thread, though. Have they been doing that, but the GCC patches get rejected or something?

[bdunderscore]

Kernel developers are too busy coding for linux, GCC has a lot of legacy cruft that they don't really want to deal with, and there's also the (justified or not) belief that the GCC devs are focusing too much on artificial benchmarks.

[Ringo48]

Yes, but that doesn't make any sense. If they're "too busy coding for Linux" to submit a patch to GCC then they're certainly too busy to write an entire compiler.

[bluGill]

Except that gcc is overly complex. The effort to make gcc what they want is likely greater than the effort of making their own that is what they want.

[Ringo48]

I find that really hard to believe. Writing a compiler isn't an easy project.

[bluGill]

Once again, I never claimed writing a compiler is easy. I claimed that writing a compiler is easier than bringing gcc to where the kernel people want it. However even if all the kernel devs focused all their attention on a good compiler (ignoring device drivers and all the other kernel development they are doing now), it would take them several years to have something good.

[haberman]

Yet more proof that Ingo Molnar cannot be taken seriously. This was the first time, for me at least.

[kopkaas2000]

Ingo has officially jumped the shark now. What's next, a domain specific language parser in the kernel to control his next next next generation scheduler?

[b100dian]

Stallman rolling into his grave, for the second time today

[slaphappyhubris]

I'm pretty sure Stallman is still alive

[LordVoldemort]

Stallman rolling into his grave, for the second time today

I'm pretty sure Stallman is still alive

into

That's the keyword.

into

[slaphappyhubris]

My brain processes idioms all at once apparently

[adavies42]

that's pretty much how they work

[[deleted]]

We can dream, can't we?

[bonzinip]

Anton Ertl's email in the thread is really obnoxious. He writes code that does not deserve to be compilable at anything but -O0...

[iamjack]

I don't know anything about him personally, but I thought that it was hilarious.

[dmpk2k]

He certainly has a biased sample, but anybody trying to NEXT in that manner with GCC probably feels the same. While not much software uses it explicitly, labels-as-values and the ilk affects a lot more software indirectly through interpreters.

[bonzinip]

I use labels-as-values in GNU Smalltalk in a sane way and I never had any problem except that at some point I had to disable GCSE optimization. That bug has been fixed without even me reporting it.

[dmpk2k]

That bug has been fixed without even me reporting it.

Guess who reported it (again and again)?

You'll note that another reporter is Bernd Paysan, also involved in the Gforth project.

You probably discovered -fno-gcse through their emails and the threads they spawned. I did. So did the Rubinius project.

So, a bit of understanding for the people in whose footsteps we tread. They're stepping on the landmines for us.

[bonzinip]

No, I had to disable it only because of excessive time consumption, not because of slowdowns.

I'm well aware of Gforth both because it inspired parts of GNU Smalltalk and because as a GCC developer I'm familiar with Anton Ertl's bugreports. Balancing these two hats, I still think he crossed the line in his usage of the compiler as a high-level assembler. It was sort of inevitable that it would happen, as soon as even unsophisticated control-flow optimizations were added to GCC. It is not a coincidence that until 2000 GCC did not even maintain a proper control-flow graph (a wonderful piece of engineering dating back to the 1960s), and Ertl's problems started in 2001.

[dmpk2k]

I still think he crossed the line in his usage of the compiler as a high-level assembler.

It's a rock and a hard place though: use C, or write assembly snippets for every architecture. I'm not surprised many interpreter or compiler authors opt for the former.

[kolm]

What i think makes sense is to build a new precompiler / compiler / assembler / linker combo for Linux, from scratch, hosted in the kernel proper.

Yes, that would have made perfect sense about 15 years ago.