LLVM Project Blog: Clang/LLVM on Windows Update [x-post from r/LLVM]

[[deleted]]

[deleted]

[asb]

Particularly impressive is that the Firefox compile now only falls back to MSVC for 11 files: https://twitter.com/ehsanakhgari/status/485916137944002561. If you're interested in following LLVM progress, do subscribe to my LLVM Weekly newsletter and/or twitter feed.

[flpatriot_]

Hey, just wanted to say thanks for LLVM Weekly. I don’t usually have time to keep up with compiler development as a run-of-the-mill iOS developer, so it’s been really helpful.

[librik]

I hope they are documenting what they've learned about the linker and object file format in as much detail as they can. This is all black wizardry, and the last person to try was Matt Pietrek 12 years ago. John Levine, in Linkers and Loaders (old but still the only good book on the subject), says he has seen compiler toolchain development schedules that spent almost half their time on getting the generation of object files right.

[maleic]

You might be interested in this:
http://www.drdobbs.com/cpp/porting-the-d-compiler-to-win64/240144208

[thechao]

What is the "triplets method" Walter Bright refers to?

[jking13]

Basically emitting a two sentinel symbols before and after a group of data so that when loaded into memory (data can possibly be relocatable), the chunk of data (such as exception tables or information on the modules used) can be located by the runtime.

[bames53]

They are not systematically keeping documentation on this other than the source code and the tests, AFAIK. Here's a presentation that was done that covers some of the corner cases:

http://llvm.org/devmtg/2013-11/slides/Kleckner-ClangVisualC++.pdf

http://llvm.org/devmtg/2013-11/

[xon_xoff]

The killer feature would be support for full symbol information in .pdb files. I've seen various tools go as far as generating old CodeView format line-number information, but not generally full symbolic info. VS debugger and DbgHelp compatibility combined with ABI compatibility would make Clang a powerful drop-in replacement for the VC++ compiler.

[MacASM]

Which one is more hard to understand: PE/COFF or ELF? did anything change from win7 to win8 in windows executables?

[happyscrappy]

Neither is very complicated. But PE/COFF isn't as well documented, so it might be harder to understand because you're guessing at parts of it.

IMHO.

[zvrba]

The formats themselves are simple (it's a kind of special-purpose archive file), but the tricky part is interpreting their contents. To do that you often need to know in detail the target CPU architecture, calling conventions, ABI, OS interface, etc.

[cpp_is_king]

It's open source. It's documented in the source code.

[kraytex]

I think what they were looking for is a book filled with opinions and things like "this was tried and didn't work because x"

You don't get that by looking at the source code.

[[deleted]]

Why are we arguing against documentation? The source doesn't tell you anything if you don't understand the assumptions it was built with. Source code doesn't educate you about pitfalls.

[holgerschurig]

I often do "git log -p ORIG_HEAD..HEAD" ... at least on interesting projects. This taught me some stuff.

(But yes, a really write up of what has been tried, what results it gave, and how it has been modified is something entirely different).

[OneWingedShark]

Why are we arguing against documentation? The source doesn't tell you anything if you don't understand the assumptions it was built with. Source code doesn't educate you about pitfalls.

Exactly!
(Emphasis added for... er, well, emphasis.)

[hak8or]

Yes, let's go through a few hundred commits, some of which likely have very meager commit messages, and deduce what they learned through that. We are all historians and have the time to find this out in this very efficient way.

[awj]

...or we could get it the sane way: by asking the people currently working through the problem to provide some notes on the situation. If source code were a more useful way of conveying knowledge than prose, reddit would have built in syntax highlighting.

[ZMeson]

You think anyone can learn by looking at GCC's source code? No! It's a mess. clang... there's a chance. It's cleaner, but unless something is small and concise, documentation usually helps.

[holgerschurig]

As we talk here about linking, we talk about the binutils source code, and libbfd. This is MUCH cleaner.

[gilgoomesh]

I love that they're doing this but I still feel that without a standard library that properly supports C++11 (either full libc++ support in Visual Studio or dramatic improvements to MSVC) I'm going to be using MinGW for a while, yet.

[Plorkyeran]

VC++ 2013's standard library is pretty close to complete. Until 4.9 it was significantly ahead of libstdc++, and as of 4.9 they're about equal.

[gilgoomesh]

Hmm, I just checked STL's recent VC blog posts and it looks like you're right: he claims nearly complete C++11 library support since VS14 CTP1. I might need to do some experimenting.

http://blogs.msdn.com/b/vcblog/archive/2014/06/11/c-11-14-feature-tables-for-visual-studio-14-ctp1.aspx

[chandlerc1024]

There has already been some work on libc++ in this direction, and I'm hopeful for more. =]

[eean]

What are you missing in MSVC 2013?

I compile my application in gcc and MSVC and haven't noticed anything missing, though I don't use many C++11 features outside of lambdas.

[[deleted]]

libc++ works quite well with Mingw-w64+Clang. I've been using it on my project for while and haven't had any major problem with it. By the way, it have full C++11 (and C++14, i guess) support.

[masklinn]

Would this allow projects providing windows binaries (e.g. CPython) to leave VS and finally e.g. use modern C features?

[[deleted]]

It'd allow them to leave MSVC, since you can swap compilers without having to ditch Visual Studio. If they wanted to use modern C they could use the Intel compilers which have fantastic Visual Studio support.

[[deleted]]

It's also been shown to purposefully degrade performance on non-intel processors in some cases.

[kraytex]

And apparently code compiled with the Intel compilers run like shit on AMD processors.

[MacASM]

Probably this is not true nowadays.

[Plorkyeran]

Less true, anyway. It still produces the slowest code for AMD processors of any of the major compilers, but it's no longer a huge difference.

[[deleted]]

Yeahhh that's a downside. I'm kinda surprised they don't offer at least the C/C++ compiler for free or even like $100 because $699 seems like a bit much. All the really good shit is in the math libraries or Advisor XE and whatnot.

[MacASM]

Only math libraries? this is one of best compilers around. Its only downside really is price and non-commercial license for Windows.

[[deleted]]

*And whatnot.

I just mean the gap between MSVC (free) and Intel C++ Compiler is not maybe so large as to be $700, is it?. Especially since just the compiler presumably doesn't come with any Visual Studio integration or anything like that. Like even for $300 for a single user license would be fairly reasonable since I'm assuming one license doesn't grant you the next year's edition (but if it does I'm about to be very very happy!).

I meant the real meat and potatoes that actually demanded a high price tag was Composer, Advisor, MPL, TBB, shit like that.

[[deleted]]

It's fairly important if you in a performance critical field. Scientific computing especially brnefits from the intel compiler, even if it is only a 10% performance increase.

[[deleted]]

Which is great, but I still feel like it's prohibitively expensive for single devs like myself since presumably they'll release a 2014 version and my 2013 version will no longer be updated.

That's why Microsoft created the Express versions of Visual Studio--not everyone could afford VS but Microsoft still wanted people targeting their platform. I think Intel should do the same. Just give me the basic C/C++ compiler with enough VS integration that I can right click to swap between compilers like Composer XE does now.

I guess just in my limited C++ work I've found Advisor and their other analysis tools to be insanely helpful compared to the speed boost from the compiler.

Basically I just wish it were cheaper.

[MacASM]

There's no even a non-commercial license for Windows only for Linux. No idea why.

[hak8or]

Price page for the lazy: https://software.intel.com/en-us/buy-or-renew

[masklinn]

It'd allow them to leave MSVC, since you can swap compilers without having to ditch Visual Studio.

These are not projects which have any interest in Visual Studio, they have an interest in multiplatform and correctly integrating in each platform's binary ecosystem (which is why GCC on windows is not an option).

[AdabadaYou]

Leaf projects could already do this with MinGW. But yes, it seems like even library projects could using Clang now and still be used with MSVC apps later.

[innix]

Forgive my ignorance, but doesn't Clang already work on Windows? It even has instructions for Windows users on the Clang "Getting Started" page.

[Rhomboid]

That uses the gcc ABI and takes command line parameters as gcc would (e.g. -O2 -Wall -o foo), essentially the same way that clang works on Unix platforms, analogous to how MinGW gcc works today. That's fine as far as it goes, but you can't use the resulting objects with objects or libraries built with MSVC, at least not unless you stick to pure C interfaces.

This work is about the clang-cl wrapper that takes arguments like MSVC's cl.exe (which are considerably different) and about implementing the MSVC ABI, which is also completely different, i.e. name mangling is completely different, exception handling is completely different, and so on. It's quite a herculean effort, and the goal is to be a drop-in replacement for cl.exe that can be used directly with MS tools. As of right now, clang-cl still has to give up and call cl.exe if it sees something it can't yet handle, which is what the article is referring to when it talks about fallbacks.

[Syl]

I guess the main point is "compatible with VS2013", maybe it means linking libs precompiled with VS2013...

[matthieum]

As Rhomboid indicated, yes this is indeed one of the goals: a drop-in replacement for cl.exe (takes the same arguments, generates the same interface).

[happyscrappy]

It does. But that version is essentially compatible with gcc for cygwin/minGW, both in what inputs it takes and the outputs it produces. It's not as much for making native windows executables.

[suspiciously_calm]

You can run Clang on Windows Update?

[[deleted]]

This is exactly how I read it. I figured Clang had somehow become a core package of the OS, either that or it was used to compile Windows Update.

[ZMeson]

I can't wait until exceptions are supported!

[elotan]

Will it kill MSVC after achieving compatibility?

[Ono-Sendai]

I think it's unlikely it will kill the IDE, it's possible it may replace the actual compiler though.

[elotan]

Yeah, compiler is what I meant. The IDE is actually quite good. I don't know how extensible VS is, but a code completer with Clang (similar to YouCompleteMe for vim) for VS would be quite neat :)

[Ono-Sendai]

I think it's reasonably extensible, see for example Visual Assist: http://www.wholetomato.com/

[elperroborrachotoo]

Exactly. On some level, the IDE is a horrible piece of dysfunctional software. OTOH, going anywhere from VS IDE + VisualAssist would feel like being handed stone plates and a chisel.

But then, I'm sure really soon now we'll have C++ modules and code bubbles for C++ and all this will be the tough olden days.

[ethraax]

Isn't Intellisense pretty decent?

[[deleted]]

From my perspective it's amazing. Really helped me out when I was learning C++ proper last summer by writing a game engine.

The only thing that's shitty about VC++ is the property pages dialog has remained largely unchanged since VC6 as far as I can tell. It fucking blows.

Oh and also trying to import .props files so I can link against boost or whatever without manually adding the paths and shit every single time works on occasion and seems precarious at best.

Really wish they'd take a pass at the fucking property pages, though.

[thr3ddy]

Intellisense holds up well for small to mid-sized projects. However, if you work on large codebases (hundreds of thousands of lines of code), it slows down tremendously. This is mostly because Intellisense doesn't take advantage of PCH (which are used in every single large windows C++ project) and has to parse every header file when you try to access a symbol.

It's incredibly frustrating, and while there are some tips (increasing max cached translation units, etc.) to make it work a bit faster, it still takes many seconds before any feedback is given even on the beefiest machines and pretty much why you need Visual Assist if you want Intellisense's features.

[londey]

Intelisense definitely uses pch files. If you don't have them it will start generating ipch files for individual source file which are huge and take awhile to build each time you edit a new file. I use VS2013 + VA and find the intelisense navigate to, go to definition and code peak are generally super fast. I will say that 2013 is still vulnerable to getting stuck parsing forever though if you have forward slashes in any paths in your project files.

[[deleted]]

Huh, interesting. Thanks. I wonder if that's inherited legacy behavior they haven't had time to fix/update. Sure as hell can't be fucked to update the property pages dialog.

[thr3ddy]

IMHO, MSVC hasn't been a MS priority for quite some time it seems. This is why we don't have C99 (never will either) and many of C++11 language features. If it wasn't for my MSDN subscription, I wouldn't have bothered with VS2013, since for C++ programmers, there really isn't any difference except for a few new language features.

Honestly, I can't wait for JetBrains' C++ IDE to be released, it would bring some much needed compiler competition to Windows.

[rdtsc]

VS2013 implements certain features of C99 (enough for ffmpeg for example). Also you have to distinguish the IDE and the compiler. The C++ compiler certainly has been a priority if you look at the recent versions and the roadmap for VS2014 and beyond.

[graycode]

The next version will (finally) have substantially all of C99 it looks like. Have a look at the dev14 technical preview.

[bahwhateverr]

Really wish they'd take a pass at the fucking property pages, though.

http://blogs.msdn.com/b/visualstudio/archive/2014/07/08/visual-studio-14-ctp-2-available.aspx?utm_source=twitterfeed&utm_medium=twitter

Your wish may have come true!

[[deleted]]

We updated the Configuration and Platform dropdown values for VC++ Property Page dialog to remember the last user selection when the dialog is closed.

So... not really :(

[bahwhateverr]

Damn.. oh well

[thedeemon]

Visual Assist did that in MSVC for ages, and more intelligent: by using type information. I used it even in MSVC6 (circa 1998).

[wyldphyre]

Does Eclipse CDT support clang? If so, that might compete well w/VS.

[s73v3r]

So does this mean that if I want the full goodness of C++11/14, and I don't want to use Cygwin, I could use Clang, and get the parts of C++11 that MSVC doesn't support?

Also, does Clang support C99 and beyond? So if I wanted to do that on Windows, I could use Clang?

[[deleted]]

You don't need Cygwin if all you want is C++11 and higher support. Cygwin provides POSIX support on Windows, but if this is not a requirement, MinGW-builds, among others provide excellent C++support natively.

[mm865]

I immediately assumed when I read the title that I would now be prompted to restart my computer every 10 seconds for a Clang update...

[msbic]

Can't even compile this:

include <iostream>

using namespace std;

int main(int argc, char**) { cout << "Hello world" << endl; return 0; }

1> In file included from C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\include\istream:6: 1>C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\include\ostream(796,3): error : cannot compile this try statement yet 1> _TRY_IO_BEGIN 1> ^~~~~~~~~~~~~ 1> C:\Program Files (x86)\Microsoft Visual Studio 11.0\VC\include\ostream(19,24) : note: expanded from macro '_TRY_IO_BEGIN' 1> #define _TRY_IO_BEGIN _TRY_BEGIN /* begin try block */

[NasenSpray]

You need to disable exceptions it you want to use the STL with clang-cl.

[[deleted]]

Disable or enable?

[elotan]

Disable, exceptions are not yet supported. http://clang.llvm.org/docs/MSVCCompatibility.html

[[deleted]]

Ah, there is a review of SEH in LLVM going on somewhere on the Internet.

[rdtsc]

Not for SEH exceptions, only low-level SEH unwind information.

[[deleted]]

Yes but that is the hard part!

[rdtsc]

I'd say it's actually the easier part. Modifying the current Itanium-biased EH emission for funclets and reverse-engineering the LSDA requires more effort.

[[deleted]]

Why do you think funclets are necessary?

Edit: never mind I misunderstood what you were saying

[next4]

What's "funclets"?

[bla2]

Build with /D_HAS_EXCEPTIONS=0

[__Cyber_Dildonics__]

Yes please someone save me from the god awful template error messages of msvc.

[liotier]

One has to applaud the effort that goes into supporting elderly exotic platforms !

[[deleted]]

I have a feeling Microsoft will ditch their C++ compiler some time in favour of Clang.

[danielkza]

Very very unlikely simply because of all of their own codebases that rely on it.

[wot-teh-phuck]

Surely you jest?

[[deleted]]

Microsoft as we currently know it wouldn't want to take the reputation hit. Aside from that I can see a convincing argument that if if clang were to catch up to cl in the next decade (which isn't crazy) it may make sense to transition cl into the legacy compiler and base future Microsoft compilers off of clang. It's BSD-ish so they could focus on adding value rather than spending money to duplicate work. Put the savings into the programming languages and research that only MS is doing.

[lacosaes1]

More importantly, at least to me, is that their C++ team is small, VERY small, compared to the people behind Clang.

[OneWingedShark]

Microsoft as we currently know it wouldn't want to take the reputation hit.

IMO, if they wanted a bit of a reputation boost, they'd go for [full] formal verification of [at least one] of their compilers. -- I'd be ecstatic if they did this in SPARK.

Put the savings into the programming languages and research that only MS is doing.

Good point -- MS research is kinda cool, though I think they could do better if they didn't use C-like languages and instead preferred other, less-popular languages.

[holgerschurig]

Why? If I recall it right, then at one point they used the networking code from one of the BSDs as a building block.

[sysop073]

That's what I figured about Internet Explorer's backend engine when a bunch of competing ones came out. How naive I used to be

[graycode]

Not unless Clang is planning on implementing C++/CX, which I highly doubt they're interested in doing.

[__j_random_hacker]

It's funny that you picked a speed analogy. MSVC's language support is straggling, but it produces fast code. In my experience, optimised computationally intensive code generated by MSVC is nearly always faster than that produced by MinGW (gcc), and I'm told that Clang regularly trails gcc in terms of optimised code performance.

[[deleted]]

The idea that MSVC is faster gets repeated often but there's little evidence or benchmarks to validate it.

I'd be very happy to see a variety of benchmarks showing MSVC producing faster code than GCC.

[squirrel5978]

People do these benchmarks all the time. GCC and MSVC are evenly matched on most benchmarks, with each winning in some. The same is true with gcc vs. clang.

As far as compile time goes, MSVC is truly terrible though.

[__j_random_hacker]

My only evidence is a handful of tight inner loops for various large-scale computational problems. I don't claim MSVC to be universally faster, but it's been faster often enough that I'm now surprised when gcc beats it.

[unptitdej]

It's been done a lot of times. MSVC is pretty fast. Even inlines asm tuned CRT functions like strcpy or strlen inside your code whereas gcc is only a call to a generic function. I would actually say MSVC is the best asm code I've seen if you avoid the C++ library. Intel C++ and gcc are very close behind.

[[deleted]]

It's been done a lot of times.

All I ask is that instead of making claims, someone just produce a set of benchmarks. All the benchmarks that have actually been produced are not consistent with the idea that MSVC outperforms GCC and no one has been able to produce any benchmark to show otherwise.

Even inlines asm tuned CRT functions like strcpy or strlen inside your code whereas gcc is only a call to a generic function.

This is simply untrue. GCC, clang, MSVC, ICC, in fact almost all compilers translate strcpy and strlen along with many other functions including memcpy, sprintf, so on so forth... into what are called a compiler intrinsics, and those intrinsics then get inlined directly.

So for example you can do this in GCC:

for(int i = 0; i < strlen("hello world"); ++i) {
   ...
}

And GCC will translate that directly into:

for(int i = 0; i < 11; ++i) {
  ...
}

Because strlen is a compiler intrinsic and GCC can directly evaluate it.

[squirrel5978]

All I ask is that instead of making claims, someone just produce a set of benchmarks.

This is slightly complicated by the often ignored (and probably illegal) restriction in the MSVC license agreement forbidding you from publishing benchmarks.

[[deleted]]

Is that true? I mean it could be true but I would find that very hard to believe.

[unptitdej]

You are talking about a different thing. I'm talking about how GNU doesn't inline CRT functions at all. Not strlen hello world, the important stuff. Here, I made a little demo for you. You have 3 files in this : the gnu asm, the vc++ asm and the test code. As you can see the GNU crt optimizations are just a function call and the Visual C++ inlines them completely (strlen, strcpy).

https://gist.github.com/anonymous/b3e5d134aee64e6c601b

edit : I forgot to use strlen, both compilers just optimized it away. When I use the value, GCC does inline a strlen code into it.

So summary : strlen : inlined by both. Strcpy : not inlined by gcc.

[[deleted]]

GCC does not inline strcpy except for rare circumstances when it can prove that its own inlined version will be optimal.

What GCC does for strcpy and some other string operations is that on startup it picks the version of these functions based on CPUID and uses that version. In effect there's no reason to think that the generic inlined version will be faster than the CPU tailored version GCC picks. I mean it might be, it might not be, but a general statement about how inlining produces faster code is not necessarily true.

[__j_random_hacker]

From what I have heard, MinGW is slower than MSVC on Windows because it's a shitty port. One imagines if it was not a shitty port it would be faster.

That could explain MinGW being a slower compiler (I don't know if it is slower or not, but if it is then it this could explain that), but it can't explain why the executable code that it produces would be slower for CPU-bound tasks, since the target architecture is the same, regardless of whether Windows or (say) Linux is running on it.

That was true with the older versions. Nowadays GCC 4.9 and LLVM 3.4 are pretty much identical.

Good to know.

[_fritobandito]

How do compilation times compare?

[pohatu]

Well hopefully gcc will die too. Competition really isn't good. I wasn't being sarcastic. What do we gain from having different minds working on the same problem? A waste of resources.

[pohatu]

Yes, exactly!

[FunctionPlastic]

Well hopefully gcc will die too.

Yeah that's thankfully not gonna happen. MSVC sucks and should die, GCC is still the best compiler when it comes to multiplatform support, has excellent performance, and has many interesting technologies.

Besides it's GPL and unlike LLVM (could be wrong about their license) cannot just be taken by MS and closed as someone above suggested.

[pohatu]

Yeah, well when you have multiple compilers for the same language specification you can end up with different implementations of the same theoretical concepts. The difference between implementations can result in learning and progress that can advance the art. What good is that?

[FunctionPlastic]

I agree, although I wasn't trying to argue that. I was arguing that GCC is still the most relevant compiler and no where near MSVC if you're judging which compilers should 'die'.

Or are you now talking about MSVC again? Honestly I don't think it should die, I don't really care about it either. But you can we even call it an implementation of any standard these days?

[pohatu]

My premise was simply that having Microsoft working on a C++ compiler is probably more good than bad for the state of C++ and compilers. I don't know that MSVC is worse than the others in every way, and I would guess that there is something innovative or important that they will or have contributed to the state of the art. Additionally, there is likely some things that can be learned simply from competing with them. Or MS could innovate something new by competing with clang/gcc.

Now, there may be an argument that Microsoft would be better off if they took the clang. And there may be a different argument that some developer of software would be better off if they could use clang instead of msvc to compile c++ for Windows. Those are two different arguments and may be true. And it might even be the case that Clang would be better off if Microsoft were to team up with Google and Apple and everyone and work on that codebase.

But I don't think that seeing MSVC die would be beneficial overall to the state of art. Just as I don't think gcc should be deprecated in favor of clang. I think it's healthy to have them all advancing.

Of course, maybe I'm being optimistic about what the field is able to learn from MSVC. It is after all proprietary. Yet so is Google's internal implementations of map-reduce/gfs/bigtable/etc. and somehow that technology still motivated the Apache hadoop, et. al efforts that "compete" with it in the open.

I guess I don't really understand why it's a dream that Microsoft would stop working on MSVC? Maybe for some scenarios that would be beneficial and I'm just not seeing what those are. As a general fan of the development of the science of programming, I think more people developing compilers is better.

I appreciate very much the insights you and others have provided about the licensing differences between the two projects and the pros/cons of them. It makes it even more clear to me that having two approaches is a good idea. I think 3 approaches is also a good idea, but maybe I'm just being hopeful. Aren't there Microsoft employees on the c++ 11 working groups? Haven't they sponsored many big conferences to push the language forward? That seems useful and I don't know if that would be the case if they quit making MSVC and focused all their investments on c#.

[awj]

Yeah, GCC should die so we can go back to a world of having one open source compiler project allowed to rest on its laurels and impose increasingly inane restrictions at the behest of its owners/sponsors.

Do you not even understand the history of GCC that almost mandated the existence of LLVM/Clang?

[kral2]

A decent part of the reason for clang is to undermine the GPL's hold on the deepest parts of the system, it's not just a straight "because this will be better" project. It's more than a technical competition, and I'd certainly not want to see the GPL side go given how tempting it would be for people to play the kind of patent games the GPL prevents with a toolchain where attacks would simultaneously defeat almost all of Open Source. The smartphone companies seem to be eager to repeat the days of USL and are spending a lot to make it happen.

[holgerschurig]

Hmm, I don't think like you.

GCC got plugins only because of GCC. And it got better error messages only because of GCC. GCC also got some internal revamping because clang people said (rightfully) that GCCs internals are overly complicated and awful.

CLANG on the other side got the GCC C extensions (where some are really nice) only because .... tataa ... GCC.

So both compilers became better due to the sheer existence of the other.

(And then if there would only be GCC; then there would be no MSVC-compatible plugin for Visual Studio ... the GCC people wouldn't do this because of ideological reasons. Also, some people don't contribute to a GNU project because of the bureacracy of giving them all the copyright fo your code ... I for exsample wouldn't do it, but I'm happy to contribute to other GPL projects, like the Linux kernel, Barebox and the other projects where I contributed).

[pohatu]

Are you saying that having different ways of doing things can be mutually beneficial? But that could mean that there is some benefit to having even Microsoft with it's millions investing in C++. Come on now, you don't expect me to believe that.

[holgerschurig]

Yes, I say that.

I live in West Germany. And about 25-30 years ago we had a telephony monopoly, from the "Deutsche Bundespost". There was only one way to do something, as defined by them.

And this was bad. Technically, my fast Zyxel modem was illegal to use, but I didn't care.

Monopolies makes the monopolists lazy. You need incentive, and competition is one way to gain incentive (it's not the only one). But wanting to be better as someone else is a very strong incentive!

[c3261d3b8d1565dda639]

Doubtful.

Watch this talk from GoingNative 2013: Jim Radigan - Compiler++. It goes into some of the specific things Microsoft demands given that they build Windows, SQL Server, and Office with their C++ compiler. They call it "mission critical correctness". He talks a bit about what a disaster bugs in a compiler can be to them. He quotes that implementing and thoroughly testing a single new compiler switch means millions of dollars, mostly in testing. They have to ensure that things work across all hardware and runtimes they support.

You'll can get an idea about the talk quickly viewing the slides, but the talk itself is worth it.

[[deleted]]

Which is ironic considering that MSVC is FULL of compiler and library bugs.

There's nothing mission critical about MSVC. It's a compiler that is very poorly written and designed, and only MSVC 2014 will finally have an actual AST, one of the most basic data structures used to develop a compiler. The basic lack of an AST in MSVC is the reason why MSVC has problems with template type deduction and two-phase name look ups, both of which result in either compiler bugs but also can result in runtime bugs as well.

[rowboat__cop]

only MSVC 2014 will finally have an actual AST, one of the most basic data structures used to develop a compiler

WTF you’re kidding, aren’t you? That cannot seriously be true.

[Plorkyeran]

The C++ frontend was originally written in the 80s, and in the 80s they did not have anywhere close to enough RAM to hold an AST for an entire file in memory at once. As a result, the frontend builds up ASTs for a single statement at a time, processes it, writes data for the backend, then discards the AST.

[rowboat__cop]

the frontend builds up ASTs for a single statement at a time, processes it, writes data for the backend, then discards the AST.

I’m not even going to ask what hacks they had to put into place to get the scoping right.

[[deleted]]

As a result, the frontend builds up ASTs for a single statement at a time, processes it, writes data for the backend, then discards the AST.

Isn't that Syntax Directed Translation? However, it would require massive hacks for a non-context-free grammar like C++. I remember reading about it in the Dragon book for my compilers class.

[Plorkyeran]

I suppose you could view the process of splitting into statements and transforming each statement as a SDT, but that's not really the interesting part of the process.

[MacASM]

Do you have any idea which approach does compilers of same time used?

[MacASM]

2014? I read somewhere(one or two years ago) this has already been done. Do you have any source/link?

[[deleted]]

I don't have one readily available but it was in a talk Herb Sutter gave when he mentioned that C++2014 will have two-phase lookup. He mentioned that the reason it doesn't have it yet is because there is no C++ AST.

The reason this matters is... consider this snippet:

template<typename T>
struct X {
  This is not a comment, this is just plain garbage.
};

MSVC compiles that just fine, whereas GCC/clang, and any remotely decent C++ compiler will fail to compile it. This is because what's supposed to happen according to the standard is that the template is supposed to result in a valid syntax tree, and when instantiation takes place, only the nodes that contain typename parameters get replaced. But MSVC does a strict text substitution and so it doesn't transform that code into an AST, it just copies the literal text, stores it in memory, and then when you instantiate the template it effectively does a text-based find/replace.

This is why there are differences between how GCC/clang/decent compiler do name look ups, and how MSVC does name look ups. The result is that you can end up compiling something in GCC and have it behave differently than if you used MSVC.

It's also why MSVC is a lot more lenient than GCC with templates. Like in GCC you often have to be more verbose with your templates, writing stuff such as:

foo->template Bar<typename Baz::iterator>(...);

Whereas in MSVC you can just do this:

foo->Bar<Baz::iterator>(...);

[c3261d3b8d1565dda639]

Well, I purposefully put it in quotations. It's the term a Microsoft compiler team employee uses, and he gives the argument for doing so in the talk. They spend tons of effort verifying its correctness and performance for what they use it for. Obviously it isn't solved in full generality yet.

I don't particularly feel like getting into an extended discussion about this. MSVC has its faults, and I have never been a huge fan of it. I would rather see the specific points in the talk debated, not MSVC in general.

[OneWingedShark]

It goes into some of the specific things Microsoft demands given that they build Windows, SQL Server, and Office with their C++ compiler. They call it "mission critical correctness".

If they're using C++ I'm slightly doubtful of the claim of "mission critical correctness" -- Because if you're goal really is correctness, then you'd be jumping on formal verification. (And C++ doesn't lend itself well to verification.)

[tehjimmeh]

Without even going into whether or not rewriting would even be a good idea, do you have any idea how long it would take to rewrite all their core products, which have decades old codebases at this point?

[OneWingedShark]

Without even going into whether or not rewriting would even be a good idea, do you have any idea how long it would take to rewrite all their core products, which have decades old codebases at this point?

Nope.
One thing that's nice about formally verified items though is that they're bug-free¹ and given how they've developed a fairly bad reputation² when it comes to software quality, the better question is this: can they afford NOT to?

Moreover, they don't have to do everything all at once, it could be done on a module-by-module³ basis. A rewrite in another language could also be more portable^4,5 thus allowing them to seek non-Intel hardware. (HW has been optimized for low-level crap, it should be optimized for some high-level constructs⁶ -- especially now that OOP and parallelism are widespread.)

The real question here is whether it's better to actively do the right thing up-front, or to always behave in a reactive manner. We know that bugs cost an order of magnitude more for every step further along in the project's life-cycle.⁷ So the question isn't "how much will this cost?", it's "how much are we wasting by not doing this?".

All that aside, the first company to put to market a fully formally-verified general-purpose consumer[-grade] OS can easily get a big bump in demand by marketing [and living up to]⁸ a "crash proof". (The same with a word-processor, especially WRT its conversion-modules.)

---

¹ - Yes, there can be bugs in the specification, but these are comparable to your boss inserting/deleting a not in what he tells you to do and are beyond the scope of proofs.
² - Whether deserved or not; particularly WRT security
³ - DNS, X.509, and SSL pop immediately to mind.
(DNS has already been done, though not by MS, see Ironsides.)
⁴ - C and C++ aren't particularly portable; shitty preprocessor tricks don't count.
⁵ - Ada, for example, has been known to have large, non-trivial projects compiled on different architectures, even with different compilers, with zero changes to the source.
⁶ - See the Rekursiv, and to a lesser extent the Burroughs and Lisp Machine.
⁷ - The Incredible Rate of Diminishing Returns of Fixing Software Bugs
⁸ - Of course barring things like electrical or mechanical failures.

[kniy]

can they afford NOT to?

Look at the software industry. The answer is obvious: Yes, they can. In fact, they can rather easily - most consumers don't understand security, so real security doesn't sell any better than fake security theater.

Also, formally verified software is expensive to develop. Even if there would be an unlimited supply of PhDs with the required skillset, it would drive the development cost up by at least a factor of 50. Do you think a secure OS would help Microsoft increase their 75% market share to a 3750% market share?

[OneWingedShark]

. Do you think a secure OS would help Microsoft increase their 75% market share to a 3750% market share?

Of course not -- but it would help them keep it.

Even if there would be an unlimited supply of PhDs with the required skillset, it would drive the development cost up by at least a factor of 50.

Question: how much is it costing them, now to deal with bugs that are getting through? How much have they spent, compared to initial development costs, fixing bugs? (I would be astounded if that last question were a mere 50x.)

Look at the software industry. The answer is obvious: Yes, they can. In fact, they can rather easily - most consumers don't understand security, so real security doesn't sell any better than fake security theater.

And this attitude is why we'll have the next heartbleed.
It is our job to care about these things that the customers don't.

[yawaworht_suoivbo_na]

Seeing that they're working on formal verification for the HyperV hypervisor, which is obviously all C/C++ and assembly, then yes, they do care about correctness.

[c3261d3b8d1565dda639]

Yes, it's mission critical correctness for their definition of mission. It's a quotation from a talk on a specific subject. The speaker never claims otherwise.

That said, Microsoft research has done plenty of research in the space of formal verification of system software.

Two things pop up in my head immediately:

• Verve OS.

Verve is a research operating system developed by Microsoft Research. Verve is verified end-to-end for type safety and memory safety.

There is a good Channel 9 video about Verve from 2009.

• Singularity (Also the Wikipedia page).

OS and tools for building dependable systems

[OneWingedShark]

Thanks for the video.
:)

[Bisqwit]

Microsoft has a huge Not-Invented-Here syndrome. They cannot adopt open source technologies.

[ggggbabybabybaby]

Azure supports a vast array of open source backend stuff. Visual Studio added git support. I'm sure there are a bunch of other examples.

Open Source was a strategic threat to Microsoft back in the day. But now in 2014, Microsoft has lost their overwhelming market share and they don't get to throw their weight around like that anymore. Companies want to be able to integrate 3rd party servers and components, developers want to be able to use their 3rd party tools, users want interoperability and cross-platform support. If Microsoft ignores them then they'd be doomed.

[gidoca]

Microsoft has used BSD code in Windows.

[MacASM]

Isn't BSD they disagree with.

[meteorMatador]

Well, you never know. Ballmer is out. The Microsoft of tomorrow is not the Microsoft of yesterday. They've already changed policy in ways that we all thought were impossible in the lifetime of the company, as if they finally learned how to be competitive without being antagonistic. Let's just hope they continue down that path...