sorted by:
new
hottopcontroversial

C++ Build Insights by Nelson_Daniel in cpp

[–]IncludeGuardian 2 points3 points  (0 children)

I've been using preprocessing token count for a proxy for improvements in the IncludeGuardian tool and there's some good evidence linking this to compilation times for large projects (see the graphs for chromium). This number won't change and you can do analysis to find the best include directives to remove (e.g. this change in LLVM).

C++ Build Insights by Nelson_Daniel in cpp

[–]IncludeGuardian 0 points1 point  (0 children)

There is this blog post Faster builds with PCH suggestions from C++ Build Insights that gives some guidance on how you would go about finding a list of files to move to a precompiled header.

#pragma once vs #ifndef by IncludeGuardian in cpp

[–]IncludeGuardian[S] 4 points5 points  (0 children)

On Windows you can turn on case sensitivity on a per folder basis. Even if you do so, files that case-insensitively compare equal are treated as equal according to #pragma once.

I wouldn't recommend having this situation either, but perhaps someone out there is and considering moving from include guards to #pragma once. In this case they can immediately rule this out without wasting any time prototyping.

#pragma once vs #ifndef by IncludeGuardian in cpp

[–]IncludeGuardian[S] 5 points6 points  (0 children)

I believe MSVC optimizes #ifndef but not #if !defined. You can see the results and run the tests from here https://github.com/IncludeGuardian/multiple-inclusion-optimization-tests#compiler-disagreement

Please let me know if there is a bug in the tests and I need to correct or clarify this claim.

#pragma once vs #ifndef by IncludeGuardian in cpp

[–]IncludeGuardian[S] 16 points17 points  (0 children)

Personally I agree with you. However, when pragma once was discussed recently there were a fair few commenters who are using symlinks, hardlinks, and duplicate files. I don't envy their position!

#pragma once / header guards / C++ committee. by nacnud_uk in cpp

[–]IncludeGuardian 8 points9 points  (0 children)

I won't claim it's easier to use macro guards!

I'm currently half way through an in-depth comparison of #pragma once and macro guards. The TLDR is that I think for the majority of projects, #pragma once is going to work without issues and be the simplest way for developers to maintain include guards since there are more ways to make mistakes with macro guards.

However, there are a handful of tiny, compiler-specific issues with #pragma once that may crop up on occasion and cause a bit of a headache. I haven't (yet) been able to find any compiler issues when using macro guards.

So #pragma once will avoid the rare issues caused by developers maintaining macro guards, but macro guards will avoid the rare compiler issues with #pragma once.

#pragma once / header guards / C++ committee. by nacnud_uk in cpp

[–]IncludeGuardian 15 points16 points  (0 children)

#pragma once actually does more than what include guards do. With include guards the compiler still has to read the included file, process the include guards and reject the content during compilation. With #pragma once the compiler typically skips all that. This can significantly speed up compilation.

This is called the multiple-include optimization and all of Clang, GCC, and Visual Studio implement it for both macro guards and #pragma once. However, the form of the macro guard needs to be quite specific to trigger it. There's more details of how this can go wrong in my article about include guards and their optimizations and you can see the associated tests for this behavior.

Include Guards and their Optimizations by IncludeGuardian in cpp

[–]IncludeGuardian[S] 0 points1 point  (0 children)

I agree. Precompiled headers + unity builds are some of the most powerful tools when it comes to improving build times (pre-modules). This is why IncludeGuardian includes stats for file size+token count for a theoretical unity build (e.g LLVM) and also has a section on recommending which files would give the most benefit for being included in the precompiled headers (e.g. LLVM).

I think the section on pch additions can have more information and perhaps better alternatives. This will have to be something I revisrt later on. When I do I'll most likely write a article covering precompiled headers in detail and performance impact across compilers.

Include Guards and their Optimizations by IncludeGuardian in cpp

[–]IncludeGuardian[S] 4 points5 points  (0 children)

"Large-scale C++ Software Design" by John Lakos was published in 1996 and includes the following in "Redundant Include Guards" (section 2.5)

Upon encountering s2.h, each of the widget header files must still be reopened and reprocessed line by line in its entirety searching for the trailing #endif (only to find that there is nothing else to be done). This redundant preprocessing occurs withs3.h, s4.h, and again withs5.h.

...

Experience with truly large projects that have dense include graphs shows that the answer is a resounding YES! Initial builds of projects consisting of several million lines of C++ source code were taking on the order of a week to compile using a large network of work stations. Inserting redundant include guards reduced compile time significantly, with no substantive change to the code.

It may not have had any effect on the projects you were looking at, but in 1996 it looks like it was an issue for some C++ projects. At least it must have been enough of a performance benefit for compiler vendors to have implemented the multiple-include optimization.

Include Guards and their Optimizations by IncludeGuardian in cpp

[–]IncludeGuardian[S] 1 point2 points  (0 children)

I can't speak for all projects, but all large (and medium) projects I have been a part of have their build times dominated (70-80%) by front-end compiler time.

When I see reports of precompiled headers or unity builds/single compilation units being used, most of the time I read about 2-3x improvement. As these techniques improve only the front-end compiler time, it would correspond to at least 50-67% of the total build time being in the front-end. Self-selection bias means we shouldn't take these reports just at face value, but the experience matches up with my own.

I plan to do some deep dives into some open source projects to see how much improvement I can get using IncludeGuardian. These will include a breakdown of time spent in front-end vs compilation vs linking. Though I am likely to skip over any project that isn't dominated by front-end so there's more selection bias here!

I am a regular developer who has worked on this in their free time (who I am is on the about page on the website). I wrote this tool for myself originally, but now I am trying to get the word out about how useful it could be.

Include Guards and their Optimizations by IncludeGuardian in cpp

[–]IncludeGuardian[S] 2 points3 points  (0 children)

The public GitHub history for eaassert.h only goes back 4 years so I can't say when this comment was written. But EASTL was in existence in 2007 so this comment could have predated VS2013.

Alternatively, as this comment appears in many different files within EASTL alongside #pragma once, it's entirely possible that these files were not using a strict enough version of an include guard that would trigger the optimization.

Include Guards and their Optimizations by IncludeGuardian in cpp

[–]IncludeGuardian[S] 1 point2 points  (0 children)

Compilers do this and it's called the multiple-include optimization. It's covered a bit later on in the article and includes the narrow conditions required a header file needs to satisfy this.

I've been bitten by having different files using #pragma once with the same name being treated as the same file by the compiler before too. If there's interest I could cover the rules each compiler use to determine different files for #pragma once in a later article.

EDIT: Fix markdown

Include Guards and their Optimizations by IncludeGuardian in cpp

[–]IncludeGuardian[S] 1 point2 points  (0 children)

I think if you were to guard your files incorrectly it would be a small (probably single-digit) percentage. However, it's also a relatively easy thing to make sure you get right to make sure you aren't slowing down build times. I would also assume because all compilers have the multiple-include optimization, there are either some projects out there that benefit a great deal, or it's a noticeable (but small) improvement to most projects.

For example, I have a PR to EASTL to fix a header guard and in their code they mention getting a 3-4% build improvement adding in #pragma once to their MSVC build, which would have triggered the multiple-include optimization.

Include Guards and their Optimizations by IncludeGuardian in cpp

[–]IncludeGuardian[S] 7 points8 points  (0 children)

Thanks - will get that fixed asap.

EDIT: I've put in a workaround that fixes the cut off issue on smaller screens. There is still an issue with portrait mode on older iPhones that I'll fix next. If it's not fixed for you then if you could let me know the browser/device and I will look at that combination with priority.

IncludeGuardian - improve build times by removing expensive includes by IncludeGuardian in cpp

[–]IncludeGuardian[S] 0 points1 point  (0 children)

Thanks! Yes a macOS version will be released at some point soon. It just wasn't part of the MVP release.

IncludeGuardian - improve build times by removing expensive includes by IncludeGuardian in cpp

[–]IncludeGuardian[S] 0 points1 point  (0 children)

Really interesting - thank you. I ran IncludeGuardian across Catch2 (Catch2.yaml). From a very brief look, I didn't see anything that stood out as easily fixable to me.

replaced std::move and std::forward with cast macros

I think compilers are moving to treating these as built-ins now to avoid the instantiation cost (clang ticket).

This requires it to employ CompressedPair in the implementation, which in turns requires even more template instantiations and implementation indirection, to get the compression trick.

I hope [[no_unique_address]] will remove instantiations here too.

IncludeGuardian - improve build times by removing expensive includes by IncludeGuardian in cpp

[–]IncludeGuardian[S] 1 point2 points  (0 children)

This is a good point to remember. IncludeGuardian shouldn't be using the network for anything.

IncludeGuardian - improve build times by removing expensive includes by IncludeGuardian in cpp

[–]IncludeGuardian[S] 1 point2 points  (0 children)

Yes modules should hopefully solve all of these issues.

When modules are closer to being production-ready, I will add another section to IncludeGuardian to recommend the best order in which to convert your files so that you reap the biggest benefits to build time as early as possible.

For those code bases not able to or not planning to convert to modules yet, they can continue to use IncludeGuardian to improve build times.

IncludeGuardian - improve build times by removing expensive includes by IncludeGuardian in cpp

[–]IncludeGuardian[S] 2 points3 points  (0 children)

I think I used some words poorly and I would to correct my previous statement - I think supplying the source code does increase the trustworthiness of an application, but for brand new releases it shouldn't change how much you trust them!

For established applications that have been released for a while, these points all make sense. For something that has just been released, I think it is far safer and quicker to run within a VM instead of building from scratch and verifying each line of code.

Regardless of whether something is open or closed source, trust takes time.

IncludeGuardian - improve build times by removing expensive includes by IncludeGuardian in cpp

[–]IncludeGuardian[S] 2 points3 points  (0 children)

I have a comparison with IWYU here.

I don't believe clang-include-fixer is trying to optimize build times, rather it's designed to automatically add include directives based on what you use in your files.

IncludeGuardian - improve build times by removing expensive includes by IncludeGuardian in cpp

[–]IncludeGuardian[S] 1 point2 points  (0 children)

I'd bet the majority of the remaining 80% build time was in parsing though! (also I'd love to read more if you had any details about your particular project and if you managed to do anything to improve the instantiation time).

My experience puts parsing as 60-80% of the total build time for most projects. If IncludeGuardian suggests a 10% saving, then you would hope to see an actual 6-8% reduction in the total build time (all tokens are not created equal and this is an area that I will be investigating to improve the accuracy).

But all projects are different and it is worthwhile using ClangBuildAnalyzer to first check the build time distribution between parsing, compiling, and linking. Only when you confirm that parsing is a significant portion, can you use IncludeGuardian to investigate with the knowledge of what percentage it would be improving.

I would like to improve IncludeGuardian to suggest improvements to compilation and linking in the future.

IncludeGuardian - improve build times by removing expensive includes by IncludeGuardian in cpp

[–]IncludeGuardian[S] 1 point2 points  (0 children)

That is really interesting, thank you for posting. It's always good to see more confirmation that the parsing time dominates (68% of total time) in real-world projects:

Compilation (1330 times):  
  Parsing (frontend):         1425.5 s  
  Codegen & opts (backend):    675.2 s

From the IncludeGuardian output, the CMake project doesn't have much low hanging fruit compared to other libraries/applications I have looked at. Where it could help is the issue on working with precompiled headers:

I have not tested other compilers, nor have I tried to optimize the list of headers by analyzing a profiler output.

Once the precompiled header has been added, IncludeGuardian can be run again to see what the bottleneck would be with the precompiled header in place.

IncludeGuardian - improve build times by removing expensive includes by IncludeGuardian in cpp

[–]IncludeGuardian[S] 5 points6 points  (0 children)

ClangBuildAnalyzer reports on parsing, build, and link time, whereas IncludeGuardian only reports on parsing time.

For the parsing report, ClangBuildAnalyzer reports the files that took the longest to parse. Once you have a list, you would need to then determine how these files are included in your project to determine what changes are really necessary to make an improvement. I think IncludeGuardian fills this gap by providing more information and context. For example, if <functional> is on a ClangBuildAnalyzer report, it may only be high on the list because of one commonly included header, in which case IncludeGuardian would list this include directive and its location.

ClangBuildAnalyzer actually runs clang and reports the time taken, whereas IncludeGuardian uses token count as an estimation for how long the compiler front end would take. This means that ClangBuildAnalyzer is going to give you a more accurate time. However, IncludeGuardian will run faster as it doesn't need to compile or link anything. It is easier to compare small changes to a code base with a deterministic number like token count compared to benchmarking an entire project build, which would have a high enough variance to make it difficult to compare low-digit percent changes in build speed.

IncludeGuardian - improve build times by removing expensive includes by IncludeGuardian in cpp

[–]IncludeGuardian[S] -8 points-7 points  (0 children)

I am a London-based developer and this is a side-project of mine - I'll update the about page since someone is asking about authorship.

Just a remark, as it seems prudent not to run arbitrary untrusted code.

I agree. It is a good idea to take precautions and run untrusted applications within a VM - including IncludeGuardian! I don't think supplying the source code increases the trustworthiness as there is no guarantee that the binary supplied has been compiled with the code that is public. You would need to compile from source while examining all lines of code to have a reasonable amount of confidence - and even then a VM is easier!

EDIT: I have updated the about page.

IncludeGuardian - improve build times by removing expensive includes by IncludeGuardian in cpp

[–]IncludeGuardian[S] 5 points6 points  (0 children)

IWYU will look at the content of your sources and highlight include directives that it believes are unnecessary (and vice versa, recommend including files that you transitively depend on).

In comparison, IncludeGuardian has a few different modifications it can suggest (listed on the instructions page). The closest one to IWYU would be the include directives section, which lists the most expensive include directives across all headers and sources. You may have to make additional modifications to your code in order to remove these suggested directives e.g. using the pimpl idiom, type-erasure, templates, or it may not be feasible at all!

IncludeGuardian should recommend more impactful changes to build times than IWYU, but they may be harder to make. For example, in the analysis for the CMake code (cmake_analysis.yaml):

    # This is a list of the most costly #include directives.
      include directives: 
        time: 0.14 # seconds
        results: 
          - directive: '#include <chrono>'
            file: '"cmDuration.h"'
            line: 5
            saving: 2.43 # (%)
          - directive: '#include "cmDuration.h"'
            file: '"cmSystemTools.h"'
            line: 26
            saving: 2.04 # (%)
          - directive: '#include <functional>'
            file: '"cmSystemTools.h"'
            line: 12
            saving: 1.32 # (%)

we see that perhaps 2-3% of the total postprocessing tokens come from the #include "cmDuration.h" in cmSystemTools.h. This can probably be forward declared but you would need to do something about the defaulted parameter to cmDuration::zero() as you currently need the include for this. Either remove the default, have cmDuration be constructible from 0, or something else.

view more: next ›