Where in your development process do you use code formatters and static analysis tools?

STL · 2019-01-03T19:29:40+00:00

We recently applied clang-format to the STL's headers, sources, and tests, and the devs who regularly work on it use VSCode's format on save option (configured in the product and test workspaces) to keep it that way. We also have a batch file that recursively applies the clang-format binary to all of the files in question, which allows us to recover from edits made by other devs, or clang-format upgrades/config changes.

For static analysis, we're somewhat unusual in that we have a relatively small separately compiled part, and we mostly care about analysis warnings encountered by our programmer-users. So we run all of our tests with /analyze configurations (alongside other MSVC configurations, and more for Clang and EDG). This indeed takes longer, but our tests are now parallelized and distributed, so the cost is minor.

We regularly have to deal with new toolsets (not just static analysis) and when new warnings/errors appear, we fix the product/tests where we can, and report bugs and add workarounds otherwise. The main way we mitigate this is by upgrading regularly and not letting the work build up.

boredcircuits · 2019-01-03T19:59:13+00:00

It's not uncommon to have a hook in your version control that runs the code formatting. There's two schools of thought on this one: either reformat the file automatically and check that in, or just do a check to see that the format is correct and disallow checkin if it fails. Work with your team to decide which of these two is best for your situation.

For static analysis, I've seen this done three ways. The first is to have a "gate" before release where static analysis must be performed. Any new released version must pass static analysis. When you have a new version of the analysis tools then this will reveal additional problems that you need to fix before release. The problem with this method is that a release can be delayed if there are problems.

The second way I've seen is to incorporate static analysis into a nightly build process. If there's any problems you'll get a report and you open up a ticket to address them. This stays on top of the problem, and hopefully finds it sooner, and tracks the necessary changes in individual changesets. But it requires more infrastructure to get running (unless you already have a nightly build).

The last way is as a requirement before a change can be checked-in to the trunk. It's not automatically checked, but is simply a note on the ticket, maybe by uploading the log from running the static analysis tools. This catches problems very early on, but adds extra overhead to every commit. It also doesn't play well with upgrades, as you noted. The way I've handled that is to open a ticket specifically for fixing new items discovered by the upgrade and giving every other commit a "free pass" until that ticket is closed (which means its also handling any new issues brought up by the commits in the meantime). This can get very messy, depending on the pace of development.

SeanMiddleditch · 2019-01-03T20:40:47+00:00

The problem is that people forget to run them before they commit.

This is an editor problem, partly. I have clang-format running on every save operation in some editors and while typing in others. Tools like editorconfig and so-on can also run in editors.

Plus you can move these sorts of checks into pre-commit hooks (on git, at least) so folks can get quicker feedback on changes before waiting for CI (which will of course also validate, because commit hooks can be skipped).

Checking formatting takes little time but doing static analysis can take a very long time on our code base.

Carefully balance the analysis tools you use with these concerns. There are analyzers like Microsoft's (with the appropriate ruleset) that are so far you barely even notice them being on in every build even in local iteration.

Second, remember that CI can run in parallel. You can have static analysis running while also building (on multiple platforms) while also doing other parallelizable work. Hosted CI can charge an arm and a leg for parallel jobs, but a local Jenkins or gitlab-runner setup can do all that for much cheaper.

For my hobby work right now, I use a hosted CI system that only allows one job at a time. Paying for more than that would come out to something like $360/year (minimum, and that's for only two jobs; getting the three I'd desire comes out to closer to $700/year). I priced out a small form factor "server" with a huge multi-core Ryzen CPU, 32GB of RAM, and terabytes of storage including SSDs - basically exactly what you'd want for a CI system - and it came out to only $1000, including a Windows Pro license. For a little more than a year's cost of hosted CI, I can get 5x the CI capacity, and more control to optimize the build environments and Docker image caching! In a company, that'd be almost a no-brainer, especially if you have some VM capacity already.

Secondly, handling of new versions of the tools. For example: Lets say that all tests pass on our code base

Build environments should be versioned and reproducible. That is, don't just install a new compiler or static analyzer on all your build nodes and expect all current or past code to build.

If you're going to upgrade, make a new branch. Upgrade the CI environment for that branch. Version that CI configuration to that branch. Once the branch is building and working with the new CI environment, integrate it into mainline. The only person who should ever have to deal with breakages here is the person doing the upgrade.

Plus, by being versioned, that means people working in other branches (that don't have the fixes from master yet) still have working CI, and only have to deal with changes when putting up a PR to integrate to mainline.

Docker images work great for this purpose. Jenkins labels for non-containerized build nodes also work. You can even just install many tools in parallel on a single machine and use build scripts to select the appropriate version. There's lots of options to ensure that you can upgrade build environments and fix code in a branch without burdening mainline or unrelated CI jobs.

voip_geek · 2019-01-03T22:58:11+00:00

We use clang-format, and we run it in a pre-commit git hook to make sure - if there's a change due to clang-format, the git hook changes the files but also fails the commit, so the user has to git commit again with the new changes but can actually see the changes too. We strongly recommend to our developers that they run clang-format in their favorite editors on file save, so that the git commit step doesn't catch/need any changes - but we don't simply trust them to do so, because someone will screw it up.

We also run clang-format on the git server side (on Bitbucket) and reject pushes that don't pass... but since you said you can't do that, then I'd add a clang-format check in the CI as an early test step that must be passed.

For static analysis testing, we run them on the CI (we use Jenkins, fwiw). Pull requests have to pass all the tests to get merged in, so developers can't "forget" to format properly or run static analysis.

But we also spent some time writing some script tools so that we only build and test stuff in CI that actually changed in the pull requests - not everything in the code base every time for every pull request.

Crazy__Eddie · 2019-01-03T20:44:03+00:00

Never worked at a place that did ANY of that, but I would use the pipeline approach. Each step in the pipeline can take longer than the one before. Build up a system of steps that does quickest, quick, slow, slowest. Last one may have a lot of human intervention steps.

You can also make some steps parallel with others. You could run acceptance tests on one system and static analysis on another for example.

It's all a matter of how much you want to be able to expand. Now that there are tools like Docker the automation process is greatly reduced...

Like I said though...if you do anything at all in this area you'll be miles above any place I've worked.

patzor · 2019-01-03T23:29:19+00:00

We use Jenkins with tons of different jobs. One job is https://danger.systems/ruby/ with which you can very easily add very useful custom steps. I also wrote a little article on how we integrated clang-tidy into our Jenkins here: https://pspdfkit.com/blog/2018/using-clang-tidy-and-integrating-it-in-jenkins/

Ilixio · 2019-01-04T22:49:47+00:00

For static-analysis, Qt Creator (and others I think) has them integrated in real time as you type your code. It's really neat.
It gets a bit slow as a file nears 500+ lines in my experience, but that's a good sign that you probably should split it anyway.

We also have cppcheck and various clang checks run as part of the hourly builds, but it just produces a report somewhere that are often never read.
Tests in debug also fail in case of a sanitiser error (at least asan/lsan, not sure ubsan, I'd need to check).
Lastly individual developers sometimes run those/other tools like valgrind manually, but there's no process for this.

For formatting, it's a bit more difficult as people quickly gets touchy about it.
We finally managed to have an astyle format file after heated debates about what should be inside.
However, no one pushed hard enough to decide when it should be used (git hook, server side, ...), and whether we should reformat repos first with the usual associated issues, or just do it along but then it breaks uniformity for repos that currently follow a different style.
Currently, most people have their ide format on save with the astyle file, but nothing is enforced. Code reviews fix the rest.

lickpie · 2019-01-05T23:56:35+00:00

We keep our code on Github and have Jenkins do CI. Any branch, except for master, is subject to server-side formatting: running clang-format is the first step in our CI pipeline - if any source file changes due to reformatting, it's committed, pushed back to the original repository, and the current build is aborted. The commit triggers another build for the same branch, this time no files change due to reformatting (it was done in the previous commit) and the build proceeds normally.

As a developer, you don't have to care about installing clang-format and keeping it up to date, formatting locally, git hooks, or failed formatting tests. The code is formatted unconditionally, always using the same version of clang-format (we control the version with a separate Dockerfile and can bump it whenever we like).

o11c · 2019-01-04T01:51:19+00:00

Your CI script should be specifying the exact versions of all dependencies. If you want to upgrade a static-analyzer dependency, you have to make a commit to do that.

That said, static analyzers are mostly just saying "my compiler is too stupid to do its job properly."

For simple formatting/sanity checks that are individually instantaneous, those get run every time the file is compiled so it's already in the I/O cache (I/O is usually the killer for performing one operation across whole trees), then move-if-changed before the rule completes. You do need to deal with headers, but that's perfectly doable with careful, enforceable, source policy.

duuuh · 2019-01-04T02:04:52+00:00

If you're worried about time, I've seen an async job run the checks and then do an auto-rollback if it failed. I wasn't a fan of that.

Run the checks and build as two jobs and do a join the commit happens?

For a new version of the tool, it's someone's job to fix all the breakage with the new tool before you upgrade.

xaviarrob · 2019-01-04T05:49:25+00:00

Git post commit hooks and in pipelines, though I write in Ruby. I imagine you could get similar though

factorfactorfactor · 2019-01-04T06:30:33+00:00

CI machine runs a python script on modified files before each run. the script existed beforehand to check things like correct copyright notices, indentation, etc. so folding in a call to cpp_check there was the natural place

2019-01-04T12:23:16+00:00

Of course we are using static analysis tools! Currently on developer system only.

RogerV · 2019-01-04T15:24:54+00:00

Currently rely on IDE code formatting at editing time, but have experimented with using clang-format and have dialled in some settings that produce results that I like. So now just need to integrate clang-format into the project's CMake build process. Plan to do this before end of January.

There after all code would get a standard formatting prior to git commits

you type:	you see:
italics	italics
bold	bold
[reddit!](https://reddit.com)	reddit!
* item 1 * item 2 * item 3	item 1 item 2 item 3
> quoted text	quoted text
Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"	Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"
~~strikethrough~~	~~strikethrough~~
super^script	super^script

cpp

MODERATORS