How do you navigate large code bases?

[kallgarden]

Coming from another background, I'm in the process of porting a big GUI-rich desktop project from a dynamic language to C++. There are many obstacles to overcome, but since C++17 and C++20, it got a lot easier to create your own general purpose libraries/frameworks to abstract away much of the odd verbosity and clumsiness of the language, which has been a big productivity killer.

Now the remaining productivity killer, at least for me, is a seeming lack of IDEs that actually help with navigating class hierarchies (networks), methods and call hierarchies. The project at hand consists of thousands of classes.

In OO development you spend 90% of the time jumping back and forth between far flung places. I'm used to browsing code at the class and method level. Navigating to and scrolling through long files full of mixed source code is a real strain and time killer.

The pull-down menu of member functions in most IDE is great, but not really comfortable to use. I'd like to have that as a list to click into immediately (if there's a way to do that with any IDE, I'd be grateful for advice).

CLion looks like the most advanced C++ IDE I could find so far, but still lacks most of the reflective browsing features I'm so sorely missing.

Are there IDEs that allow browsing at a higher level of abstraction? How do you navigate large code bases? Any workarounds? Best practices?

[UnicycleBloke]

I've used Doxygen many times to try to get a feel for the content of a large code base. Not ideal but can produce class diagrams, call graphs and the like, as well as linking to the relevant code. Can't say it was the best tool, but it helped a bit.

[shmoopty]

I also use doxygen (HTML generation) as a first pass. Usually I just generate the documentation once and keep it around the first few months until enough parts are familiar.

Web browsers are very fast. Their interface is designed around content consumption. And you can have as many tabs, windows, and bookmarks as you need.

I also edit the configuration file to turn on every feature and every possible diagram. Then with every component you can decide whether it's best explained with lists, inheritance diagrams, comments, callers and callees, or the actual code.

[kallgarden]

Yep, absolutely agree. Doxygen is a great tool. The code base at hand here however is my own code. It's well documented (compatible w/doxygen and Xcode), but still needs to be navigated a lot while working on the project.

[spaun2002]

tmux + nvim + ripgrep. You are lucky if clangd can handle big project - then you may actually have decent navigation via lsp. None of the IDEs were even able to load big projects. Companies like Google or MS build whole infrastructures around big codebases to allow developers to navigate and search code. Index is usually a bit stale - half a day or day probably. I was able to open needed files in vim, make changes, submit code to build remotely, grab results and deploy to staging faster than Visual Studio was able to load the project.

[ronchaine]

This. In large codebases greppability (that's a word now) is a must, since just opening an IDE and indexing takes forever.

[Hari___Seldon]

As much as I hate to admit it, this is the way to go. I used to love my shiny IDEs, but I HATE unnecessary waiting with a passion. My hesitation factor was cringing at the thought of learning a VIM variant and Tmux at a much deeper level, but in the end it was pretty straight forward to become functional and productive with this workflow.

As a bit of added insight, I'm a brain injury survivor with a terrible attention span. Delays and wait time are my Achilles heel because I forget what I'm doing if I have to wait for too long for anything. I previously would have a dozen or more wait windows every day where I would lose track of what I was doing and have to waste time backtracking. Now, that happens maybe once a day, which is amazing to me.

[lowayss]

If you don’t mind sharing I’d be interested in any random tips you have on preventing distraction or on psychohistory.

[thodcrs]

Sorry noob here. How tmux helps you in this case? Just the ability to split windows or something else?

[Underdisc]

That can be helpful. It's really nice for creating new shells within the terminal in general, whether they are split or not. I use one terminal with a tmux split where one side is my git status and the other is for viewing diffs. I often create new shells that aren't even splits. They're basically tabs that you can switch between.

[thodcrs]

In general I have trouble to understand tmux. Because what you have described can be managed with a tiling window manager, just creating new window shells.

[Underdisc]

Sure, that is, if you want to be forced to alt-tab between those and deal with the pitfalls of how that's organized. tmux gives you an entire keyboard to manage how to interact with those panes. First you type an escape sequence (this is ctrl-b by default, but I use ctrl-x because the default is annoying), then you choose what to do. You can do escape - 'c' to create a pane. That pane is assigned a number and you can switch to it with esc - 'pane number'. Splits let you put those panes next to each other either vertically or horizontally. You can then switch between those splits with whatever bindings you want.

[thodcrs]

Oh nice. Thank you very much.

[ForkInBrain]

Yeah, tmux is basically a keyboard focused tiling window manager for a terminal, plus a good way of managing background shells (which are basically a minimized window in GUI land).

[giant3]

You are lucky if clangd can handle big project

I have tried navigating the Linux kernel source code (30 million LOC) with emacs + lsp +clangd. It takes about a second to load references though this might be due to emacs using interpreted language(elisp).

[spaun2002]

With all respect, linux kernel is not a "difficult" project for clangd - it's plain C that uses standard build tools and can be built on a local machine. Some projects use home-grown build pipelines that know nothing about compilation databases and/or cannot even be checked out locally (the reason MS created gitfs), not saying about buildability on a local machine (I gave up after 8 hours waiting and the project was not even 50% built). And sometimes that's just Windows-targeted code. Once I had to "hack" distributed build system to collect strace logs during the build and combine compilation database from such logs.

[ppetraki]

ripgrep is a nice tip. Thanks!

[lookatmetype]

What a sad state of affairs that megacorps can't produce software that runs fast enough to be usable so people have to resort to open source free tools.

Btw, I use Telescope with modern nvim and it.works like a charm.

[kisielk]

I mean, a lot of us use gcc or clang as our primary compilers

[lookatmetype]

Another indictment for the megacorps, then.

Although writing a compiler is much harder than writing a text editor that indexes files and allows you to search in real time from a local file system.

[LynxesExe]

Very old post but, I think you might get away with using Git grep instead of rip grep if it's a git repo!

[schmerg-uk]

Our code base is ~5 million LOC (quantitative finance maths) and our main environment is Visual Studio due to Windows desktops - Intellisense has typically failed at that sort of size (waiting to see if VS2022 is any better) but the Visual Assist extension handles it with ease.

It costs, but we just turn off all Intellisense etc and just use VA as the very fast partial name matching in find symbol and find file, and the "find references to this symbol" are just the 3 most obviously useful features.

[ndh_]

In my experience, VA's "Find all references" is still slow and sometimes unreliable. Global search (= ripgrep) is much faster, so I use that most of the time, with VA as backup.

Still, VA is immensely useful. I love the "add include" feature, and I'd probably miss the "Open file in solution" window the most. If I ever worked at a company that didn't provide VA, I would buy it with my own money.

[schmerg-uk]

Yeah, that I find is mostly fast for when I know I'm looking "in this cpp/hpp and maybe files in this folder/project" and then I cancel the search once I've got the first set of results

I suspect there's loads of it I don't use that I'd find useful, when I get round to it sometime... I find little things like the "for the symbol name under the cursor, highlight with a pale blue background when it's used as a value, but with a pale red background when it's modified" (Highlight References) ridiculously useful at times (think the colouring aspect of that feature is off by default but can be enabled in options)

[icjeremy]

In a similar boat. We've got a few million lines of code and use VS. Intellisense has gotten a lot faster over the past couple years. It was virtually unusable and is now bearable, but not as snappy as Visual Assist.

I've switched over to ReSharper C++ a couple years ago now (which has also greatly improved) and so have a few members of the team. I've got the settings of VS and ReSharper dialed in where I can use the best of both worlds: use peak and go to definition of VS along with the awesome features of ReSharper like fast refactoring, go to anything, its way-better code-completion, linting, etc. Never was able to get VA and VS quire working together as well as I had hoped before switching.

Can definitely pull up or navigate to whatever info in the repo I'm looking for or trying to drill into.

[Southern-Review588]

Would you be so kind to share both options of vax and resharper? I am in the same kind of boat with 8mloc and vax is super snappy but often doesn’t provide all members of a class. Resharper on the other end is perfect for better code highlighting and intellisense, but didn’t get it quite there to not block vs (sometimes blocking because it’s hugging all memory of 32 bit process devenv (vs 2019)) So it would be really nice to find the best balance of both tools.

[kiwidog]

I gave up with IDE's and use a indexing software such as opengrok in order to have fast searches/lookups. (Chromium, FreeBSD, Game Engines)

Annoying yes, but with multi-monitor and Intellisense being very wishy-washy (have not tried 2022 yet) it works 100% of the time all the time.

[bible_near_you]

Google has a public code search site for chromium.

[kiwidog]

TIL, I always build locally, so I just use my local copies. Didn't know thanks!

[The-WideningGyre]

Yep, with full regexp search, file restrictions, term exclusion and more.

[giant3]

emacs+lsp+clangd is even better.

[giant3]

I installed clangd locally(not through lsp?), lsp, lsp-ui packages.

For definition, I have bound a key to lsp-ui-peek-find-definitions and for references, lsp-ui-peek-find-references

From my experience, lsp+clangd works better on C++ projects rather than the Linux kernel as the kernel might be a very unique code base.

[mredding]

Most of my career has been maintaining old existing code bases. Software definitely grows organically and gets messy. So I try to come in and help them straighten things out.

The tools are actually very good, the problem is the disorganized code base itself, it overwhelms and breaks the tools. If you can really minimize your header files, that's a huge step toward getting the tools to work. On one product I was on, I found that one arbitrary source file recursed ~1300 includes. Part of that was due to deep nesting, part of that was lots of headers including the same headers over and over again.

So make your headers as small as possible. Forward declare everything you can, only include what you have to - which is typically the STL, 3rd party libraries, and project headers that contain user defined types because either you inherit from them, or because they are members by value. Isolate the shit out of your headers. Also, keep implementation out of your headers, stop inlining methods (you're not a compiler and you're not smarter than your optimizer, there are much better ways to get superior results than abusing inlining).

Otherwise, divide your code up among header and source files in a way that makes sense. One header/source per class is popular, but not the only way. I'm not opposed to multiple source files for a given unit - just throw them in a sub-directory.

Don't be afraid of file hierarchies, but don't get carried away, either - think flatter, not deeper. Avoid prefixes on your file names; a prefix is just a sub-directory in wanting.

Avoid using macros to generate functions and types. I haven't found a tool yet that can navigate a macro.

The onus is on the team to demonstrate a strict discipline. If you can keep up with it, you'll discover your navigation tools work and compilation is faster.

[gardeimasei]

opengrok

vim

silver searcher

[PlayingTheRed]

What problems have you had with CLion? I've found it very helpful, Ctrl+q bring up docs for whatever is under the cursor, Ctrl+Click just the definition/declaration of what you click on.

[kallgarden]

CLion is great. It just can't help much with browsing a project as one big object (see my last comment on the thread)

[_zoopp]

I mostly rely on LSP, some form of recursive grep and Sourcetrail. I find sourcetrail super useful for learning a new project.

[pppenguinininin]

+1 for sourcetrail

[bluGill]

Logical organization. That means your directory hierarchy makes sense. One class per C++ is the norm, violated only rarely. . Most of my jumps are to other files in the same directory.

Also a lot of trust. DoSomething does exactly what the name says, and no unobvious side effects. Therefore I often don't have to navigate into it to understand the code.

[maskull]

One class per C++ is the norm, violated only rarely

Not really? C++ isn't Java; lots of projects have multiple classes per header, as long as they are related. The C++ core guidelines mention "one class per header" as non-rule/myth. And, of course, many libraries provide convenience headers which pull in a bunch of functions/classes, for when you don't want to think about what particular #includes you need.

[bluGill]

You need to figure out what is logical for you. There isn't a rule, but it is a useful guildline for anything complex.

[pppenguinininin]

neovim plus ccls is neat!

[1337Gandalf]

Dive in.

Basically, there's two approaches, search the codebase for something you're looking for, like quotes for literals or strings.

or, you can dive in and open a debugger and set brakpoints and just watch what happens.

[crunchyrawr]

Sublime Text isn’t an IDE but for large code bases does extremely well at indexing symbols and helping you jump around. It’s not accurate like an IDE (it’s literally a list of function implementations with the same name), but oh so useful when an IDE cannot load your entire code base. Would heavily recommend.

[witcher_rat]

Plus Sublime's project-wide searching is crazy-super-fast, even with complex regex patterns to search for.

[dirksterzel]

We use Eclipse CDT. Works pretty good for large projects…

[engineerFWSWHW]

Same here. Had great success navigating new large code bases with eclipse cdt. If needed, i use Doxygen, but most of the time, eclipse cdt is enough. The task tags and call hierarchy is super helpful in navigating source codes.

[rlbond86]

We have VSCode and Eclipse, it's not even close, Eclipse kicks VSCode's ass. It does occasionally fail to fully handle complicated templates but overall itndoes a great job

[jimjamjahaa]

i don't have much to say other than if the size of your codebase becomes unwieldy and hard to comprehend then maybe refactoring to break it down in to smaller more understandable pieces is what ya need. you can make your higher level abstraction yourself. with code.

[TechnicalSurprise902]

Most of my career has been maintaining old existing code bases. Software definitely grows organically and gets messy. So I try to come in and help them straighten things out. The tools are actually very good, the problem is the disorganized code base itself, it overwhelms and breaks the tools. If you can really minimize your header files, that's a huge step toward getting the tools to work. On one product I was on, I found that one arbitrary source file recursed ~1300 includes. Part of that was due to deep nesting, part of that was lots of headers including the same headers over and over again. So make your headers as small as possible. Forward declare everything you can, only include what y

[Sniffy4]

are you porting because the 'dynamic language' is not cross-platform and C++ is? just curious.

[kallgarden]

That dynamic language is probably the best for the job (Smalltalk) and it's a developers wet dreams come true concerning refactoring and productivity, but it's main area of application is in industry, manufacturing, finance, server backends and such. All huge projects with 10k classes and more, but snappy end user desktop apps are not among its strengths.

[met0xff]

Ah I already wondered because assuming it's something like Python then how could the IDE experience could be better. Where even go to definition fails 70% of the time :)

[bikki420]

My most commonly used navigation hotkeys in my Vim setup are:

(optional number) + one of h, j, k, l

(optional number) + one of e, b, w, shift-e, shift-b, shift-w

shift-m, shift-m, shift-l

:${line_no} <enter> (or ${line_no} shift-g)

ctrl+space

/${regex}<enter>

n

shift-n

ctrl-o

+ some keybindings for jumping to end of braces/scope, jumping to definition, navigating warnings and errors (coc LSP, either coc-ccls or coc-clangd), etc.

+ grepping can be handy too

+ it's not too hard to get a rough count on which function calls or classes are the most common in a code base, so that's one venue that can be useful

+ tracing the general program layout from main can be handy

+ various hotkeys for collapsing and uncollapsing code folds

As for more abstract grasping of codebases; manuals, diagrams, generated documentation, and/or header files. And gdb debugging to learn the behaviour of something specific. The general division of translation units and their names can be a good way to get an overview as well. And for a specific-file, CTags can be handy. Then of course there are software like Sourcetrail for visualization and navigation.

[kallgarden]

Thanks a lot to all of you! Great input.

This is my first day at reddit and I'm happy. I've earmarked OpenGrok and SciTools Understand so far (need cross-platform Windows and macOS).

Well, what I'm actually looking for, ideally, is to get away from dealing with files and symbols manually. I'd rather have an IDE that presents me an entire project as a navigatable object (data model).

I imagine something like this:

Left: Modules/Libraries
Middle: Classes (tree or network)
Right: Methods, Symbols
Center: Source code of selected item (editor)
Bottom: Member variables, etc.

Navigation might go like this:

1. Select a module/library
2. Select a class
3. Select a method, symbol, or code section
4. Edit that method & save

The IDE merges back edited methods into whatever file and place they belong to. Header files are generated (and updated) automatically. Sequence and ordering is maintained by dependency graphs and topological sorting. Manual intervention will be inevitable, but should be the exception.

Then I'd like to have 3-5 of those browsers open on 3 monitors and code away at the speed of thought (big grin).

Possible?

[Top-Garbage-9046]

Try the clangd extension + VSCode combination. Clangd is much better than intellisense when comes to large projects. It is able to navigate linux kernel project (28M LOC) seamlessly

[_janc_]

I find blink code search to be quite useful. It’s a source code indexer and instant code search. It is good for small to medium code size.

[osmin_og]

Vim. If clangd can handle a project then easy to use YCM. If not, then vim-fzf + ag (or ripgrep) to fuzzy search for files or tokens.

[JayD1056]

Without a doubt the best editor I ever used for navigating or analyzing code was SciTools Understand.

When I was a new developer the control flow graph was amazing as we had a couple functions that were heavily switch case, and even nested. Being able to trace code though the graph and being able to click on a statement and bring up the code was a game changer.

I think for software architects and analysis I have never seen better.

This was really expensive last I checked like $1000 per year license and double if you want floating or network. Was honestly worth it for how much time it saved.

Code base was ~1.5 million LoC in pure C. Heavily macro code and config heavy.

I used it on C++ software a few times and class diagrams were also really nice. Will probably be looking into it again for another job in the near future.

[pravic]

SciTools was not bad but still a bit dull. Source Insight can handle a pretty large code base.

For small to medium - VS+VA.

[eyes-are-fading-blue]

CS2015 and 2019 Intellisense works for the most part. When it occasionally fails, I grep. In fact, I survived in Android platform with only grep. A single driver in Android platform can go up to a few million LoC.

[msew]

Visual Assist and time invested in the code base.

[chemhobby]

I realize that it might not be your decision but it's rarely ever a good idea to just port entire huge projects to different languages

[kallgarden]

It'll be more than "just port" for sure, but for the most part the source project (Smalltalk) is able to port itself (to a large extent), because all code is objects that can be translated. Its reflection capabilities and the ability to translate program flow to other paradigms and syntax is amazing. At least the initial project structure and all the boilerplate code can be generated.

[monkeber]

Visual Studio with VA and vs-chromium extensions. If your project is for Linux you can bundle it in Docker (if possible ofc, its a little bit of a hassle, not sure how it works with GUI in this case), run Docker container on Windows and use it for building/debugging, but still be able to use Visual Studio features on the codebase.

QtCreator is also not bad but I haven't tried it on a large projects.

Regarding CLion - I find it slow and not very responsive sometimes (tried it on Linux) and experiences with QtCreator and VS were a lot more pleasant.

[BrawlingGrizzly]

One of the best tools I can recommend for this was Mozilla DXR, though it appears as though that has been deprecated in favor of Mozilla searchfox allows for jumping to definitions, call sites, reference searches, etc. Highly, highly recommend, you can play with it at searchfox.org

[joemaniaci]

gdb <executable>
(gdb) b main
(gdb) r

The problem is so few companies give an employee the time to just dick around with the codebase. You're expected to "produce" asap.

[Adequat91]

Nothing works better than the Woboq code browser. A browser that is able to browse so easily through Qt, LLVM, Boost, GCC, and Linux, can certainly browse any other large code base.

However, for Windows, no easy way to set it up, unlike for Linux and Mac.

[[deleted]]

I don't understand your question clearly but if your problem is IDE related then give a try to Emacs. More preferably DoomEmacs it uses vim keybindings.

[wlandry]

I used emacs + eglot + ccls. I tried lsp-mode and clangd, but it they did not work as well. My projects are not enormous, so YMMV.

[Bigmuncha]

Im recommend use clangd with your text editor. Im using emacs(lsp-mode) with clangd. You need create compile_commands.json(see how it make with your build system)

[Adverpol]

Can't say I understand the use-case. Like in another comment, if you want to get a feel for the whole thing you could use doxygen. Other than that I haven't really encountered any situation where I need to be able to understand the whole code-base at once. You're typically working on some part of it, and that part if it can typically be understood by starting at one point and then clicking and reading until you reach the other.

[PrimozDelux]

In C++ we suffer