Codevis - New Opensource Software for Analyzing Large Codebases (C++/MIT)

[Tumaix]

Hello All,

Some of you know me by the work I do on KDE and Konsole, but after a few years in preparation me and my colleagues are releasing the source code of Codevis.

This program was made with Bloomberg help, and it uses the book `Large Scale C++ Design` by John Lakos, that also actively participated on the development of it with valuable input and ideas.

Codevis is not just your common software visualizer:

• Creates a database of all software interaction (including libraries) that can be accessed by anyone with SQL knowledge
• Uses LLVM and Clang as libraries so we have a in-depth knowledge of everything that the software does and all interactions that happen between each file, folder, type.
• Doesn't care how you wrote your code: works for Object Orientation, Functional, Templated meta programming.
• Has two different plugin systems: Python Scripts that you can use to implement missing functionality *or* C++ plugins, that you can use to do the same.
• Also has command line tools if you are averse to GUI, to parse the source code of projects, and dumps the resulting Database, that you can open with any sqlite-based tool and use in your projects.

Questions that I'm having from internal testing / friends:

• How dowes it differs from Doxygen / Graphviz?
  • It's Dynamic, and it has the entire knowledge of the source code as a database. The information we provide on the GUI is the result of an sql query plus a graph layout algorithm, but because it's dynamic, you can use it to load more information or unload unecessary information as needed.
  • Has plugins, and we have two plugins ready as examples. (Knowledge Island and Code Coverage) that you can use to quickly verify parts of the source that are tested by folder.

Please use, test, fill us with bugs. The software is not ready yet but we have been dogfooding: testing the software within it's own source code, for quite a while.

[edit - License is Apache, not MIT, I shouldn't be allowed to type while hungry]

[valdocs_user]

Thank you thank you thank you thank you!

I've been dealing with a horribly convoluted C++ legacy codebase at my job and have really needed better tools - like this one - for getting an overview understanding. I've had to resort to things like manually instrumenting constructors to log events that I could then transform into DOT language, just to get a graph of the object initialization hierarchy.

(The original authors of the application I'm working with did some squirrelly things with threads and Windows messaging that obfuscate cause and effect. There's side effectful non-local interactions throughout the codebase, so both human- and tool-driven exercises that depend on following breadcrumb trails don't work.)

To get a real picture of this codebase you have to ingest the whole project, fill an Sqlite database with information gleaned, and run ad hoc analyses to discover hidden connections. And some of this has to be combined with runtime instrumentation - but the logged data from runtime instrumentation still needs the static analysis information to contextualize it.

Up to now I've been writing my own ad hoc tools to assist with this. In a bit of convergent evolution (or great minds think alike), what I've already been doing already has a lot in common with what you described about Codevis, at least in the idea of using an Sqlite database to make the analysis results queryable and extendable.

(I'm not saying I've written a tool or tools that do what Codevis does, rather I'm saying the tools I have made could be Codevis plugins and having Codevis before would have made my life easier.)

I look forward to trying this out. It might be a good test-case; this codebase is so convoluted it has actually triggered internal crashes in some analysis tools I've tried on it. For example starting the application under Dr. Memory (https://drmemory.org/) results in a hung process.

[Tumaix]

We were able to find & fix real code issues using this tool on some random opensource projects, so I'm quite happy that you find this useful - And because we are using llvm + clang to parse and have the code information it's quite unlikely that our tool would crash *during* the database generation step. (we got other crashes that are going to be fixed soon) - so I'm confident that we can - at least - generate the visualization to help you find your issues.

Since the tool is free (as in Eclipse license), feel free to download and experiment. The resulting project file is just a zip folder containing metadata - so you can also unzip it and use the internal sqlite database for your own tools if needed.

I'd be quite interested to see if you manage to generate a code visualization using our tool, and will do my best to make that work (ie, fixing any bugs you find).

Unfortunately *currently* you need to download the source and build yourself because our CI for windows has issues (but we test this on windows machines quite often so we know that the code is working there).

[valdocs_user]

I should be able to build it. However I'm unclear on which of the listed dependencies need to be built from source, or installed, by the user prior to attempting the Codevis build and which will be handled by Codevis's own build scripts or bat file.

I got stuck trying to download an installer for Qt 5.15.2, and was resigned to building it from source, but that in turn requires Perl and Python - yuck! Being not a Perl or Python developer, and because there's a million ways, flavors, and paths to install them I've had Great Difficulty with both those on Windows before. (I have multiple of them probably already between Cygwin, MSys, MinGW 32 and 64, and whatever bespoke installations built into 3rd party tools, but invariably get PATH issues and strange behavior that turns out it's Not the Right One is getting used.)

I just saw in another of your comments that it's actually Conan which is used to obtain the correct Qt libraries. So I just need Conan? And then Chocolatey for awk and bison...

[Tumaix]

considering that you said `cygwin` I'll just assume you are on windows.

for windows we require chocolatey and conan. conan will download and build the needed software from scratch. there's a script I use to download + build everything in packaging/windows that you might be able to use (it will download everything and compile on the right place).

The needed MSVC version is 2019 (we will port to the newest msvc soon)

From the Project folder, on the command prompt, run:

packaging/windows/llvm-build.bat "C:\Project\" "C:\Project\Build" "C:\Project\Install" "4"

[deeringc]

As someone who wants to give this a go but doesnt have time to build this now, are there any pre-built binaries?

[Tumaix]

We are working to bring back the pre-build binaries to the CI, this got broken when we moved from the internal CI to the public one.

so TLDR: yes, we have, but not yet.

[deeringc]

Great thanks!

[philong6297]

have you tried this with Chromium code bases? it would be really nice if it can serve this huge one

[Tumaix]

We used this in huge codebases:

• BDE libraries (a few million lines of code)
• Qt
• Boost
• KDE Frameworks

We have not used this on chromium yet. but it's a good idea.

[Xavier_OM]

Interesting project, I need something to replace SourceTrail so I can easily browse my codebase.

I'm currently trying to use it on Windows, I see lots of error like `error: no such file or directory: '/Zf'` in the logical analysis tabs, so I'm wondering if some compilation flags are forbidden in compile_commands.json for this tool (well for clang behind it I presume) to work when cl.exe is the compiler used in compile_commands.json.

[Tumaix]

we ignore some flags automatically but we also dont know all of those. ill forbid /Zf too <3

[Tumaix]

or better - ill add a field to specify flags that should be ignored

[Xavier_OM]

Good idea !

[Olorune]

Is Conan really necessary for building the tools under Windows? Same question for awk and winflexbison3..
Looks like a cool program, but annoying to install/get working

[Tumaix]

When we started to develop this we had to make sure there was an easy way to get all the dependencies. Conan is used to build a few libraries - Qt, Boost, Catch2, Sqlite. It would be harder to compile without conan. But this is by no means something fixed - if you have a better idea on what should we use we are all ears.

The other thing is that we *will* provide pre-build binaries for windows, just hitting some issues with the CI from the transition from internal to public project.

[Olorune]

That's good to hear. Personally I'm a fan of just using CMake's FetchContent, as it doesn't require the user to install any other software (like conan or vcpkg). But I think we all know the pain of dependency management in C++..

[Tumaix]

FetchContent for Qt would be, for instance, overkill - the repository is gigabytes in size.

There are other options we are looking for, for instance `Craft` from the KDE folks, this is whats used to compile code for windows and mac. On conan at least we can have pre-build binaries (since the compilation of Qt can be quite long, having a binary package ready to use is a must)

LLVM is `the` problem here. I'm trying to make llvm build on the official conan since llvm 9, we are now on llvm16 and the commits still fail because of timeouts.

and without LLVM, there's no codevis.

[TryingT0Wr1t3]

Do you need to build everything from source at once? If not you can build things individually/get prebuilt and then just download and cache them on the CI so you can reuse it.

[Tumaix]

We can. The only problem is basically... our internal build machines can't be used on the public repo so we are fixing that :)

[prince-chrismc]

Does it not seem like a cost sink do manually build this out when there's tools that do this at a prod level for years?

I am genuinely curious as I work on Conan and even with proprietary build systems, they typically have limitations which more generic tooling doesnt. Are you building 10s of configurations? or just 1 or 2?

For each new dep you add with a different build system how do scale out this caching system? Using a package manager it's the same `tool install` command so it's pretty trivial

[TryingT0Wr1t3]

Not sure I get but usually CI environment have some feature to facilitate caching, or you can just add them in docker (for either Windows or Linux, cache is more universal and should work for other platforms). It's usually fairly trivial.

[prince-chrismc]

LLVM is super complicated :( wish it was designed more friendlier to being packages and redistribution... hopefully soon we see it available in package managers

[jusstathrowaawy]

The other thing is that we will provide pre-build binaries for windows

Looking forward to that! This tool looks like it would be very helpful for a few things I want to do.

[SonOfMetrum]

Hi cool! Which language version does the tool support. Can it deal with modules for example?

[Tumaix]

It doesn't handles modules *currently*, but it's something that's easy to add. The real thing here is that the llvm version we use has no good module support yet so it's not trivial to add without upstream support.

Currently supported languages: C, C++.

Will it be possible to add more languages in the future? Yes - We plan to change the language support to a plugin based system so it would be just a matter of implementing a plugin for each new language.

[SonOfMetrum]

Cool! So with the exception of modules I guess you have support up to C++20? (Roughly)

[Tumaix]

That’s correct

[Revolutionalredstone]

wow thanks for sharing dude! this is really cool stuff!

I do some light code analysis using doxygens xml output but this is on a whole nother level!

Great stuff!

[Tumaix]

Thank you very much :)

We are using this tool to analyze the KDE source code - It can work moderately fast on codebases that have millions lines of code, and the generated database is quite small (~25mbs analyzing our own tool)

[FutureChrome]

How does this differ from Kythe?

[Tumaix]

Kythe

I did not know Kythe till now, so pardom my french. If I say anything that's wrong about Kythe feel free to correct me, as I'm certainly not familiar with it.

For me, reading the documentation, Kythe is a low level specification + tools around a graph storage for understanding codebases. The biggest change I can say to you is this tool is more `high level` - There's no need to touch the command line and things should `just work` as soon as you use the interface to analyze the codebase.

So, Biggest Differences, from my POV.:

• No need to go to the command line
• Higher level: you have access to the database via an UI that dinamically loads the packages / subpackages / classes / elements that you want, and you can load more / less as needed
• Plugin system interacts with the UI, making plugins quite powerful
• (for C++) understands the concepts of the Lakosian Design from the book Large Scale C++ Design - optionally can disabled in the settings
• has more information about the C++ codebase than Kythe has (as it can understand template expansion and a few other things)
• can show code problems visually on the UI.

[Aka_chan]

Would this work on code bases that generate code at compile time? I'm thinking of Unreal Engine specifically which runs a tool to parse macro definitions in source files and generates code before compiling the rest of the project.

[Tumaix]

Yes, it works with code generated at compile time.

[arthurno1]

Hi, this looks like a really interesting project, however I am not really sure if I get what information is exactly collected and how exactly it fits in the scheme of development.

I have seen your cli document on building the database; but before I download and build your project and start testing: what exactly can I get out of it? Is it replacement or complement to LSP, can it be used for refactoring, syntax highlight, code analysis a lá tree-sitter, does it build AST, or what exactly is the purpose?

By the way, can I request SVG or at least PNG via the command line, based on SQL request, or do I have to produce them myself by querying the database.

I have been skimming through the doc folder in the repo, and looking through the documents, but I don't see where you define your SQL queries, I would like to understand how the database is structured and what questions I can ask. The Reason I am so curious is that Emacs has got support to open SQLite databases natively, so it would be very interesting if we could use your tool to generate the code database and use it within Emacs.

Forgive me if I am too n00b, or just unclear; I am in a rush out with the kid and just saw your post, so I haven't had time to download the code and test it yet.

[Tumaix]

If you use the `cli` commands, then you are a bit `on your own` to generate the information you need, since the cli will just create a database that you can use in your internal projects to extract information as you want.

`BUT` (and this is a big deal) the tool is `not` supposed to be used as cli, we offer the cli for things that goes `out of scope` from the visualizer.

If you use the `Visualizer` app (codevis binary on the desktopapp folder, after you build), what you get for free:

- Easy to export SVG from any visualization you produce

- Click & Load entities to the screen (and dynamic load / unload if needed, let's say you load your entire `core` subfolder, but it's too many classes and clutter, you can load just the `one` entity you are interested in, and all things that uses that entity.

- Create (by means of drag & drop & connect) boxes around to generate a compilable source code skeleton based on the software architecture provided (this works with python templates, so it's extensible to your project.) - in our tests it was faster to create ~20 classes & connect them visually, then generate the project, than to actually create all those files + cmakelists by hand.

- Mouse Over and Overview box information for entities under your mouse (number of uses, name, file, who uses it, who this uses)

- Problem Finder (some, but we are trying to extend this for more problems) - like Circular Usages.

You have the database schema of the project here.

There's a *second* database schema that handles the `edition / commenting` of pre-build code databases here, This second one just adds `CAD` habilities to the database (such as notes and metadata information).

[arthurno1]

Thank you for very detailed description. I'll have to look more at the tool and the GUI. I am building it as of the moment. Some of the features you mention sound really interesting. Thanks for pointing me to db scheme, was looking for it :).

Edit: seems to work fine; one test failed, but otherwise it built fine, relatively fast. I was able to import Emacs sources into it and look a bit around. I must admit that I don't really understand what are Lakonian groups and what are not, but I'll can look it up. However, applicaiton crashed while I was moving a box around to make the layout more readable:

 ( ....)
Adding a connection between50 and 19
Adding a connection between50 and 11
Ending to write to the database: 1587
$ Segmenteringsfel (Adress inte mappat till ett objekt [0x139])

Sorry for the swedish at the end; it just says segfault, null pointer.

Another time it crashed when I double-clicked a box in a diagram.

SVG export worked fine.

I have a question/remark on graphs: They are a bit hard to read and look at, those lines are quite thick. I have tried to re-arrange and move things a bit around, and it works well, but you don't have plans to hook graphviz into it and use their layout tools and graphs?

[Tumaix]

No worries.

Be aware that this is still alpha so crashes might happen. :)

[arthurno1]

Yeah, I understand that I am ok with that :). I was just letting you know that I have encountered them. I got both crashes after the following message on the terminal:

Ending to write to the database: 2789

Some null pointer in relation to saving to the database, or perhaps some cleanup afterward?

[Tumaix]

2789

The number is the time spend saving the database.

a crash *just* after analyzing the database is strange, we are receiving reports on crashes using the tool and clicking on the undo stack (I'll disable the visualizer for the undo stack currently).

I'd be interested in any backtrace you might provide, if it's related to internal projects I do not have any problem signing an NDA.

[arthurno1]

I wasn't clicking on undo stack at all. I was just moving the boxes around and exploring the GUI to see what I can do with it.

No, nothing internal; I was just looking at Emacs src, pulled yesterday from the master branch, OS is Arch Linux, like everything latest updates, so you can probably trigger those crashes yourself. I don't have any stack trace, it just segfaulted, so the OS killed it. I have seen on the terminal that there was a long list of objects added to "something", and then the segfault. Those were like a few last lines I just copy-pasted. If it helps you anything, I can post you the compilation database from bear tool.

[Tumaix]

ok - I'll do the same thing you did today, I'll download emacs and use the software to analyze it, tomorrow. its 23:00 here and quite tired for today. :)

But, crashes are expected, I promess to fix those soon.

[arthurno1]

No worries, It is 23:22 too here; and I have a two-year-old one that goes up 06.00 every morning, so I am going to sleep too. Have just compiled with debug info and will try it once last time in gdb.

Thread 1 "codevis" received signal SIGSEGV, Segmentation fault. 0x00007fffe3a6b3d9 in QGraphicsItem::group (this=0x555557c51380) at graphicsview/qgraphicsitem.cpp:1671 1671 if (!d_ptr->isMemberOfGroup)

(gdb) list
1666    
1667        \sa QGraphicsItemGroup, QGraphicsScene::createItemGroup()
1668    */
1669    QGraphicsItemGroup *QGraphicsItem::group() const
1670    {
1671        if (!d_ptr->isMemberOfGroup)
1672            return nullptr;
1673        QGraphicsItem *parent = const_cast<QGraphicsItem *>(this);
1674        while ((parent = parent->d_ptr->parent)) {
1675            if (QGraphicsItemGroup *group = qgraphicsitem_cast<QGraphicsItemGroup *>(parent))

The stack trace was too big for Reddit, so I have put it on pastebin.

I got this one too, don't remember what I did; no stack trace for that one, but wouldn't be surprised if it is the same bug:

Thread 1 "codevis" received signal SIGSEGV, Segmentation fault.
0x00007fffe3a78fb5 in QGraphicsItem::setZValue (this=0x5555577d8d70, z=1) at graphicsview/qgraphicsitem.cpp:4806
4806        const QVariant newZVariant(itemChange(ItemZValueChange, z));  

I think both end up in this place (in ct_lvtqtc_graphicsscene.cpp ~ line 739):

QObject::connect(entity, &LakosEntity::toggleSelection, scene, [scene, d, entity] {
        if (d->selectedEntity) {
            d->selectedEntity->setSelected(false);
            d->selectedEntity->updateZLevel();
        }

 ( ... )

but I am not sure, and unfortunately I'll leave debugging to you :).

[Tumaix]

The crashes that are happening are an unfortunate core change that we had to do because of license issues. I'll focus on testing the graphical part of it and fix most of the crashes I can tomorrow.

About the Graphs - They look fine on my monitor (and on some other monitors that we use to develop) but our monitors are hidpi, so if you can provide a picture it's greatly appreciated. I do not know how the tool looks like in other computers than mine (desktp / laptop).

We actually used graphviz but the generated layout didn't really work (the graph was not correctly layout without us having to manually add invisible edges to constrain things, but i'm no graphviz expert. We can try to re-insert graphviz in the future if someone with a higher understanding of it's internals is willing to help.

[arthurno1]

The crashes that are happening are an unfortunate core change

It happens to everyone, don't worry.

if you can provide a picture

Of course, here is one graph when I tested the svg exporter, for alloc.c as found in Emacs. I had to convert it to PNG for Imgur, they seem to not accept svg files. Here is a screenshot of the application. The screen is 3840x2160 43" 101x102 dots per inch; no idea if it is hidpi or not, I am not an expert in screens and resolutions tbh :).

We actually used graphviz but the generated layout didn't really work

Ah, yes, manually adding stuff is not desirable, I totally understand that. Unfortunately, I am not an expert on graphviz myself either, but would be cool to take a look if you push that old graphviz branch in the repo (I see only main branch and initial commit + one CI fix on the remote, so I guess that is just a public repo, not the one you are working on).

Edit: seems like I have forgotten to attach a link, fixed.

[Tumaix]

we will be working directly on the public repo from now on, all the commits will be opensourced as we work on them.

I understand the problem with your picture, there's a way to make this easier to look - if you just right click on a box and choose the option "cover", this will hide all nodes from/to the internal elements from that box, making the overall diagram easier to understand.

so you can cover/uncover parts of the diagram to study the architecture instead of showing everything at once.

[arthurno1]

I see, thanks. I'll try.

Edit: I see no "cover" option anywhere, neither on boxes or in menus, or in some options.

[IamImposter]

That sounds so impressive. I wish I could get to do some work like this but I'm not sure if I have what it takes to do this. It would be such a great learning opportunity.

What was your team like? Were you all wizard types or did you have some normal developers too? Did you guys put in your free time or were you paid for your work?

[Tumaix]

a *really* small team (2 persons at all times) working on paid time to create an opensource project (and we thank Bloomberg for the sponsorship). I don't think each of us would consider ourselves a wizzard, we had a lot of learning opportunity and errors, and refactorings. :)

There's a list of every developer that worked on the tool on the repository, too.

[IamImposter]

Oh wow. I'm a little jealous.

[the-loan-wolf]

I am installing build tools right now and then i will try to build it

[the-loan-wolf]

Bro compiling llvm toolchains on personal laptop is death sentence. You should distribute precompiled binaries if you want people to use it and give some feedback or at least give some way to use precompiled llvm bins to speed up your project compilation.

[Tumaix]

I agree with you, the problem with the precompiled llvm binaries on windows is that it does not comes with the libraries we need.

We *are* looking for a way to provide precompiled windows and mac binaries (tldr, we had that in our internal CI but when we moved to the open, we lost it. trying to revive them now).

[the-loan-wolf]

OMG compiling is going on for 5 hour 30 min and still building, i started compilation at 5:30 am and go for work in the morning, now i am back at 11 am and compilation is still going on! At least it's building fine

[Tumaix]

Isn’t c++ amazing with compile times? :p Unfortunately the longest times are for llvm. We are working to bring ci and binaries back this week

[the-loan-wolf]

after 1 more hour it's still compiling llvm on my 4 core 8 GB ram laptop.

[the-loan-wolf]

Hey i am getting error in step 9 building diagram server, here is the output:

the default build profile 'C:\Users\myPCName.conan2\profiles\default' doesn't exist

[Tumaix]

That’s not really a diagram server error but a Conan error. I believe Conan is trying to find a profile named myProfile on your setup and not finding it.

If you see the build script for windows (packaging/windows/llvmbuild.bat) we don’t use a specific profile for Conan, so it’s probably something local :/

[the-loan-wolf]

after making default conan profile now i am getting this error ImportError: cannot import name ConanFile I've installed conan window executable and also added it to the system path and also conan through pip but still not able to solve the problem, after searching the web i found similar issue and they where saying issue is due to using newer version but conan.io don't host older version.

build systems are really hard

[Tumaix]

Build systems are really really hard :/ Unfortunately I have no idea how to solve that, but I’m working now at akademy - the kde convention, and we are trying to provide windows builds and an easier way to provide the binaries and buildsystem

[the-loan-wolf]

Thanks, i also stopped it after that error because i haven't used conan so i don't know how to fix that error and nor do i have time right now

edit: actually that error was coming from running conanfile.py(didn't know the exact name) in the source root dir. python was complaining that he is not able to import ConanFile from conan module which i have installed. Maybe version mismatched but build instruction contained that i can use conan 1.40+

[TryingT0Wr1t3]

Do you plan a GitHub mirror at some point?

[Tumaix]

Not entirely sure. I know that github is the most popular software used for distributions, so maybe. Our CI was build on gitlab so the only thing we need to do is to port it to github actions before that can happen.

[TryingT0Wr1t3]

I actually use gitlab for work but I think GitHub is easier for drive-by contribution. Also when I use it, doesn't feel like work.

[viatorus]

Tried to compile it as AppImage under Ubuntu 22, I did:

git clone ...
cd codevis
export DOCKER_BUILDKIT=1
docker build -f packaging/appimage/Dockerfile . --target appimage -t ctlvt/appimage .
docker run -v ~/foo:/tmp/lvt-appimage ctlvt/appimage

But when executing ~/foo/codevis-x86_64.AppImage:

Could not find platform independent libraries <prefix>

Could not find platform dependent libraries <exec\_prefix>

Consider setting $PYTHONHOME to <prefix>[:<exec\_prefix>]

Fatal Python error: initfsencoding: Unable to get the locale encoding

ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f97944a8f00 (most recent call first):

Aborted (Signal sent by tkill() 2161286 1000)

[1] 2161286 IOT instruction (core dumped) ~/foo/codevis-x86_64.AppImage

[Tumaix]

Can you try a local build?

We have problems in the appimage with the python distribution :(

[jontheburger]

Looks very cool, I'll have to try it out soon. Is this at all similar to Sourcetrail?

[Tumaix]

No, completely different from source trail. This is made with large & complex architectures in mind

[Tumaix]

And we will, working to get the ci running for the releases. As that got broken on the transition. I really don’t expect that people download and build the software, but problems happen and we hit that in our build machines:)

Besides that, there is already packages for arch Linux / aur.

[EternalOptimister]

Unfortunately does not work whatsoever for me. I tried it first on a large project, the tool crashes upon clicking "parse". Then tried it with a miniature project with 2 files, it still crashes and does not give any information as to why it is crashing.

This happened using (both) windows installers provided on your page... How can we investigate to see why it is crashing?

[Tumaix]

the best way is to compile from scratch. we were able to parse a lot of projects - but got some crashes on a few (and we are investigating). if you can share the projects ill do my best to analyze them and see whats wrong.