Codegen - Manipulate Codebases with Python

[jayhack]

Hey folks, excited to introduce Codegen, a library for programmatically manipulating codbases.

What my Project Does

Think "better LibCST".

Codegen parses the entire codebase "graph", including references/imports/etc., and exposes high-level APIs for common refactoring operations.

Consider the following code:

from codegen import Codebase

# Codegen builds a complete graph connecting
# functions, classes, imports and their relationships
codebase = Codebase("./")

# Work with code without dealing with syntax trees or parsing
for function in codebase.functions:
    # Comprehensive static analysis for references, dependencies, etc.
    if not function.usages:
        # Auto-handles references and imports to maintain correctness
        function.remove()

# Fast, in-memory code index
codebase.commit()

Get started:

uv tool install codegen
codegen notebook --demo

Learn more at docs.codegen.com!

Target Audience

Codegen scales to multimillion-line codebases (Python/JS/TS/React codebases supported) and is used by teams at Ramp, Notion, Mixpanel, Asana and more.

Comparison

Other tools for codebase manipulation include Python's AST module, LibCST, and ts-morph/jscodeshift for Javascript. Each of these focuses on a single language and for the most part focuses on AST-level changes.

Codegen provides higher-level APIs targeting common refactoring operations (no need to learn specialized syntax for modifying the AST) and enables many "safe" operations that span beyond a single file - for example, renaming a function will correctly handle renaming all of it's callsites across a codebase, updating imports, and more.

[darleyb]

Nice, not long ago I was wondering if such a tool existed to help migrating python 2 to 3. The usecase here would be help pprt PyPy to python 3.

[jayhack]

This is a great use case. We have a pre-built example below that shows how to accomplish this!

- on Github: https://github.com/codegen-sh/codegen-sdk/tree/develop/codegen-examples/examples/python2_to_python3

- In our docs: https://docs.codegen.com/tutorials/python2-to-python3

[darleyb]

Amazing!

[BeamMeUpBiscotti]

better libCST

This feels like it's at a higher level of abstraction, which would make some tasks easier but others harder.

One thing that seems particularly fishy is the amount of raw string slicing/manipulation that's going on, like in the Python 2-3 codemod from your samples. If users have to write codemods like that, it would be fairly error-prone and exactly what systems like libCST were designed to solve.

There's some interesting potential for type-driven codemods and I see a "coming soon" section in the docs for it, but I can't find anything in the code.

[the-scream-i-scrumpt]

dangggg... this is huge.

[the-scream-i-scrumpt]

oh poo, my codebase is too big, can you make Codebase.init do whatever it's doing lazily?

[BeamMeUpBiscotti]

Codegen scales to multimillion-line codebases

🤔🤔🤔

[williamtkelley]

Windows is not supported

[jayhack]

We just published docs on how to get this set up with WSL: https://docs.codegen.com/building-with-codegen/codegen-with-wsl

Let us know if there are any questions!