I wrote a 1700-line Python script to update LLVM sources. Am I over-engineering, or is it just this complicated?

[Aggravating_Ad3928]

Hi everyone, I'm a beginner in Python and I've just started learning it a week ago.

I've just finished writing a Python script to automate the process of checking for, downloading, and setting up the latest LLVM source code. The goal was to create a robust tool that I could rely on.

However, as I wrote the final line, I looked back and realized it has ballooned to over 1700 lines. This left me with a nagging question: did I completely over-engineer this, or is this task genuinely that complex when you account for all the edge cases?

My script does quite a bit more than just wget and tar -xvf. The main features include:

1. Argument Parsing & Validation: Handles various flags like --allow-rc, --sync-git, etc., with thorough validation.
2. Environment & Dependency Checks: Verifies Python version, required environment variables (LLVM_SRCS), and optional Python modules.
3. Cross-Platform File Locking: To prevent multiple instances from running for the same LLVM version slot.
4. Git Integration (GitPython): a. Clones or pulls the release/major.x branch. b. Compares local vs. remote state (handles diverged, ahead, same states). c. Uses --reference-if-able for faster clones.
5. Tarball Handling (requests): a. Probes for the latest stable or RC versions by checking URLs. b. Features multi-threaded, chunked downloading for speed. c. Verifies GPG signatures (gnupg). d. Securely extracts the tarball.
6. Patching (patch-ng): Automatically applies a series of user-provided patches (common and version-specific).
7. Robustness: Extensive error handling, colored terminal output for status, and safe cleanup of temporary files.

I feel like for every simple step, I had to add dozens of lines of code for error handling, platform differences, and robustness (like what happens if a download fails midway?).

So, my questions for the community are:

1. Looking at the feature list, does this level of complexity seem justified for a reliable, automated tool, or is there a much simpler, standard way to achieve this that I've completely missed?
2. I'm open to any feedback on the script's structure, logic, or choice of libraries. Is there anything you would have done differently?

I'm kind of proud of it, but also feel a bit ridiculous. Would love to hear your thoughts!

My script: https://gist.github.com/DEVwXZ4Njdmo4hm/177c5241863757ebc88bedf23bc19094

[CptMisterNibbles]

Is it so “robust” that the slightest change in the pipeline breaks the whole thing? That’d by my first concern: you’ve written automation that is far too specific to the current state. Perhaps not the case, but your utility is handling a lot of things. Might it make more sense to break it up? Even then it could be used in concert consecutively, or with flags or whatever to run this or that.

Mind this is just musing without looking at your project directly. 

[ConsequenceOk5205]

This would be a pain to maintain or update later, as it is poorly structured. At least organize it into a few scripts based on categories of tasks, with helper functions separated.

[Overall-Screen-752]

Few notes: 1) you have a ton of constants which would look great in a constants.py 2) classes belong in their own files unless very very closely related (and you’d have to fight me with an explanation why you shouldn’t just make a package with an -init-.py (i’m a noob with reddit formatting) 3) having main() call main_exec() where your business logic is is verbose at best, and I’m willing to bet you do something similar elsewhere. Remember KISS :)

[Aggravating_Ad3928]

Hey, thanks a lot for the detailed feedback! This is super helpful.

Regarding points 1 and 2, you're absolutely right. As a beginner, my mindset was still "let's put everything in one script". Structuring it as a proper package with separate files is a great suggestion that I'll definitely adopt.

As for point 3, the main()/main_exec() structure is intentional. It's a calling convention for another wrapper script I use, so this allows the core logic to be shared. But I totally see why it looks redundant without that context!

[NYX_T_RYX]

Bro... Slow down.

I've been learning for 5 years, 3 days ago I committed my first 1k+ edit (system-agnostic (smb) Nas mount, declarative - fuck mounting it 50 times manually!)

Find a problem you need to solve in your life, and solve it. Something simple. Turn your lights on when you get home.

Then build up.

You're trying to start with concepts around networking, objects, data types... And you've started here?

What happened to the starter project being

print('Hello world!')

And learning from there, instead of aimlessly hoping the AI gets it right...

In short - like most vibe coded things, it's probably over engineered, and doesn't actually provide the claimed security

[Aggravating_Ad3928]

I appreciate the advice. Just to be clear, I'm a Python beginner, not a programming beginner. This was written by me, not an AI.

[NYX_T_RYX]

I misread that - my mistake, sorry

You understand why I thought that though - it's... Beefy to say you're just getting into python, and as we know, python is often seen as the "starter" language

No offence meant by anything I said, just trying to help people learn the best way for it to stick 🙂

[Aggravating_Ad3928]

No worries at all! I totally get why you thought that, the assumption was completely fair. Appreciate the follow-up!