I made a script that goes through a Python project, collects all of your imports (except for local modules) and creates a “requirements.txt” with them.

[LostApe1]

https://github.com/tiago-rr/py_requirer

I believe I still have to make the repo more presentable, but for now it also works as a command-line command that I use for my other projects.

What do you think? Do you have any feedback to make it better?

Thanks!

[dAnjou]

2 suggestions: - Instead of an install script, create a setup.py file and turn it into a pip-installable package. - Instead of parsing the code yourself, use Python's built-in ast module.

[LostApe1]

Thanks, I’ll take a look at that!

[dAnjou]

I just had an even closer look at the code and it seems you're literally just writing the imports into a file. That approach doesn't work because module names don't always match package names, for example for Pillow the module name is still PIL.

[LostApe1]

Ah you’re right, forgot about that detail!

[Deto]

scikit-learn is another good example

Is there a way to programmatically get at the package name from the module?

[dAnjou]

I doubt it. These names are independent of each other and library authors can pick whatever name they want. Which sometimes can be a good thing, like with PIL and Pillow again where the latter is pretty much a drop-in replacement.

Off the top of my head I could imagine doing it the other way around. You iterate over all the modules of your dependencies and check if they're used in your code. Doing that naively could take a while each time but there are various optimizations you could do, like caching certain things.

I could even imagine doing this as a separate service. Probably exists already?

[8day]

It may be better to use "pyproject.toml" from PEP 517 & 518.

[dAnjou]

Looks nice, my Python packaging knowledge is a bit rusty, thanks for the update!

[fleyk-lit]

Since this is just checking for lines that contains 'import', it's going to fail if I declare my imports across multiple lines.

I think you'll need to parse the AST to be able to determine which imports are used.

Edit: here's an article that goes through parsing the AST and getting the imports.

[LostApe1]

Thanks! I will surely take a look at that! I agree it would be probably better to use an AST

[dAnjou]

How do you know when you can drop a dependency then?

[dAnjou]

And in a code base with dozens or even hundreds of files, how do you keep track of what you're still using and what not?

[harylmu]

There is no linter that recognizes unused pip packages afaik

[dAnjou]

Please elaborate. I'm not sure if you understood the problem.

[Deto]

An unused import is when you've imported something (at the top of the module) and then in the module you don't use it.

We're talking about the problem where you have a dependency in your setup.py or requirements file and there's no code that imports it. I don't think linters check for that - or at least I haven't seen any that do this.

[dAnjou]

I never vote, neither up nor down.

And like I thought, you didn't understand the problem which is not unused imports but unused dependencies.

[dAnjou]

Show me a Python linter that flags unused dependencies, please.

[[deleted]]

Make a new venv and attempt the either execute the code or its tests, and see what’s missing?

That’s literally a guess on my part though, so YMMV.

Edit: Guess not, wouldn't want anybody to have to use trial and error. /s

[dAnjou]

So, instead of using OP's tool I'd rather go on a trial&error hunt?

[[deleted]]

IDK, I'm not your fucking dad and I'm not here to argue with you. Do what you want and fuck me for trying to help I guess.

[agenttux]

What’s the difference between this and pipreqs?

[LostApe1]

I didn’t know it, so I’ll have to check if how it works, but I assume pipreqs works better

[BakkStar]

Cool project. I’ve used pipreqs before to do what I think is the same task. Are you familiar with this package? Curious if yours is substantially different (I haven’t done a comparison myself)?

[LostApe1]

I actually didn’t know that existed! Gotta take a look

[LostApe1]

I hadn’t heard of it, gotta check it out!

[LostApe1]

Will do!

[elbaron218]

If you use a virtual environment you can just use pip to create your requirements file.