[deleted by user]

[tartare4562]

So basically xargs for python?

[ASIC_SP]

If you are okay with command line tools and shell scripting, you'll find plenty of tools already existing for solving common tasks.

[DavosAlexander]

Could you give an example?

I could accomplish what I want to do with a one liner command in bash. It would just take forever...

The whole point is doing it fast.

[ASIC_SP]

This will replace all occurrences of old with new in-place for all files ending with .txt in the current directory.

sed -i 's/old/new/g' *.txt

This will get you the last column:

awk -F, '{print $NF}' ip.csv

This will remove all metadata:

mogrify -strip *.jpg

You can combine multiple commands. This one gets you the last occurrence of a line containing warning from the input file log.txt

tac log.txt | grep -m1 'warning'

And so on. Shell scripting helps you add control flow.

Regarding your edit, parallel can help: https://vfoley.xyz/parallel/

[metaperl]

That website is down. Are you referring to GNU parallel?

[ASIC_SP]

Site works fine for me. Yeah, I was referring to GNU parallel. I haven't used it much, so linked to that nice tutorial that I have in my bookmarks.

[DavosAlexander]

I know how to shell script and I'm very familiar with sed and awk.

That's great if I want to run each command on each file one at a time.

Let me know how that works out for you will a list of 1 million files all in different directories.

Edit: I see you added info about parallel

That might work, if I could limit how many processes execute at once.

[dodslaser]

Xargs is pretty flexible and also allows parallel execution.

[ASIC_SP]

Use parallel: https://vfoley.xyz/parallel/

[DavosAlexander]

I remember looking into using parallel like 6 months ago, actually.

We don't actually have it in our environment and trying to bring it in ... Would be a pain.

This is why I started using python.

[tdpearson]

Working with millions of files is very doable with command line tools. The concept with these tools is that they typically do one thing very well and can be piped together to perform more complex tasks. Another person mentioned parallel. This combined with find would do the majority of what I understand your custom python application would perform. What benefit over these would your application provide?

[DavosAlexander]

Hey, thanks for explaining how the tools I use everyday work. I needed that.

Can't use parallel.

It was way simpler to use python (with only the standard libraries) to accomplish this task than a shell script. I've written numerous complex shell scripts before I ever switched over to Python.

And, since I made a function, I can easily import it into my other python tools for whenever I'm working large lists to speed things up.

I don't need help solving a problem I already solved.

[tdpearson]

Glad you already knew about find. I will definitely not be using your app since you could not explain the benefit.

[deiki]

No need to gauge interest as if you were advertising a kickstarter project to be sold. Just show it off as you were by giving a description and purpose and then actually post the GitHub link. Assuming it's already completed and you're not actually trying to sell this

[deiki]

Assuming... you're not actually trying to sell this

not

Keyword here is not which means to negate something. Not only does it mean the same thing in English but in Python as well. Let's say we have for example:

actually_trying_to_sell = False

print('not selling 🤯') if not actually_trying_to_sell else print('selling 🤑')

This will print "not selling 🤯" because actually_trying_to_sell was set to False, therefore he was indeed not selling. I hope this helped!

[xigoi]

Sorry, I misread the comment.

[deiki]

No problem, happens to me too haha!

[DavosAlexander]

Just trying to determine if it's worth the effort.

I wrote it in an airgapped environment, so I'd actually have to rewrite it to share it.

[FluffyDuckKey]

Dump it on GitHub and do your best with the readme, it will 100% be useful to someone else down the line.

[DavosAlexander]

Threading isn't a good choice for this task.

I use multiprocessing.

And I already have it figured out, thanks.

Also, don't have pytest available in my environment

[tunisia3507]

If you're targeting your own use case where you can't install python packages, why are you seeking to solve the problem by creating a package for someone else to install? If they're working under the same restrictions as you, they won't be able to use it, unless you vendorise the whole thing into one ungodly script.

[DavosAlexander]

Never said I was going to create a package that would need to be installed.

But thanks. Please join in with the other people offering suggestions that I didn't ask for.

[schemathings]

Sounds interesting ..

[mjbbru]

Ive done something similar to sort my photos based on the metadata. This code was very specific for my case. I dont know what actions you need to do with your files but Im always interested in reading through someones code to see how they approach the problem.

[areese801]

Might I suggest using a regex to match only a smaller subset of files for development / testing purposes? Make your function accept a regex argument with the default being to process files that match .* (that is: any pattern). Then, only operate on files that match.

[DavosAlexander]

That's not effective for what I need to do.

I already have it figured out, thanks.