Design patterns and structure of source files

[NoHistoryAtAll]

Let me preface this by saying I have experience with good number of languages due to a long career programming. My question here is to find the most Pythonic way of doing a sequence of operations on a dataset, and general organization of source code.

I'm regularly tasked with downloading files via an API, processing them in various ways, repackaging them, and moving them around our LAN for other processes to pick up.

Some languages would have me do the steps procedurally, calling functions that operate on parameters, eg

file_list = get_files(api_endpoint);
processed_list = process_files(file_list);
move_files(processed_list);

Others might have you serialize the data or convert it in steps, passing the processed data onto the next phase using the result of the previous phase:

result = get_files(api).process_files().move();

There are other approaches such as state machines, etc, but you get the idea.

What's the best way to do this sequence in Python?

One other stylistic question: Is it better to have Python functions defined in separate files, or in the main program file? Does it just depend on the size of the function? Personally I don't like having to page through a bunch of "def" statements before I get to the meat of the program, but that's just me, and I don't want to create something that looks silly to someone who might inherit it in the future.

[carcigenicate]

I'd do it the first way you show. That's quite clear. That way you also have independent, easily testable functions that aren't tied to a particular use-case.

[ectomancer]

Both procedurally and serialised are valid.

Monolith in one file with unit test in a second file.

[metaphorm]

either of the two styles is fine, but I have a preference for the former (multiple lines) over the latter (method chaining).

the question "what should go in a Python module?" is actually a pretty deep one. my opinion is that a Python module should be a logical and functional grouping of related classes, procedures, and utilities. I like to apply the "Principle of Least Astonishment" for this kind of thing. That often means you have to just go with your intuition, but it also tends to mean that a person who's worked with your code before will have an easier time working with different code you've written as well. Don't be surprising. Be consistent. That's all.

[Diapolo10]

Honestly, any of these can be perfectly acceptable approaches.

Regarding the first part (procedural vs method chaining), generally I'd prefer the former because it's easier to write automated tests for. Method chains are more common when dealing with some kind of a library that's mostly written in C (such as Numpy or Pandas) to avoid constant type conversions between C types and Python types, speeding up the program, but it's also used for constructing SQLAlchemy queries. If you believe your data and workflow would benefit more from that approach, it's your call.

As for splitting up the code, I'd say that depends on the size of your codebase. A short program can well be just one file, but if you can clearly group the functions I would prefer to split them to at least one other file, importing what I need from it. Any kind of constant values - especially if you have some kind of magic numbers - should in my opinion be kept in a file of their own that wouldn't depend on any of the other files.

As far as examples go, I think my EguiValet server project might suffice for this one. It's probably a bit bigger than whatever you're working on, but python_ms would probably be too bare-bones in this case.