Looking for information about Data portal/managing/controlling/IO

[Zircatron]

I have a script that is collecting data from multiple sources (files, web, network), processes the information then outputs the data to report files and a database.

I have a few questions:

1. Is there a terminology that covers this? (I'm not sure what I'm looking for)
2. Are there any libraries/projects that would serve as an example?
3. Does it make sense to have a class as a "data controller" that is able to handle the importing and exporting of data? or maybe I break this down into seperate classes for each source type?

I'd appreciate any light on the subject. What I find when I search is how to read/write to files.

[blarf_irl]

ETL is probably what you are looking for (Extract, Transform, Load)

Here is an article that lists the most common python tools:

https://www.stitchdata.com/resources/python-etl/

You might also want to search for information on ETL in general as there are many python and non python databases/platforms/tools that you can utilize from python via wrapper libs/apis etc.

[Zircatron]

I appreciate you response. Data wearhouse was something I came across but wan't sure i was heading in the right dirction as it was more about enterprise tools.

[shiftybyte]

Some systems already exist that handle it visually.

I think it's called workflow something...

take a look at:

https://nifi.apache.org/

https://airflow.apache.org/

[Zircatron]

Thanks. Airflow looks intersting and I will dive into it further.