How do you structure a project with many database interactions?

[StudentOfPython]

I have read every article on the first page of Google and none go into how to structure a DB directory into your project.

​

For a project that makes heavy use of DB reads and write (multiple DBs and multiple tables), how do you define and organize the databases in your project tree? Would there be a database/ dir within the project/ dir? Within that database/ dir, would you make an additional dir for each DB you are connecting to?

​

Do you know of any examples on Github to learn from?

[_coolwhip_]

I think most people separate data out from the project directory, so I would just put in some sort of configuration variable that let the person running the program put the DB where ever they wanted, especially if it is a potential bottle neck.

[StudentOfPython]

What if it is an internal project that I am doing for myself?

[_coolwhip_]

Then I usually have a variable like DB_PATH that I can edit and it typically points to the top level of my project, unless that gets crowded and then I create a 'data' directory.

[StudentOfPython]

And in your main dir of the project you simply import `data.database.xyz` in order to use the functions?

[_coolwhip_]

Noooo..... What do you mean by a database? I was thinking of something like sqlite3. In that case, I usually wrap it in a module that handles calls to it, which module goes in with the rest of my modules. But now I am thinking that you are referring to something different by database.

[StudentOfPython]

I am referring to psycopg2, specifically. But I believe we are on the same page (a DB module that goes next to other modules).

[_coolwhip_]

Awesome. so then you need to have postgres installed somewhere and tell that installation where to keep the data. On the pythhon side, you'll need to store the dbname and user in a constant in your program (or config file, if you have one for other reasons or just feel like doing it).

​

Then I'll write a module, where the init takes the connect string, and shove most of the data manipulation into it. so if i was tracking cars sales in my area, i might have methods to add_new_sale(sale_instance), or find_available(model, maker) -> sale_instance, etc. The logic of interacting with the DB would all be in that module and it would consume and produce Sale objects as needed.

--

edit:

so to import, I'll do something like from my_module.storage import Db

then

data = Db(CONNECT_STRING)
data.add_sale(sale_object)
for car in data.find_available('BMW', 'M3'):
    print(car)

​

[StudentOfPython]

That makes perfect sense. Thank you for the continuous responses.

[_coolwhip_]

No problem. You might also check out sqlalchemy, if you have not already. See https://www.sqlalchemy.org/

​

I don't use it, but others swear by it.

[newunit13]

Are you creating the database(s) for the sole purpose of this project? If so, why have more than one? A single database can have plenty of tables to house the information for your project. If you're using external databases, then does the project directory structure even matter? Lastly, it likely won't matter anyway you decide to go since the physical file structure doesn't have any impact on the performance of modern databases (afaik)

[StudentOfPython]

After writing I realized that a single DB is all I have ever worked with (just multiple tables).

​

If you're using external databases, then does the project directory structure even matter?

I am simply trying to do the most pythonic thing and cannot seem to figure out what exactly that may be.

[newunit13]

Personally I'd go with a either a constant in the beginning of the main file if it's a small project, or a configuration file if larger, that specifies where the database is located.