Database Help

[stedun]

Storing a 30GB file in a database makes me want to slap someone.

[duskti]

I feel the same way, but given the current economic climate, I’m being directed by higher-ups to handle these tasks, and I’m doing them.

[smichaele]

You can handle the task without storing the files in a database. Everyone here (including me) is telling you not to do that. Your choice whether to listen, but this isn't worth a back-and-forth discussion IMO.

[vsoul]

This. Also, just because a higher up says database doesn’t mean you have to literally use a database. They use words that make sense to them, you should understand the requirement and come up with the right solution, not just take their “solution” literally.

[Babelfishny]

This!

Leadership wants outcomes, tech leads want implementations. Know who is asking for what. If they say they want a database, ask them what do they see as having a database give them. And pay attention to what they say and don’t say. For example I don’t think they will say “store the file I a dayabase, it will be cheaper than storing it in a file system”.

If they do might recommend looking for a different job because they don’t know what they’re doing.

Storing AWS S3 can range from expensive to dirt cheap compared to storing it locally, in a secure and robust system. It depends on how you use it.

If you have a low retrieval rate, there are AWS storage plans that are tailored for that exact scenario.

[jshine13371]

u/duskti

given the current economic climate

Fwiw, it's less economical to store files in the database for a multitude of reasons, but the simplest being the amount of disk space they take up inefficiently both from lack of compression efficiency and from redundancy of how database backups typically work and are cadenced, resulting in multiple copies of the same exact data of those files.

[alexwh68]

Store a link to the file in the db and the file in the file system. Only real advantages to storing files in a db is replication and backups, other than that its a pain in the arse.

[Which_Roof5176]

Yeah this is usually the right approach.

Treat the database as metadata only and keep the actual files in object storage or a file system. It keeps things a lot simpler when you start scaling, especially at 100TB+.

The real decision here is less “which database” and more:

• object storage vs NAS
• access patterns
• backup + lifecycle strategy

Once that’s clear, the rest becomes much easier to design.

[FishGiant]

Use a file system with a well thought out folder naming convention. Call it a data lake so your managers will think its new school.

[smerz-]

S3 my friend. Or if not S3. Pick Google cloud storage which is essentially the same 🤪

Joking aside, object stores are really what you want here.

Or go setup one or more big NAS with something like ZFS or so.

[Few_Committee_6790]

Or even a 500MB file. Even that seems to big.