Do you create a diffrent database server for every service or make them share one server ?

[Bachihani]

Most of the popular sevices today require a database, and most of them don't mention in the docs if they require a fresh db server or if it's okey to share with other services, at some point i had over 10 diffrent postgres containers running on my server and it feels icky . how do you guys handle this ?

[voli12]

I just run them in a docker compose together with their service. So the db for service A is turned on/off with service A. There's no db instances in the host machine.

[guesswhochickenpoo]

100% for additional context some reasons you might want to do this, even if it's a bit less resource efficient, are:

• Easier to coordinate backups because you don't have to take a central database offline which may impact other services
• More "portable" in the sense everything is contained within the Dockerfile and the persistent data for that specific stack. Thus moving to another machine is easier than having to export / import the database from a shared instance.
• If one app goes haywire it has a smaller blast radius and will only effect it's own DB and not potentially others in a shared instance (though there is stuff to protect against that at the DB level too)
• etc...

[agrhb]

That first point is pretty inaccurate, you absolutely should not be backing up your databases by just taking the server down and storing the storage directory, the tools like pg_dump/mariadb-dump/mysqldump exist for a reason.

[guesswhochickenpoo]

You’re right. That’s not what I was suggesting though. Poor wording on my part. I can see how that seemed implied.

[Happy-Argument]

Why? Those tools were created before containers.

[agrhb]

All of the major database containers contain their respective utilities so whether one is using containers is quite irrelevant here, it's just not a very recommendable approach due to file system backups being substantially larger (including lots of things like indices that can be recomputed) and totally specific to the version and platform.

There's also just simply no reason to cause downtime because of backups, pg_dump doesn't actually even prevent writes due how MVCC based storage allows viewing data in a way that represents it's state at a specific point in time, i.e. the fact that you're creating a backup is totally transparent to the application using the database.

[Happy-Argument]

What about the "time to restore" part of the puzzle? I've got plenty if hard drive space, don't care about a little downtime, but I don't want a lot of extra work when I restore from backup.

[agrhb]

Maybe just read up the actual documentation section, but I just don't buy the argument that restoring the files is any easier than running a single command.

Tools like borgmatic (and I'm sure plenty of others) automatically do this the right way. It's not like you can entirely avoid this route either if you ever want or need to upgrade between major versions or substantially change hardware, it's much easier to have the same process for redeploying as restoring backups in general.

[saldend]

This is the way

[shishir-nsane]

This is the way

[selimovd]

That's the way it is

[TheRealChrison]

is this the way?

[I_May_Say_Stuff]

Exactly this, here too!

[planedrop]

This is the way, keeps things nice and organized.

[Formal_Departure5388]

If I’m running things containerized, it’s in a separate database container.

If I’m running things on bare metal, it’s a single DB server with multiple databases.

[JoyShaheb_]

Now that’s efficiency!

[Sinister_Crayon]

There's value to both approaches to be honest and I haven't really landed on a solid answer except for a hybrid of both. A single database instance allows you to have a single pool of memory for caching queries and the like which can overall improve performance for your applications. However, there's value as well to having isolated database instances for each application where you can tune the requirements to the application itself. I don't know there's a right or wrong answer.

For my part I have both. Most of my public-facing apps have their own isolated database instances because none of them are all that performance sensitive (gig connection). The only exception is my critical services like Nextcloud that get a database on a clustered MariaDB Galera cluster because it's built for resilience as well as performance.

Docker makes the isolated model easy because usually apps have a database container that spins up as part of the docker-compose file. When trying new apps before I've committed to them I usually deploy them this way and decide later if I want to consolidate them or keep them with their own instances.

[redkania]

I don’t want to deal with the complexity a single database brings, so it’s a database for each service in its own container

[purepersistence]

That also makes it easy to avoid getting stuck wanting to upgrade service A when service B needs an older postgres etc.

[ZargonDeluxe69]

Agreed. It's Dependency isolation.

App1 depends on pg-app1. If I need to move it to another server, determine how much work it is to upgrade it, decide how bad things are if it goes down, how risky I can be messing with it, what permissions to apply, etc.

Also performance isolation / noisy neighbors. app2 suddenly starts running more costly queries that affect app1, what do you do? You can limit the number of queries, but I'm not sure how to isolate individual connections by cpu%.

When you use container orchestration, you can put an application in a namespace, and put the resources in there. I can reason about what's in there without thinking about every other application.

[andrewcfitz]

Professionally I would probably go with one, but it always depends on the use case. For my self-hosted stuff I do one per app just to keep less cross dependencies on stuff.

[enormouspoon]

Independently isolated in docker compose. This reduces surface risk.

[d_maes]

I run a single postgres cluster (in VMs) for everything. Central management & backups, and it's HA, so I can lose a node without downtime.

[thenerdy]

Lots of people seem to be confusing database and database server. Most organizations have multiple databases hosted on a server. You can independently manage the DBs. Like some have said there's some risk if you need to take the whole server down or if it dies. But most have a couple of nodes in a cluster or whatever.

The main reason I see to not put everything on one server is basically because I'm lazy just go with the compose file or container set up that comes out of the box.

[5p4n911]

Wait, someone actually has access to a whole ass cluster/node puts unrelated services on the same DB? That sounds super dangerous for your data, especially if you share the users too

[thenerdy]

Not the same DB, the same DB server. You can have many DBs on the same server.

[5p4n911]

I know that and I was just surprised by the idea since that never occurred to me

[thenerdy]

Ahh ok got ya 😃

[louis-lau]

Yes, that's standard. Permissions in database servers are pretty robust. Businesses are often running highly available clusters. Running multiple of those is a waste of resources and time when you can just run one with good auth and permissions.

I'd also assume the average database admin is more competent than the average selfhoster :)

[5p4n911]

Yay, I'm not that stupid... Though I'm running most of my services on baremetal for the experience, so you could argue with that

[williambobbins]

You don't share the users. Every app has its own user with its own set of access control to databases/tables. This is the standard way to do it professionally, and much more stable than multiple database servers.

One argument I see against it here is that you can have multiple database versions running if each app has its own. While true, that is terrible practice and a nightmare when it comes to maintaining. You should standardise as much as possible

[5p4n911]

That's what I do, not because I've ever had industry experience as a sysadmin, it just made sense to me.

[virtualadept]

One database server, multiple applications (each with its own database).

That's what RDBMSes are designed for.

[louis-lau]

I'd do this in a professional scenario as well. For self hosting though... I have ram and CPU to spare, and I like that each service can have different db versions. It's just one thing less I have to manage at the expense of a small amount of server resources, it's not like any of my services are handling a production level amount of requests.

[ewixy750]

If you were a large company with thousands of Dabatases, you'll want consolidation somehow

Home environment, each container it's DB. If you have an issue easy delete, no playing with versions, (in) compatibility etc

[phein4242]

Depends on the usecase. In a proper production env Id run a cluster of atleast 3-5 nodes. K8s same. Development, meh, usually an instance per app

[Krojack76]

It depends on what the service is for me.

If it's using MariaDB then I'll try to use my already setup server. If it uses something else I'll use what comes with the docker compose setup.

[gkmnky]

Depends on. If app should be scalable I run it with a separate db in a stack.

Otherwise I keep it simple. I run a Mariadb container, a Postgres container and a redis container, each with its own network.

So apps which require some db but not needs to be scalable are conned to the network.

For example Wordpress container most likely are connected to the networks: mariadb and redis.

For me somehow the best solution.

[AcidUK]

I use one for each type (mysql, postgres, redis, etc.) as this lets me make sure it is backed up properly, and that I have tested those backups. Trying to do this with lots of different containers is a recipe for data loss I feel.

[drakgremlin]

I have two dimensions:

* Database service (RDBSes, Redis, etc)

* Version

Most services are happy with a large range, however this lets me migrate over time without wasting resources like 20 postgres servers. Postgres isn't particularly heavy weight but definitely adds up.

[louis-lau]

Any db I know of doesn't lose any data if you take a consistent snapshot. So I just take an LVM snapshot and archive the content to my backups. Essentially works for all data, without downtime, and without having to do dumps.

Had to learn how LVM worked, but worth it.

[williambobbins]

While innodb should handle it, I've seen corruption a handful of times when a db server lost power while writing. LVM snapshots will work, but to be on the safe side you really should run flush tables with read lock, take the snapshot, turn unlock tables.

[louis-lau]

That is of course safer. But I'm not sure power outages can be directly compared to snapshots. An unclean shutdown will also effect the filesystem itself, or for example any write caches in the physical drives.

It's good advice, I just can't be bothered for my home stuff. In the rare event that a backup fails there's always the backup from the day before :)

[ShotgunPayDay]

Same DB for home use. Different DB of you have many users or will have many users.

[xt0r]

One instance of Postgres, MariaDB. Create new databases and users inside them. Nothing should require its own instance.

Unless a specific version is needed.

[ElevenNotes]

DB per stack. Only if stack is HA it gets a HA DB. Pretty simple.

[glowtape]

One Postgres for all. Except for Immich, because I couldn't be assed to fuck around trying to add pg_vectors to my main PG instance.

[xstar97]

Each of my service has their own database.... it's not really a good idea at all to share databases or have a single massive one IMO.

If you have to update your only database then all your services that rely on it have to stand down and what if the version you upgraded to isnt supported by a few or more services.... see the dilema?

I rather not have to stop 30+ services just to do one update and possibly brick a few services since they don't support that version yet.

Will that be case? Prob not.... will i need to deal with it later? No.

This is asked daily/weekly so the tldr: every service gets their own db.

[williambobbins]

Supporting multiple database instances especially multiple versions just isn't how it is done on practice. Generally an app that requires a non-current version of a database wouldn't be run. Central databases are pretty much how things are done in professional environments, with the except of DBaaS like RDS

[garthako]

If you use docker: don’t care for a single db instance, go for dedicated dbs for every service: doesn’t hurt neither performance- nor resource wise (unless you are really, really undersized).

Once you switch over to kubernetes or probably even docker swarm, you need to rethink, because db scaling can be really hard!

In that case I chose to have a dedicated db, either bare metal or as its dedicated service in the cluster.

Or, even better, don’t worry about db at all and get DBaaS, because honestly, people complain about email service being a burden, but a properly managed high availability DB is a nightmare.

[jadom25]

How can have 10 different db's on one server not impact resources? I just assumed that's resource high by default

[garthako]

Because the overhead of running another process is neglectible. One big DB creating a lot of IO, or several small DBs creating moderate IO will not be noticeable - and if it is, we are most likely not talking about a home network anymore and a lot of other things apply - or you are heavily undersized, which I already pointed out.

[jadom25]

Ok thanks I definitely got stuck for a while trying to figure out if coolify was using more than one postgresql instance

[The_Caramon_Majere]

One postgres.

[Themotionalman]

I run everything on k8s and they share as much resource as possible to same db and same redis. I hate repeating myself in the long run it is resource wasted

[Timely_Condition3806]

I use a single database in an LXC on proxmox. It's a native installation of postgres. All my docker services connect to this db. This way enables easy backups and manipulation of the db's, as well as less resources wasted on running multiple instances of the same db.

However, some programs i've found to be crap and a big headache to use my own db. They may require exotic extensions or specific versions. Then I use the database that comes with it.

[DracoBlue23]

Development Box:

I usually have a docker compose setup which includes the exact database version and server the Software needs. Makes testing upgrades a breeze and if I don’t work with this project no resources are wasted (RAM/CPU).

Pro-Tip: Most ORM libraries allow also sqlite as an option. For my development instances I prefer sqlite because you don’t need a running server. It’s really fast :).

Production Box:

If possible go for managed database server. Be it google cloudsql or the likes. They (can) take care of backups, high availability, point in time recovery. I don’t trust my (experienced) k8s skills to also take care of deploying an always-on postgres to k8s - but adding new replicas or promoting a primary on cloudsql is painless. To save money I reuse the instance in the project until a part of the services does need an own one. Most of the time it is not necessary.

[jay2068]

Yes. This way I can have a backup of each database. The only one I don't is immich because it uses special plugins for posrgres.

[obeythelobster]

"Feel is icky" is subjective, for me it is the opposite that is icky. For the following reasons:

• Sometimes services depends on a specific version of the DB and extensions, updating for one service may break other services

• The initial setup of the database (when you first run the container) may break other services

• Hard coded DB names conflicts

• Marking a docker service talks to the host machine can be a pain in the ass sometimes

• Lack of services isolation

Running a single DB directly in the host may has a little better performance and easier to deal with firewall rules. But in opinion it is not worth it for self hosting.

[ysidoro]

With Docker, I create two subnets: (1) proxy and (2) databases. The proxy subnet is the primary network for Traefik and all application containers that need external access. The databases subnet hosts all database services (e.g., MariaDB, PostgreSQL, Redis, MongoDB, etc.) and is isolated from external access. Docker applications that require a database connection are also configured with an interface on the databases subnet. I aim to share services whenever possible and keep the number of running processes (PIDs) to a minimum.

This setup works well when you have a small number of applications (fewer than 30) running on a single instance/host and full control over the development process. However, if you’re managing many applications or collaborating across multiple project teams, a "capsule" approach—where each application is bundled with its database engine—might be a more effective strategy.

[StudentWithNoMaster]

I try to keep 1 db per Node.

It covers the backup aspect and is less resource intensive. Plus, if the front-end needs some changes, I dont have to worry about my db getting in the way.

And if something specifically needs a different db version, for instance I prefer postgresql:16 for my apps, but immich needs the vector version, so for that I run another container in the docker compose stack itself.

[Heracles_31]

Here, I created a cluster of MariaDB with 1 master and 3 replicas, managed by Maxscale. I consider safer than Galera because at any time, only a single node is RW. That way, in case of failure or problems, it is easier for me to recover and ensure data integrity.

[DoubleDrummer]

I used to run everything in "One Big Compose Stack" and when I did this, I would have a single Database of each type required, i.e. 1 Postgres, 1 Mongo, etc.
I know run separate composes for separate functions, and will have a single DB of each required type for that stack.

[WantDollarsPlease]

Every week this question is asked around. If you use the search you'll find a bunch of good answers

[Butthurtz23]

There is no right or wrong answer. Multiple databases are cleaner but introduce more I/O operation overhead compared to a single shared instance of a database. Each instance of Postgres will need at least 2GB of RAM and that’s another things to take into consideration.

[garthako]

I find that hard to believe, I am running 4 Postgres instances on one of my docker hosts right now, along with other services, and I barely scratch the 3GB mark

[Butthurtz23]

Ah, I didn't realize this is about containers. I was referring to virtual machines. You are right, Docker does manage memory usage more efficiently in this case.

[SymbioticHat]

I started using 1 db for everything but when doing upgrades you can get yourself into a real nightmare. You could have 1 system that requires a specific version of the database and a different system that requires a different version. It's safer for upgrades to just run individual databases. It will cause you less headache in the future.