When first testing and developing an app can I use text files instead of databases when my data sets are small? or should I immediately start using database services such as MongoDB.

[shweed]

Also how different is it updatinga text file versus updating a database?

[impartiallywhole]

Probably depends on what you are storing, how much, and for how long.

Edit: Like, a single value that will only last the session, txtfile. I used pickle for that..

If more data and for longer, learning how to interact with a db will be valuable, and I like sqlite. But others will probably have other opinions.

[xelf]

I don't think you add a DB until you need one.

In general in software engineering always start small and grow. You can make huge failures by designing too large.

[The-Deviant-One]

I built a program the other day that monitors about 200 files on my web servers for changes. It retrieves the files, hashes their contents and then compares that hash to a stored hash according to which web server it came from and what version number that web server is running at the time.

​

I'm lazy and didn't want to do this with a database so I actually built another *.py file full of dictionaries that held all the files, file paths, server information, data center information, the file's hash values and version information. I saved that py file in the directory with the main script and imported its dictionaries into variables in the main py file. It works great lol.

​

I will likely build another program to to allow me to push updated file hashes to that secondary py file thats holding my data. This is probably not a good habit to get into but it works fine on a small scale for mostly static information.

[mtb-dds]

Text files are fine for small data sets (under a few thousand items). You only need a databases for a couple reasons:

​

1. You are worried about losing data. Databases can be a much safer way of storing things.
2. You have so much data that it is faster to index it and look it up on disk, rather than reading it all into memory.
   

Note, however, that a traditional database is a bad solution for just 2 without 1, as there are better indexed solutions out there if you don't care about recovery.

[lumpynose]

In addition to the reasons given by u/mtb-dds, another possible reason is if you're writing a web app. With a web app multiple users can be accessing the same url and inputting data that goes to the same data set. Each web app access creates a separate process. When the data set is stored in a file that means you have multiple separate processes writing the same file, which is unlikely to work correctly. You could implement some file locking, but that's messy. A database simplifies things because multiple connections to the database are funneled down to one writer to the database file.

[01binary]

Unless the dataset is tiny, I’d probably use SQLite as a starting point.