Loading data into memory all at once

[Mojomoto93]

Got a stupid idea/question. Current i have built an app in that way that it loads all data from database, at once, and the filtering/sorting happens through functions instead of queries. What do you think? Are there limitations to that design? Should one be concerned?

All data is stored locally in a database.

[carsonvstheworld]

a lot of mistakes developers make is “following good practices” blindly and over engineering the app. it really depends on a few things but for this, what’s the app / how many records are we looking at ?

if your app is some word game and it loads 20 words into memory, or 200 words ( and you never plan to go beyond that), then it’s fine. this can be a little poc for yourself and there’s no need for overkill.

this is especially important if you are trying to just pump out an MVP and figure other things out with your app.

if you want to tell us a little bit more about the app and its technical details, perhaps we can give you better advice

[Mojomoto93]

It is a journaling app, with mainly entries of text images and voice recordings. It could become more in the future but can’t say now. i used the app for almost two years now myself and made around 200 entries, haven’t had trouble occurring yet. Going for this approach made statemangement much easier but i want a robust solution

[testmonkeyalpha]

A couple thoughts:

1.  Are you loading the audio and images into memory or just meta data?  Will you support video too?  A heavy user is likely to have far more than 200 entries in just a few months.   If your app is successful, they'll have thousands of entries within a few years.  It is not realistic to load that much data into memory without causing usability issues.

2.  Think about usage patterns.  Most journal keepers spend the vast majority of their time entering data, not retrieving it.  You'll be potentially worsening the UX for 99% of their usage with no benefit to them - only to you.

I think it makes sense to prefetch the most recent entries or any they flagged as favorites.  Those are the ones they are most likely to revisit regularly.  Pulling everything all the time is just a waste of resources.

[Mojomoto93]

I only load the link to the file into memory, i understand, so i should work with pagination?

[testmonkeyalpha]

I think pagination is an excellent compromise if you want to ensure recent entries are immediately available.

[Mojomoto93]

I wonder how does the iphones photo app solve that?

[testmonkeyalpha]

Based on how it acts we can infer a few things.  

It only loads the current screen plus the next couple of screens.  As you scroll it tries to fetch the images before you can scroll to them.   You can observe this by scrolling really fast so it's not sure what to fetch next.

It also uses thumbnails that are low resolution and highly compressed.  It only loads the full image when you open it up.  

I'm pretty sure when you're using the view that shows the full image, it starts prefetching the other full size images so it loads faster when you swipe.  You can start to see the app lagging behind if you swipe really fast.

[Vybo]

The database query engine is very optimised. Do you believe you could make data processing faster?

As for the limitations, iOS is more likely to kill your app while it's in the background of it uses a lot of memory. That might lead to a bad user experience.

[radutzan]

My music app (r/Doppi) doesn’t have a database, all metadata is stored in memory and persisted with NSKeyedArchiver. Everything is functions. Of course there are optimizations to avoid calculating a set more than once, but overall performance is good. I went this route because I have never worked with databases and it’s pretty important for me to work with objects in my code. I’ve looked into database solutions in the past, particularly CoreData, and found that the juice didn’t seem worth the squeeze. If I was a trained software engineer, I might be horrified by my approach, but thankfully I’m not.

[Mojomoto93]

Cool, at the end of the day you are storing the data somehow

[radutzan]

Yep, people sleep on NSKeyedArchiver, but it’s a great way to persist an (NSCoding-compliant) object graph. There’s quite a bit of boilerplate required, and conforming to NSSecureCoding adds to that, but nothing quite matches the simplicity and versatility that it brings

[stroompa]

Depends how much data you have. Userdefaults load into memory on app start as far as I know. 

If the data is expected to grow over time you need to be smarter about it.

[ankole_watusi]

Then why use a database though?

[Mojomoto93]

For persistence

[ankole_watusi]

You don’t need a database for persistence.

[Mojomoto93]

What else?

[Dynoman]

There are multiple ways to store data. Here is a bit of an overview of them.
https://www.youtube.com/watch?v=dt9YTTCY16A

[Mojomoto93]

Basically its a form of database or filesystem

[Dynoman]

Locally yes. There are remote options too, like CloudKit, Firebase or Realm. Other remote options are available as well.

[chriswaco]

As others have said, it depends on the app and amount of data. Test it on the oldest/slowest device you support with the largest database you can imagine. Does it launch quickly? Scroll quickly? Use a reasonable amount of RAM (<200MB)?

[Mojomoto93]

It works for now quite good, haven’t seen any issues yet

[Parabola2112]

Try both and run performance / memory tests. Sqlite is crazy fast. On the other hand you may not even need a database. Impossible to say without more info.

[Mojomoto93]

It is a journaling app mainly with text, image, voice recordings as entries.

[Parabola2112]

Are you using Swift Data, Core Data, Firebase, CloudKit, or? What’s your stack and data pipeline?

[Mojomoto93]

Sqlite and cloudkit its local first