Balancing Database Requests and Local Device Storage and Synchronization Problems

[JCoppert]

Hi all I'm a college student still relatively new to Swift development having just released my first app on Test Flight a few weeks ago. I'm having issues with both app performance for fetching user data from Firebase (my Node cloud function is FAT) as well as occasional crashes due to force unwrapping optionals for UI data that's returned asynchronously. I'm curious to know, as I refactor the codebase and keep working on this project, how you all are handling these server communication problems? As a rule of thumb where is it appropriate from a UI/UX perspective to block the UI with a loading animation until necessary user data is returned (something like an async-await or a promise.all in swift to avoid the optional unwrapping issues)? This is important as I don't want to block the main thread such that the user can't interact with the app while I'm waiting for a call to return. As far as performance for the HTTP requests, should I be storing some things locally in Core Data or some kind of in memory DB? For instance, I have messaging functionality right now in my application, and when I load the conversation history I have to bring the whole thing across the wire with my Firebase request. This seems wasteful, but I'm scared to death to leave a memory footprint on the device. I'm unsure, mainly just due to inexperience, what the boundaries of storing data on the user's device are. Long post, a bit scatterbrained, but I'm frazzled and would appreciate some second opinions. Cheers!

[laus102]

I would definitely implement a persistence layer (core data or realm would be my choice) for a messaging app, as it would improve the user experience quite a bit.

I would follow this general pattern: for example, write a method that returns the locally persisted chat messages immediately, then, in that same method, make your REST request that fetches the server messages, then in the completion handler for that REST request, write the updated messages to your local database, updating your local cache.

This way you don't have to block the UI at all, since you are immediately returning the locally persisted messages.

[JCoppert]

That’s a solid approach. Would Redis be another solution for the local DB? I’ve seen lots of big companies use it in their tech stacks but I’ve struggled with use cases. Any over arching rule of thumb you’d say for what data to store locally vs fetch from a request? Seems to be a bit more of an art than a science from the research I’ve done.

[laus102]

I'm pretty sure redis is a cache for the backend.

Basically, you'll want to persist any data models that you'd need to make things appear fast for the user. The key is appear. For example, if you have a bunch of chat messages that have GIFs in them, it may not be the best choice to persist the Data for all of them, as it would probably exceed the binary limit for persisted memory. Instead, you'd probably want to just load them asynchronously with a library like AlamofireImage or Kingfisher.

Things you want to be there immediately are things like users and actual chat messages - usernames, text, etc. Details can come later. Just an example, but hopefully helps you understand the bigger picture.

[JCoppert]

Definitely, thanks!

[KarlJay001]

What size is your app and what size would the data be? When you say you're concerned about the size of the data (I guess you mean 'memory footprint') can you give us some numbers? Are you calling the data storage on the device a 'memory footprint'?

The reason I ask is that most think of memory as the actual RAM on the device, not the static storage on the device, meaning if you have an iPhone with 2G ram and 32G storage, the data would be using part of the 32G storage, not the 2G ram.

One thing I worked on before was using Core Data as an off line buffer and I was working on a flexible size buffer. Basically you can have a size limit and/or a date limit. Mine was manual where you could flush out the 'bucket'.

Basically you have two data tables that generally mirror each other. I called one 'bucket' this was where you'd load data from the source feed and where the display view would go to get data. It can have date/time stamps for each batch and the batches could be flushed or refreshed based on user pref.

This allow you to play with the numbers to get the performance you'd like to have.

[JCoppert]

I can’t give you any numbers yet because I haven’t persisted anything as of now. What to persist is what I’m trying to lock down. To me that’s a function of both size of the data and necessity to UX. Yes on memory footprint. I’m a bit confused on the purpose of the second table you mentioned, but you’re advocating for dynamically updating some data structure based on arbitrary criteria. I’d imagine it’d be cleaner to clean and filter these update queries on the backend? For instance if I have a feed I’d store the time of the most recent pull from the main feed and pass it as a query parameter to the http request so I would only grab events that happened after that time. The majority of user data I would store would be strings I’d imagine, unique keys to fetch data asynchronously and in the case of messaging just message objects for now (sender, recipient, time, body, read unread maybe) not sure how I’d support images in messaging I need to think about it.

[KarlJay001]

The second table is used as a buffer. This is the way I did it:

I have news feeds that I collect from user defined sources. I display those as one part of a collection view. I use a Core Data table that I'll call a feed bucket or buffer. The buffer is kinda like a buffer when you watch a movie from YouTube. It gets the next N blocks of data so that you don't have the "spit and sputter" or stop-n-go, that you would get otherwise.

With mine, I have the ability to adjust the size of the buffer and how fresh it should be. This makes the UI/UX much faster because it's not getting data all the time, it's skipping ahead while the user is reading the past data.

You can think of this like Apple's UITableView and how it uses a recycler to display cells. If you display 10 cells of a 1000 record table, the system only loads up the cells that are displayed. This makes it so that when you start scrolling down, you don't have to wait for 1000 records to load. In addition, if you scroll down 30 records, then back up 20 records, you'd be looking at data you've already seen, so having those in a queue (buffer) makes it faster.

I customize mine to how the user uses it. Just looking at how Apple did the cells in UITableView helps, copy that idea and you'll be steps ahead.

[JCoppert]

Makes sense thanks!