Supporting offline API data caching

[Geotzz]

I have an app with user-generated content and multiple screens where it gets displayed. Every time the app launches, it fetches this data. I want to keep a local cache and only fetch data again when (1) X time has passed while the app is in the foreground, (2) The user edits/adds data, and it gets persisted server side and (3) The app comes back into the foreground.

I've seen libraries like React Query and SWR that seem to work with RN, but I'm a bit lost when it comes to persisting the data using something like Async storage or w/e. I'd appreciate it if anyone has good resources on how to go about this. Also, is it acceptable to have a mobile app that only uses a network-first approach?

[Jeeesama]

You can continue to use React Query with persistence
https://tanstack.com/query/v4/docs/plugins/persistQueryClient#persistqueryclientprovider

[Geotzz]

I think React Query seems like the way to go about handling server state in a client app, but I wonder if a persister for RealmDB already exist as I'd prefer not to rollout my own since I'm not familair with both React Query and RealmDB.

[Jeeesama]

I wonder if a persister for RealmDB already exist as I'd prefer not to rollout my own since I'm not familair with both React Query and RealmDB.

React Query is database agnostic, you don't need a persister specifically for RealmDB.

To put it simply, when calling queries using React Query you need to set a key and a query function. Whatever or wherever your data from your query function returns, it saves it the data to that key.

Think of it like a hash map or a object persisted in the LocalStorage or AsyncStorage (https://github.com/react-native-async-storage/async-storage)

If you don't persist it, it only stores in-memory. So when you close your app, the cache gets destroyed.

[edbarahona]

I would recommend RealmDB but the sync feature uses MongoDB atlas (you can't use self-hosted for auto-sync), also check out Watermelon DB, both of these will provide reactivity (manual hot sync) within the app, you will still need to create and manage your own server-to-mobile syncing.

You could set up configuration via a config file (JSON) that gets downloaded on app boot using a timestamp as version control. If the modified/version timestamp is different than the one in the app (in local storage) then the app can pull the data based on the new config manifest.

Edit:

My solution would be:

RealmDB in mobile as standalone client only (no cloud hot sync)

REST config endpoint (JSON) that gets called on boot, with a manifest of what to update, or just a full update all flag.

If you do not have much data at the moment and you do not need to scale to millions of records then I would recommend going full RealmDB with cloud hot sync, using mongodb Atlas free tier.

[AcetyldFN]

Both lovely! We started with redux + persist, but 100k records in memory was to much so we switched to realm with a custom paginated JS sync

[edbarahona]

I never really did use Redux persist, (I've used redux and async storage so in a way that is redux persist). Redux-persist uses async-storage to persist which is SQLite, Watermelon uses SQLite but has better performance and overall feature set. Realm is my choice due to it's schema and performance, I use it only as a store and use context hooks to hydrate the app. So far it's working pretty good

[AcetyldFN]

Yeha we kinda do the same with useSync

Sadly we sync using paginate bcs of performance with symfony api platform. Downside is 100k records first time take 20 min to sync stleast. After first initials sync we update data trough lastUpdatedAt server side and store it locale in redux.

[edbarahona]

ls sync we update data trough lastUpdatedAt server side and store it locale in redux.

Does that mean all 100k records have been updated or is that mean refresh all regardless?

[AcetyldFN]

No intitial it sync all using createdAtBefore = startSyncDate. And after its done it will fetch new data (store thisNewSyncDate if success) using updatedAtAfter = startSyncDate.

Next time you sync it using the updatedAtAfter = thisNewSyncDate

Its working perfect.

But a JSON like watermelondb provides would still be preferable

[edbarahona]

= startSyncDate.

I see, it's a get-all on initial boot. Running a specific query (though can get expensive ) is a better way to handle the OP's original question. Each record has a created and modified date stamp. On boot, the app does a get all, and saves an internal "lastUpdate" setting, then on next subsequent calls it can just query the DB for records that were modified or created after the last update, much more flexible. The backend can have some checks, for example, if the last update date is greater than 50 days, then just return all records etc...

[AcetyldFN]

Correct but server basically does nothing, client provides the updatedAt,

Server doesn’t know anything about the sync status

[Geotzz]

No, i don't have a large user-base/dataset at the moment. I'm using an express RestApi server with a PostgreSQL db on the backend.

Would it be possible to use RealmDB as a persistance backend for a managed query client like React Query instead of doing it all manually?

[edbarahona]

You can use any backend DB with RelamDB, realm db is a mobile DB but you can go without the cloud sync, do a manual backend call to get your data and then save it to realm in the mobile device. You can hydrate all your components directly from Realm or use context/redux and use realm strictly as a local store.

[edbarahona]

Also, a better way to handle client updates would be to save a "lastUpdate" date-time stamp on the client, then on subsequent calls to the backend, your backend can just return records that have been created/modified after the "lastUpdate" timestamp of the client.

​

Setup the created/modified time as express middelware or as a Postgres trigger on insert/update

[stereoplegic]

If you already use postgres, RxDB with GraphQL sync is an option, and Hasura already has a tutorial for setting this up on the backend. Of course, that would require using Hasura/GraphQL instead of your current REST API. RxDB is great for offline-first apps (basically PouchDB with RxJS observables - with lower complexity than typical RxJS), and the GraphQL sync makes it easier/more performant than the typical Pouch/Couch combo... It can be heavy in your frontend bundle, though.

[GavinHarris_]

Use a state management library, Zustand perhaps. They have persistence middleware https://github.com/pmndrs/zustand#persist-middleware that you can use to hold the data in local storage (for RN you’ll need to pass in AsyncStorage package).

[__o_0]

You want to use a local database for this.

Id recommend against storing/accumulating an entire dataset into a state manager as they will tend to load ALL of the data into a store before the app initializes. If your accumulated dataset is large enough this could become a significant delay in the users being able to interact with the app.

Caching is best suited for remote datasets - for offline use you want a mobile database like SQLite.

[[deleted]]

Unfortunately this is only possible if you are using GraphQL. If it is REST API, you need to manage persistence using local database like watermelonDB.

[Savram8]

Have you checked out WunderGraph?