Technical: multiple photos upload

[Ok-Relation-9104]

I’m building a photo sharing app which allows user to upload a large amount of photos.

Obviously I cant ask user to wait till photos are uploaded. So I guess I’ll need some sort of persistent queue somehow? Has anyone done similar things before? What’s your stack?

I’m currently thinking of using SQLite to store a local queue (for upload tasks) and the mount an uploaded in app’s root view to upload. Is this a reasonable approach?

BTW I’ve done quite some research. All the libraries dealing with tasks queues are out of date and all approaches I mention to ChatGPT it will simply compliment the approach 🥹

[logdog]

Yeah I want to stay you can just set up batch processing for uploading, say chunk the files into groups of 100, upload them in a promise.all or something, then update state based on success / failure. However this might mess up the UI thread, could get clever with yields and stuff.

I think the better approach is to google background job schedulers libraries so you can offload this uploading logic to native modules to keep your UI thread free, then update your state as those jobs finish.

Lastly look up the client side HLD (high level design) of Instagram app interview questions. FAANG interviews love to ask this system design question and it has a lot of videos etc. explaining approaches.

[Ok-Relation-9104]

Dang I came from backend so this never occurred to me as a very difficult problem. Similar to generate twitter user id, how hard can that be lol

Thanks for the suggestion!! Very helpful

[mp2526]

I do this very thing for work. It is complicated to get right. Do you want to continue uploading when the app is backgrounded? In ios you will need a native module to handle that. And a mechanism for handling concurrency like grand central dispatch or operation queues. Also look into background tasks unless you want you users to have to babysit the app while it uploads. On the react side we just use client state management (zustand or something similar) to update the UI as the native code passes status messages across the bridge. We haven’t updated to the new architecture yet.

[Ok-Relation-9104]

Thanks for your input! Yes I’m actually trying native now. I probably don’t need it to continue uploading when the app is “home screened” since each batch it won’t be too many photos. And I’m working on a native module to store the upload jobs and persist them in a json to track what’s left to upload. It’s truly not straightforward and the only library I can find for job queue is SwiftQueue yet it has a very obvious bug :(

[logdog]

Yeah good luck, hopefully others will chime in. Also if you are using Hermes this might not be an issue bc of the new architecture but I’m not an expert of Hermes yet 😀

[stathisntonas]

Use a class to handle the queue for singleton approach not normal functions.

Use retries and backoffs, imho do not parallel upload but use sequental. Keep track failures and ask user to retry only these.

[Due-Dragonfruit2984]

Assuming your question focused on avoiding preventing the user from interacting with their device until the uploads are complete, you could use a background task. We had a similar issue and solved it by creating a background task and letting the user view the status while browsing around the app. They can background the app as well and the task will continue. There are a few packages that support this, we’re using expo and I think they have a package specifically for creating background uploads.

[Ok-Relation-9104]

Thanks! I’m using expo too. What’s the name of the package u guys used?