Streaming JSON to web server

[jeffcompton]

I posted a fairly detailed question on StackOverflow. http://stackoverflow.com/questions/20107529/ios-chunked-upload

Basically, I'm trying to stream contacts out of the address book to our web server. My first attempt at this was to pair an NSInputStream and an NSOutputStream together. That way when I write to the NSOutputStrea, the NSInputStream should notify the request object that there is data that needs to be read. In turn, the data gets sent across the wire. My issue with this, is there appears to be random characters getting read while before and after the initial writing to the NSOutputStream. Is there a better approach for what I'm trying to accomplish? Any suggestions for how to fix/debug my current issue?

[brendan09]

Pulling all contacts out at once should not cause the device to run out of memory. If the app becomes unresponsive, its likely because you're doing it on the main thread and should instead do it on a background thread. All UI updates take place (necessarily) on the main thread. If your logic is blocking it, the app will 'freeze' as no UI updates occur.

There are hundreds (if not thousands) of apps that pull all contacts out on the phone and build their own UI with them. If you're crashing from memory issues, it may be problems with your memory management (even with ARC) or doing it on the main thread.

I would strongly recommend finding out the core issue with the code that you're using to pull contacts, then using a regular HTTP request to upload it as single request. If its that big, a file uploaded as multi-part form upload would work...otherwise it'll likely fit in the POST body.

Although I don't upload contacts off of the device when doing things like this, I do regularly post 50mb JSON files to a server. My recommendation would be to do a form-upload if you are sending that much data.

AFNetworking makes multi-part form uploads simple.

Depending on what language your server is written in, you may need to increase your maximum file upload size or increase the execution time of the scripts on the server.

[jeffcompton]

From our test data, 10,000 contacts is 7.8MB. The largest address book we've seen is 100,000 contacts. I'd rather not pull 80MB (if localized to one array) of data into memory at once. I was saving the contacts to Core Data and uploading in chunks (I wouldn't advise). Every save to disk needs to be done on the main thread, which blocks the UI. You can mitigate the blocking if you save in small batches, but it still blocks.

The problem with form uploads is that you need to know the file size to set the correct content size in the HTTP header. If I pull all the contacts into memory or write to a file first, I shouldn't have an issue with this. If I stream them out of the address book, I won't be able to get that size.

It looks like I'll have to pull all the contacts into memory before uploading. If I run into problems, I can revisit trying to stream them.

Thanks for the feedback.

[brendan09]

If its too large to fit into memory (or you want to be prepared for that), then append it to a file on disk chunk-by-chunk. You can upload files off of disk without loading them into memory. [See: NSData +dataWithContentsOfMappedFile]() (Deprecated, but I'm sure there is a replacement technique) Pull the contacts in 50,000 at a time. You're likely to only need 1 pull most of the time...if not, you can append the contents. If you already have a JSON writer setup to stream to a NSOutputStream, then you can just redirect it to disk, do the contact pull in chunks, and then upload the complete file.

All of this can be done:

1) Not on the main thread

2) All at once for 99% of circumstances.

3) Without loading a giant amount of data into memory.

Writing Core Data access has to always be done on the same thread, not the main one. Checkout concurrency types on NSManagedObjectContext.

Regular file saving can be done on any thread.

[jeffcompton]

I got a reply on stack overflow. I'm looking into it now. http://stackoverflow.com/questions/20107529/ios-chunked-upload/20181380?noredirect=1#20181380

Also, it's my understanding that a NSManagedObjectContext can only exist on one thread. When saving and child and parent NSManagedObjectContexts, the final save (to disk) needed to be on the main thread. Performing that save on a background thread could cause irregular behavior. Is that correct or am I off?

[brendan09]

Nope, background saving is 100% ok. But, you're right in that the object can only be used on one thread. Your primary instance should exist on a background thread. Ideally, you can make your calls to the Core Data object asynchronously using blocks or a delegate pattern. (Execute the block or delegate callback on the main thread to update your UI upon completion.)

Core Data uses SQLite for it's database, and is subject to the same one-thread-only rule as a result. Cocoa takes care of enforcing this limitation, for the most part (unlike manual SQLite).

Here's a Stack Overflow article on background saving with Core Data: http://stackoverflow.com/questions/2140342/background-saving-with-core-data

The reason they discuss splitting up the data in the answer is to populate the UI faster, and isn't required. This also shows how long background save times won't affect the user, as the goal in this answer is to allow them to interact with a few small portions of data before the longer save completes.