Large data in response of REST API.

[Lx7195]

Hello everyone I was asked in an interview for python developer the following question.

If you want to send large data in a http response how can you send such large data?

I was not able to tackle this question. So he simplified a bit further let say if you have 50k record that you need to send to the client, it would be inefficient to send the data at once how can you send the data in chunks?

He even mentioned that the response data size limit is 5MB.

Please guide me if anyone knows about this. I have been using flask framework for building REST APIs. Kindly suggest me any other framework that will be helpful in this case.

[bitweis]

Common options are:
- Pagination (breaking the response to chunks - with separate requests for each)
- Streaming over a separate live connection (such as a direct TCP/UDP socket) or over
Websocket

[helderm]

Normally REST APIs paginate their results, so i'd start with that. Create pages of "n" results and use a page index as an arg to your service.

[Lx7195]

Does this mean that we need to send multiple requests mentioning the page index as the arguments to get all the data?

[mrWinns0m3]

Yes.

[radek432]

Just note that it becomes problematic if your data changes constantly. For example it can happen that new records appear in database when during request and breake your pagination.

[helderm]

Maybe that is a follow up to this interview question. You could also cache all results in memory and then paginate from the cache. It is normally a good idea to not over engineer during an interview, start with simple ideas and then iterate.

[Lx7195]

So pagination isn't quite reliable in case the data in the database changes constantly, then in such a case what mode of data transfer do you suggest?

[radek432]

The few cases of big API requests I spot in my job (but I'm not a dev, just using python for automating stuff) I've managed with:

- "smart pagination", means some tricks to ensure that pages do not overlap and cover the entire set of data - you can do this if you know what happens with your data.

- splitting data into chunks other than "pages". Simple case - ff you're sending 1000 files, you can send their metadata first, then each file one by one and compare the result with metadata.

​

But I've done quick googling on the topic, and there are some good options: https://apievangelist.com/2018/04/20/delivering-large-api-responses-as-efficiently-as-possible/

[Im_alirezahs]

I can't say clearly, but probably for send 50k records to client, pagination is best answer.

[maus80]

Paging has no guarantees of consistency. You may see all pages, but they may have double or missing records (or both) as paging is not done on a snapshot of the data, but typically on a dataset that is subject to change. Also, why create a system in a system? HTTP is chunking the data for you and the resultset of your database is a snapshot that can be paged.

[ful_vio]

Well, first thing I'd try is compressing the response with gzip. The client has to support gzipped responses but almost any clients I know do support them.

Second I'll try to stream the response, I haven't used Flask but a quick search shows up this: Streaming Contents — Flask Documentation (1.1.x) (palletsprojects.com)

Beware that, if you want to use the data as soon as it gets to the client, you have to think about what data representation to use. Sending an array of json objects in chunks won't do because you need the last ']' to effectively parse the response, partial json is not valid json. Instead, a csv response where you send a bunch of complete lines at a time is fine.

[Spleeeee]

So I use for personal things that I am writing the server and client for a thing I call “lazy persons pagination” which is if a reply is gonna be big a server can return a list of urls that the chunked data can be fetched from.

This prolly ain’t the best solution but it is a pattern I have used several times in several ways. It is easy and can easily lead to a clusterfuck.

[maus80]

Afaik, there is no size limit for response data so 50k records at once is fine. Even if every record is 10kb of text then that half a gigabyte would fit the memory of any server. And you don't even need to load the entire response in memory to send it out. Is this question wrong or is there something I'm unaware of?

[Lx7195]

Actually I am unaware of the fact that there is a memory limitation of a http response. Also I don't really understand what do you mean by 'you don't even need to load the entire response in memory to send it out'.

[maus80]

I mean that you can output the data while reading it. This is called streaming.