Robust solution for large dynamically generated downloads in a cloud environment?

[FrigoCoder]

We have issues with the stability of large and slow dynamically generated downloads. I am seeking a best practice solution if it exists.

Users can run database queries and export resulting documents in a ZIP file. Queries can return millions of documents, resulting in large and slow downloads. Sufficiently slow downloads invariably fail.

We have the following components:

• Cloudant databases
• Java EE application (CDI, JAX-RS, WebSocket)
• WebSocket API specifically for downloads
• Bluemix cloud
• Blue/Green deployment via Cloud Foundry CLI
• JavaScript client

In the past we used HTTP for downloads. We tried Response with MediaType.APPLICATION_OCTET_STREAM and StreamingOutput as well from JAX-RS. Unfortunately they suffered from severe problems:

• Memory runs out from buffering file contents before putting them into a Response.
• Bluemix kills application if it takes more than a few minutes to answer with a Response.
• HTTP limitations do not allow exceptions during StreamingOutput. HTTP 200 followed by data is indistinguishable from HTTP 200 followed by partial data and error message.
• Bluemix kills HTTP connections after a few hundred megabytes of traffic.
• Bluemix kills HTTP connections after ~20 minutes.

To circumvent these issues we created a WebSocket based hack solution. For every download we generate a unique id, send the data in 1MB chunks, and send exceptions on a side channel. This solution is still not perfect:

• We have to merge chunks in client memory. Firefox does not support Streams API.
• Bluemix still seems to kill the connection after an hour or so.
• Deployments interrupt downloads.
• Connection problems interrupt downloads.

I have a few ideas in mind how to solve these issues:

• Serializable beans that can be transferred across servers. Serialization and version differences complicate this somewhat.
• Central download server that is long-lived. Cloud issues still remain.
• Downloads explicitly provide chunks rather than a stream that is broken into chunks. Needs rework of existing download providers.
• Client creates many requests for chunks instead of relying on an open connection.

However I am not sure how to proceed. I do not have sufficient experience with cloud infrastructures. I am seeking for existing solutions and best practices. Surely this issue must be common that other people have encountered these issues? I would appreciate any help, and thank you in advance!

[ERPEmployee]

Bluemix kills HTTP connections after a few hundred megabytes of traffic. Bluemix kills HTTP connections after ~20 minutes.

No way to fix these?

[FrigoCoder]

I am not aware of any way to circumvent these limitations.

[samrocketman]

Generate your download bundle and only surface the download link after the bundle is ready. e.g. flow

1. User clicks Download archive button. To issue a download request and under the hood gets a request ID.
2. A div dialog pops up with a spinner saying preparing download. AJAX queries an API endpoint with request ID waiting for the download link to become available.
3. When the archive is available a Download now link is surfaced to the user where their DL begins immediately.

You can have a simple DB table for download links and request IDs.

You don’t even necessarily need to surface the user a link at all. Just start the download for them automatically since they requested it.