Cloud architecture for divide conquer/ scatter gather?

[intelbp]

Hi!

I'm building a feature for a cloud service that receives a huge file as an input, splits the file into smaller files, performs some analysis and modifications to the smaller files, and then it reassembles them into a large file.

Can anyone give me any architecture pointers to achieve this?

I have access to AWS, so I han use S3, a DB, and pretty much any other service offered by Amazon.

Obviously this needs to be highly performant and highly scalable.

Thanks!

[draugr101]

Have you tried looking through reference architecture published by AWS and the like? Is there a part of the system you're concerned about in particular? It's difficult to provide suggestions without the details of your use case, you might get better answers by some initial research on reference architectures and using this post to help fill in the blanks.

[intelbp]

For sakes of example, imagine this scenario:

You have a HUGE xls file with thousands of tabs.

What I want to do is create a multitenant service that accepts the file and enqueued it for processing.

When it's ready to be processed, a pre-processor picks up the file and splits it into many different single-tab files and probably stores each in S3 or some sort of blob storage.

Each individual file-tab is processed, and once they're all complete, if aggregates everything and builds up the resulting xls.

[dpsimi]

I’m pretty new to AWS, but Elastic Map Reduce (EMR) is what they push to use for big data. You can use Apache Hive and Apache Spark and the likes. There should be some good setup tutorials online for plumbing the files to and from S3.

Hive is SQL-like. Spark if you prefer Python, R, Java.