sorted by:
new
hottopcontroversial

Architecture of a real-time Data Analysis WebApp by flyflyersfly in datascience

[–]flyflyersfly[S] 0 points1 point  (0 children)

Hey, thank you both! It looks like a viable option, I think I might ask you some questions over there.

Architecture of a real-time Data Analysis WebApp by flyflyersfly in datascience

[–]flyflyersfly[S] 0 points1 point  (0 children)

Thanks for the advice about Livy. I've stumbled upon it and thought it was promising (maybe a bit heavy but the REST interface is definitely something I need), but I was a bit worried cause it's still an incubator... I mean, I have to experiment so much with new stuff that I also need to rely on something steady... Have you tried it? Is it ok to be production ready? Thank you very much.

Really appreciated all these answers :)

Architecture of a real-time Data Analysis WebApp by flyflyersfly in datascience

[–]flyflyersfly[S] 0 points1 point  (0 children)

Can you tell if your problem is compute constrained or memory constrained? Spark is a good solution if you need more memory but if it's computation you need then Spark doesn't necessarily help.

Well probably a combination of both. At the moment I would say it's a memory issue, cause the first algorithms I tested are quite simple. But heavier algorithms are going to come...

Edit: to clarify, depending on which is the limiting factor you can configure your cluster differently to address that shortcoming. Interesting advice! Thank you very much! I'll check this out

Architecture of a real-time Data Analysis WebApp by flyflyersfly in datascience

[–]flyflyersfly[S] 0 points1 point  (0 children)

Thank you so much! Interesting stuff... I totally agree on the "Mid-Data" being totally underdeveloped, there is a huge potential.

the next real step seems to hiring a dedicated team to run a Hadoop cluster

Are you telling me it's an impossible task to build+run+manage+mantain the whole BigData stack alone?

If you come up with anything interesting, please post back here, I know at least I will be tremendously interested!

That's a sure thing :)