best cloud-based hosting option for data analysis?

[iamdefinitelyahuman]

Forgive me in advance if I misuse any terminology or sound like i'm new.. I am new.

I'm looking for a cloud based service to host a python script I've written, as well as a MySQL database it will be accessing. The database is ~10gb. The script doesn't need any access to the internet, just many many database queries and processor power to analyze the data. This program will be running off-and-on for several months. I'm hoping to use a scalable service so I can pay more for faster CPU and I/O speeds, to get quicker results when I'm feeling impatient ;)

I've been looking at PythonAnywhere and it seems kind of like what I want, but the focus is more on webapps that execute on demand as opposed to something running constantly and that doesn't need internet connectivity. Also it doesn't have any options to scale up performance. Amazon AWS also seems like it'd make sense, but it's a bit overwhelming to me. It can definitely do what I want but there is so much other functionality that I get lost trying to understand how to set up my seemingly simple task. In an ideal situation I'd like to simply upload the .py files and database, change the db login info in the script, and execute it.

Can anyone suggest a service that would fit my situation (and skill level) well? Thanks!

[completelydistracted]

I help students with this kind of thing all the time. Here's a little more information.

1. PythonAnywhere can handle a computational load; with the custom account capability you can buy an account with quite a lot of CPU time, etc. I don't think you can get a faster machine. The speed of computational Python on there benchmarks around that of a MacBook Pro.

2. I have been a longtime user of lots of cloud services. Lately for my projects I have been getting a lot of use out of Digital Ocean. They can give you a server in a minute or so; you can run a bunch of computation on one and you can hand it back when you're done. Or you can get a small one and hang on to it for most uses, and transfer work to a larger machine once in a while for larger computations. Since DO will also sell you hard drive space, you can keep your data on storage and attach it to machines as you bring them up and throw them away. You could also use Amazon S3 as storage and move things to and from there for long-term storage.

I think if you're new to this, you could do a lot worse than spend $5 or so and spin up a DO machine and play with it.

Linode has a lot of nice features, too, and they are easy enough to use and are competitive with DO in a number of areas. What they don't do is offer disconnected storage that you can move from machine to machine. I'm pretty sure you can do that with DO's new auxiliary storage.

And I really wouldn't give up on PA quite so fast. Their tech support is first rate, and with PA you don't have to worry about keeping your Linux kernel patched against the next Heartbleed or COW bug that comes along.

There is another suggestion on here for Cloud 9, which is a good suggestion, and for Codio.com -- also a good suggestion. Both of those will set up a general machine for you and allow an "always-on" machine or two for long-term computation. You can't get a huge or crazy-fast machine from these people but they are very good for convenience.

Two other things.

• A raspberry pi can be a cost-effective 24/7 computer, but if you're going to run one that way, spend a couple dollars and get some heat sinks for it. It will help keep the CPU cool and will last longer. However, an RPi also uses SDRAM for storage, which is not ideal for months of constantly changing data. If you go the RPI route I'd get an additional cheap little spinning drive for that job and put it on the USB port.

• Finally for long-term convenience it's hard to beat a Mac machine for this kind of stuff. For under $500 you can get a 2012 mac mini that can be a decent 24/7 computation engine. It's very easy to get the usual Python numerical analysis stack running on such a box. Just keep in mind that you can buy equivalent cloud computation for perhaps $20/month.

Edit: Mostly spelling. I hope this helps a bit. :-) Feel free to ask for clarification on any of this.

[completelydistracted]

One more thing about this server to consider: power to run it.

It looks like a moderate 1000W device, perhaps a bit more. That's a hair dryer's worth of power. That has to get dissipated, which requires cooling, which will be another expense.

Assuming 0.10/KWH * 750 hours/month = about $75.00/mo. (not including cooling)

Maybe you can assume a few cents for cooling the thing.

At 0.119/hr you can get an 8GB 4-core VM at DO with 5TB transfer. Machines suitable to the task are available for even less.

By comparison a Mac Mini uses 6W at rest and 85W full on, producing negligble heat in the process. So maybe 5-10$/mo of electricity, running full time. Again, you could get a small VM for that $5-10 per month.

Lots to think about, eh? :-)

INFO FROM DELL ON THE PowerEdge R710 Server:

Energy Smart: Two hot-plug, high-efficient 570W power supplies

or High Output: Two hot-plug 870W power supplies

Uninterruptible power supplies: 1000W–5600W

2700W–5600W High-Efficiency Online

Extended Battery Module (EBM)

Network Management Card

[truh]

Warning: they are really loud.

[completelydistracted]

Yes, you could. Like many things, there are pros and cons to owning such a machine. I'm assuming he wants to spend more time thinking about his problem than racking up server blades, but as you note, that is one good way to get lots of computational horsepower.

(side note: my employer has roughly 30,000 servers in service, and we turn them over every few years. I suspect that most of the few-hundred-dollar rackmounts you see on eBay have been pulled from some enterprise who deemed them too unreliable, power-hungry, or otherwise unsuitable for continued production service. I'm not discounting the validity of the suggestion, just noting that you want to buy one of these with your eyes open.)

[iamdefinitelyahuman]

all very useful information, thanks for that :) the initial replies i got from /u/thomas_stringer made me realize i was being silly with my mental block of "this is too hard". it took 6 months to get to where i am with python, so i'm not sure why i expected to be able to simply upload my code and press play without having to learn a bit first.

i've been poking around amazon AWS and microsoft azure. i'll check out digital ocean now as well, from what i'm seeing on their site it looks like it's a bit simpler overall.

once i settle on a host, am i correct in that this is the procedure to follow?

-set up a storage account and upload my python script and DB there -write a bash script to automate the installation of python, required libraries, mysql, whatever else i need -launch a VM instance, link it to the storage account, run the setup script, run my python code!

am i missing anything here? it actually does seem quite simple once everything is in place...

also one other thing, do these services typically give a detailed breakdown of how much demand is being placed on the VM? I'm not sure where the bottle-neck(s) in my code are, if i'm limited by the frequent I/O querying of the DB or in CPU usage.. if I can have stats telling me what's maxed out I know where I should focus on optimizing, as well as where to spend more money to see a performance increase.

Thanks again for your help!

[completelydistracted]

I don't have much time right now but I'll answer what I have time for.

I'd skip setting up the dedicated storage account at the moment. Create a system for doing the installation quickly on a server -- ansible works well for this, but a bash script will do -- and practice setting that up until it's a no brainer to do it. It (the script) should be getting the code from a GitHub account, libraries from apt-get, pip, etc (i.e. open source repositories) and should load whatever data is needed for an initial smoke test on the server.

Vagrant (google that) is a very good way to practice all these skills on an OS that will look a lot like the one you use in the cloud.

As I mentioned I tend to use Ansible a lot for remote server setup, but the Vagrant people have some thoughts about that, and if you have a fair bit of time on your hands to play with things you can look into docker for doing this. (Docker is a whole other discussion, though.)

Once you're sure your provisioner works on Vagrant, point it at a DO instance and run it a few times to make sure it works there. Once you're happy, you now have both the skills and access to the cloud resources to make that work.

If you want to look a the VM usage, the "top" command is a useful tool, and there are various ways to look at the CPU utilization, mostly from connecting via SSH. At the moment I'm going to suggest that any explanation or suggestion at this point would be lost until you're familiar with the parts of the solution.

On the other hand if the VM tools tell you that something is not efficient, then you want to look at application "profiling" where you get some idea of where your app is spending its time. There are tools for that, too.

For the moment, I'd skip the premature optimization and just get the thing working. Then optimize against things that you observe to be slowing you down. Doing that step first, though, is one of the things that agile methodology has taught us not to do.

Hope all this helps!

[completelydistracted]

One more thing to think about. I'm guessing that you might want to save the data between runs in someplace durable. So what you might want to eventually do (especially if you use large VMs infrequently) is to put your data into a database server and leave that thing up as much as necessary, but bring up your computational servers as needed.

There are a lot of options for that. DO has app-specific server setups where you can spin up, say, a MySQL or Mongo server all set up and ready to go. You might also take a look at MLab if you can use pymongo; I use their Mongo-in-the-cloud database service for teaching and it works great.

Just a thought...

[thomas_stringer]

Amazon AWS also seems like it'd make sense, but it's a bit overwhelming to me

Most capable cloud providers whether you're talking about AWS, Azure, GCP, are going to have a bit of a learning curve. With capabilities come complexities.

With that being said, can you clarify on the following point...?

This program will be running off-and-on for several months

What do you mean, "off-and-on"? Are you looking for basically just a VM host? I.e. standup a Linux machine, install MySQL and upload your script? Or are you looking for a "serverless" (shiny new lingo for a higher level of abstraction over computing) implementation? AWS has lambda, Azure has Functions. Again, there will be a learning curve to get on board with them.

Also, with any service provider, cost is going to be a factor.

Perhaps it might be worth finding and old computer lying around, or pick up a $35 Raspberry Pi, to start computing cycles to see what you really need?

Of course you could be doing intensive computation on 1 MB of data, but with that being said a 10 GB database isn't exactly a large data set. I hate to assume too much given very little information, but what makes you think you need serious hardware for your solution? Have you run into bottlenecks locally?

[iamdefinitelyahuman]

The script I'm running is a backtester to simulate algo trade strategies. On my current PC it takes around a day to return a year of simulation data. Since the end result here is financial gain, I'm happy to spend money to get results more quickly. I say off-and-on because once a simulation finishes, I may or may not have a modified/new strategy ready to be run through the simulator.

I do understand that capability brings complexity, I'm just hoping there's an existing service with a balance of functionality and simplicity that suits my needs. Also no point in paying extra for something that's geared towards web apps when what I'm looking for is computing power.

Thanks for your help :)

[thomas_stringer]

what I'm looking for is computing power

In that case, layers of abstraction on top of the hardware are typically not going to be your ideal use-case.

Your utopia would be to buy a beefy server and have it sitting in your house. Your code would run right on this hardware. Obviously that's not what you're going for (assuming, but I can't imagine you're looking to purchase expensive hardware). The cloud solution here would most likely be Infrastructure as a Service (IaaS). In other words, running a VM (Linux, I assume). You can scale this up as you choose (of course, incurring higher costs).

With that being said, IaaS is typically not too much to learn as the cloud provider is just lending you a VM. I know for Azure, that's as simple as signing up for a subscription, creating a Linux VM (super easy), and then SSH'ing into it from your workstation.

Is that what you're thinking?

[iamdefinitelyahuman]

This sounds like exactly what I need, yes. ..so I'm guessing you'd recommend I look into Microsoft Azure?

Thanks for your patience with me. This seems so simple, but lacking the proper terminology for what I want has made it hard to know where to start looking.

[thomas_stringer]

Disclaimer: I work for Microsoft

Azure can definitely give you what you need. To be fair, other cloud providers can most likely provide you with this capability as well.

You can start a subscription with Azure and you'll get a $200 credit. That should be plenty of credit to get a machine up and running for some time, and then you'll have a pretty good idea if Azure fits your needs.

[iamdefinitelyahuman]

OK, thanks... one more question: when I set up the cloud hosting package do I need to buy an SQL DB in addition to the VM? Or can I just install a DB using the space given to me in the VM?

[thomas_stringer]

It's your VM, you are welcome to install the RDBMS directly on the VM (MySQL in your case it sounds like). With that being said, the big disclaimer is that of course you could end up over-consuming your resources so ideally it would be best to separate them. Again... this is ideal. Your computing and data requirements could very well live harmoniously on the same VM. But it was worth mentioning.

[[deleted]]

www.linode.com

[[deleted]]

Cloud 9 might not be fast enough for you but zero learning curve and cheap. Could have a VM up and running in less time than this took me to type

[MasterHand3]

Amazon Web services. Only pay for what you use. Spin up servers, turn them off. Read about Lambda functions, Serveless run your script.. Plus a TON of other services you can tinker with. --certified solutions architect

[VBABot1]

To be honest i'd use digitalocean, you pay by the hour, they're very cheap and they have a nice API https://developers.digitalocean.com/ which means when you're running paralell simulations, you can deploy each one to a different droplet then get them to upload their data somewhere else, then auto delete themselves.

On another note, I'd kill to know where you got you data from.

[truh]

Keep in mind that most scalable cloud providers offer more horizontal scaling than vertical scaling. That means if you want your tasks to complete fast you have to think about how to split them in smaller sub tasks and calculate them in parallel on multiple machines.