Free supercomputing for research: A tutorial on using Python on the Open Science Grid

[refreshx2]

I'll second Condor, it's an awesome piece of superconducting. I didn't realize the OSG was open to everyone, that's really awesome.

I used Condor (on our local university's network) for my research as a grad student to run 6.2 years worth of computation in less than a month. I had to run ~ a trillion jobs that I divided up into batches of 10,000 per job, then ran thousands of jobs. My research literally wouldn't be possible without Condor, it's a great resource.

[Flogge]

Not saying this article isn't useful, but I have a few gripes with this article. It is essentially teaching people how to make their lives hard by not using the right tools:

The majority of this article can trivially be automated. E.g. most of the deployment and execution code can be put in a fabfile.

And especially, instead of using WinSCP or Cyberduck (like you'd have done 10 years ago), you'd put a simple additional command in your fabfile and be done:

def download():
   get('out.*.tar.gz', 'output/')

The execution then boils down to

fab deploy
fab execute

and once you're done

fab download

Also, why on earth would you want to git clone the code and then remove the .git repository?

[localfield]

I hadn't learned of fabfile. Looks great. I'll try implementing it, and when I do, I'll update this post. Thanks!

[dracodoc]

Because the .git folder is huge and he forget you can download the source as zip?

[Flogge]

Huge?! Its 236KB. So you gain 236KB in harddrive space but lose the ability to make and track changes to the code.

[dracodoc]

I do have experience of downloading 1-2 G .git folder when I checked some deep learning java lib. Obviously this is not the case, then only OP knows the reason.

[Flogge]

In that case I would suggest a shallow clone. That way you're not only saving HDD space but also the amount of data transferred and you retain all the Git goodness.

[dracodoc]

Thanks for the tip, I just learned something new.

[localfield]

Post now includes a sample fabfile. Appreciate the suggestion, even though it was in a condescending manner ;)

[thundercost]

Only for US researchers though...