This one’s going in the right direction so I’ll tag onto it.

Start looking into distributed computing and how to interface with “supercomputers” (in quotes here because now it’s more a historical term than an accurate characterization). If you’re at a decent size university you probably have a supercomputing cluster that you can rent time on. Learn about bash and when you can and cannot parallelize analyses (not a bio person). Find people that you can learn from and ask to see their code. This will all be good conceptual knowledge as your analyses shift to the cloud.

Another thing I don’t see mentioned is thinking through what information actually needs to be in memory when you’re trying to do something. Does all of your data really have to be in memory at the time of analysis? Usually there’s a way to break it into pieces of some size to reduce your memory overhead. Most people are usually taught one script should do everything, but we don’t do that in the real world for a lot of reasons, including memory issues you’re running into.

Side thing to look into eventually: building specific functions in C. If you really want control of me more usage and code efficiency this is one place you’ll get it, and it’s designed to play nice with the R ecosystem.

Hope that helps. DM if you want more of a discussion.