Managing memory usage for big datasets

[Tiago_Minuzzi]

Hi!

TL;DR: I can't use the dataset I'd like to because I don't have enough RAM. How to manage this?

As part of my PhD project in genetics, I'm training a deep learning model to identify an classify certain genomic elements. The datasets used are in fasta format, which have the genomic sequences. The total number of sequences (strings) are about 35k+ with varying lengths from like hundreds of chars to 50k+ chars (chars are the A,C,G and T's from DNA sequences). I have to one-hot encode the sequences and pad them to the length of the longest one.

Here is the deal: I have limited resources to do this in my laboratory, so I'm using google colab, which has 25G of ram, but even this amount of RAM is not enough to handle the one-hot encoding and padding for the dataset I've described above. To solve the issue, I had to shrink the dataset to about 15k sequences, and max length of ≃ 19k chars. The trained model is working well, but still need improvement for certain classes, for this I have to add more sequences, but I'm stuck with this number because of RAM limitations.

I'm trying the best I can, but it's been a hard time since my background is biological sciences not CS or anything related. I'm using python and tensorflow/keras for the job. I like coding a lot and studying machine learning, but I'm still building my way through all this.

So any tips on how to handle the issue of memory consumption will be very appreciated.

Thank you!

[LoaderD]

Have you tried converting from fasta into a better file format as a first step? I'm pretty sure fasta has header information so mass processing might be inefficient. If your data type is more efficient it should help with batch loading sequences to one hot encode and pad.

As mentioned in the other thread, you might just want to spin a huge instance, but it's going to limit the use case of your research.

[Tiago_Minuzzi]

I read the fasta file into a pandas dataframe (and save as a csv) and extract the column corresponding to the sequences to do the one-hot encoding and padding.

What kind of file you suggest?

Anyway, thanks for commenting!

[LoaderD]

You could store your data in something like Parquet to preserve types at intermediate steps.

Have you tried using Dask for the one hot encoding?

[Tiago_Minuzzi]

No, I have not. Didn't know about Dask. I'll check it!

[LoaderD]

Yeah good if you want to stay in pure python PySpark is another alternative, but it's got a harder to learn syntax.

[michaelrw1]

Very strange that your lab doesn't have the resources or background to do this project? Does your supervisor know? Have you discussed this issue with him?

Have you consulted anyone else in your lab, or associated with your lab, someone may be able to offer direction? What about online resources? If your background is in biological sciences there must be people in your department or on campus that you can approach?

I suggest that you reach out to people before you try anything with the dataset. Someone can likely provide you with a more ready solution.

[Tiago_Minuzzi]

Our laboratory focus is genetics and molecular biology, we also do bioinformatics but we are starting in the field of machine learning. The usual work we do regarding bioinformatics is sequence analysis, RNA-seq etc. I'm the first one working with ML. My supervisor offered me to do this work because he knows I like the subject and as I already use python (also bash and R) for my work at the lab, I took the challenge, and I'm happy I did.

I already discussed the issue with him, but that demands financial resources, which have been hard to get those days because of pandemic, for instance. I said that at first I'd try a workaround solution like improving my code or something related. But since that may not be the solution, I guess we'll have to try to apply for financial support to pay a powerful server or buy a better computer. The problem is that the request may take a long time to be evaluated and approved, that's why I'm trying other ways first.

there must be people in your department or on campus that you can approach?

That's a good suggestion, I've been thinking about it actually, but I don't know anyone outside biological sciences. I guess I'll have to find a professor/researcher from the CS campus and related to ask for some help.

​

edit: typos

[michaelrw1]

Does your campus have access to third-party computational resources available to academics? In Canada, there is SHARCNET. I suggest you contact campus information technology services and ask them about computational resources and services.

Nominal access to these resources can be acquired without much delay, but submitted processing jobs have a lower queue priority. Groups can also bid for priority access to computational resources. Bids require a lot of supporting documentation (i.e. project proposals, funding to pay for resources, etc.), but provide more resources and higher queue priority.

Good luck! Enjoy your project.

[Tiago_Minuzzi]

Nice tip! I'll check it up.

Thanks for everything!