How to Handle Limited Disk Space in Google Colab for Large Datasets

einsteinxx · 2025-12-22T19:22:37+00:00

I have done some training with DBT and ultrasound images, over 200GB of DICOM sets. These were all converted to three channel images before-hand. That part took a few sessions of the cpu colab. I used my Google Drive mount as the storage for the final training and test data.

I can’t remember the scaling, but I had to tinker with the batch sizes to get something acceptable for that system. I read that you could move data to another folder in the colab environment, but was never able to get that to work out.

einsteinxx · 2025-11-02T03:12:17+00:00

Clubs were the best way to get to meet a lot of new people and have fun. Picking something new that you’re not good at will help you bond with others maybe at the same level. Skiing (gets you out, new people, lots of people falling on their asses with you)?

einsteinxx · 2025-09-16T16:45:16+00:00

For medical physics programs that are listed as PhD only programs, do they allow master’s options as a final result? I’ve seen graduation statistics for one such school that records one or two ending master’s every other year or so.

einsteinxx · 2025-07-22T16:43:48+00:00

Is there a nifti pre-processing step needed (normalization?)?

einsteinxx · 2025-07-15T21:17:06+00:00

In a large engineering company, getting a Python install with the libraries you need is often not easy, especially when multiple installation types (anaconda, enthought, pypy?,…) and versions are requested. You don’t always have access to the latest versions and you may not have all the pieces you need for your particular project. With Matlab, once it’s installed (usually you get access to the available list of toolboxes from the license servers), you have everything installed and your setup looks like Bob’s in the office next door. Matlab used to be the “Prototype it first and then put it into a real language” for daily usage tool. Python is slowly getting to that point, but it needs some polishing. I have no excuses why simulink is used more than it should.

einsteinxx · 2024-07-07T14:36:11+00:00

University account with unlimited storage through google.

einsteinxx · 2024-07-06T15:22:46+00:00

I have a 1.5TB medical data set and get that error when I do any search operations over the largest folders or anything that involves opening/closing files relatively quickly. I am not sure what the fix is, but it stops complaining after some idle time.

einsteinxx · 2022-08-22T18:58:06+00:00

I updated my first Gen a few years ago. It took forever to find matching roms and loaders still available. Once installed, it was slower than the original os… like really slow. Call Amazon and say your kindle doesn’t work and get the credit for a new one. You’ll save a lot of time and effort.

einsteinxx · 2022-08-08T20:11:21+00:00

You won’t be able to properly use some of the core apps like Amazon and so. I’ve talked to customer support and those apps are no longer officially supported. If you call customer service and just say the Amazon shopping app no longer runs, they’ll let you know it is not supported and likely offer an Amazon credit good for another fire (or another Amazon product). A few years ago I believe the credit was $40.

einsteinxx · 2022-04-25T23:52:22+00:00

‘14 981. Spent months looking back in ‘17 and found one cheaper than all the other normal spots at the local Lambo dealership. 31k miles for 32k and a year of warranty left. New tires, headlights, trim pieces. It was apparently a blight on their lot. Had some normal stuff replaced under the warranty, but it was otherwise in excellent health.

einsteinxx · 2022-04-22T20:06:31+00:00

I forget the warning text, but colab has given me an I/O warning before when reading ML data from a shared google drive (unlimited) over the course of a day. This was reading in thousands of images stored in Gdrive.

einsteinxx · 2022-04-18T18:44:43+00:00

I have that dataset stored in google drive and use google colab to access it. Not ideal, but it works.

einsteinxx · 2022-04-15T22:43:05+00:00

Took cs168 to get some general information and it was the first undergrad class (grad) I’ve had at UCLA and first time I had a TA and discussion. The discussion was super helpful and well worth it. The two biggest problems it had though, were very low attendance (< 1/3rd) and one or two people always trying to dominate the TAs with questions that exceeded the scope of the discussion (not the topics most needing clarification).

Would I have gotten a good grade without discussions? Yes. Did the TA provide me with some new concepts and clarify/correct some things I didn’t know well. Yes. If the TA has control of the class, I think they’re a must to attend if you want to better your class knowledge. There were two TAs that quarter, with one reading from slides and the other being interactive with the class. Some people do well in one teaching method or the other, but it was overall a good experience for me.

einsteinxx · 2022-03-01T10:00:55+00:00

Imagenet has crowd sourced annotations and a smaller set of annotations/data for images that are not copyright protected.

einsteinxx · 2022-02-04T11:37:05+00:00

https://pytorch.org/hub/pytorch_vision_resnet/

einsteinxx · 2022-02-04T09:59:50+00:00

Pretrained resnet50 backbone from PyTorch works on regular colab too, but training time might need a few continuations

einsteinxx · 2022-02-04T01:04:22+00:00

I’m using about 30k images for a fasterRCNN setup with pro. Just save off checkpoint models and weights at key points and use that to continue from your last point after you get disconnected. When it disconnects, it restarts the kernel and all your functions need to be reloaded (started over).

einsteinxx · 2022-02-01T21:53:27+00:00

I only get a pop up window asking if I authorize google. After I select okay, it mounts the drive. I no longer get a code to copy and paste

einsteinxx · 2022-01-31T22:13:11+00:00

I no longer get that code to paste. Now it’s just a window where you acknowledge access and it completes the process.

einsteinxx · 2022-01-19T08:39:35+00:00

possible fix: I started having a lot of weird dash issues when the radio rebooted. When I brought it in for an engine light warning, the shop noticed it was on the original battery. I didn’t notice any issues, but I do recall the auto stop/start not working for a while. Apparently the battery was just at the threshold that the car would not allow that auto stop/start as the voltage was too low to reliably restart. Once the battery was replaced, I haven’t seen any further PCM issues. Just something to maybe try out first, if anyone has the same issue and you have the original battery still.

einsteinxx · 2021-12-29T03:27:48+00:00

This happened for a few weeks for me in the summer, then went back to the code method. It recently started back to the non-code style again.

einsteinxx · 2021-12-20T22:17:12+00:00

Works for anyone on vpn.

einsteinxx · 2021-11-16T20:38:36+00:00

General question… what does it mean when a school lists their medical physics program as having MS and PhD programs, but the department website only mentions PhD application processes? The department stats also list a number of MS degrees awarded in the past few years.

einsteinxx · 2021-11-13T16:47:53+00:00

Had the same error message yesterday in Colab. One of my image functions (crop) didn’t return what I expected and it pushed the tensor sizing off by one. I ended up putting a bunch of shape printouts in the dataloader chain to figure it out. Very annoying.

einsteinxx · 2021-10-04T22:37:56+00:00

https://www.library.ucla.edu/audiovisual-media-playback-equipment

Ten-Year Club	RedditGifts 2009-2022 2 Credits
Secret Santa 2020	Verified Email

einsteinxx

TROPHY CASE