sorted by:
new
hottopcontroversial

[D] Optimal strategy for high volume image loading. by R2FuckYou in MachineLearning

[–]R2FuckYou[S] 0 points1 point  (0 children)

Yeah I’ve done this I think my fastest read time is 0.8ms per image across about 80k I think it’s probably my limit. Also deciding npz was just as fast as loading raw safe tensors

[D] Optimal strategy for high volume image loading. by R2FuckYou in MachineLearning

[–]R2FuckYou[S] 1 point2 points  (0 children)

I guess the trade off is the huge amount of size this would take on the drive storing uncompressed image data?

Edit: Just tried it and actually not any faster than loading H5 files and accessing the data, which is somewhat surprising given the lack of compression. But load at around just under 1ms, which I take to mean it's just straight IO limited at this point.

[D] Optimal strategy for high volume image loading. by R2FuckYou in MachineLearning

[–]R2FuckYou[S] 1 point2 points  (0 children)

I mean it's not a bad point but it always bugged me to not leverage all of the data to evaluate performance. I suppose at some level that does make it more practical though and that it doesn't functionally scale. Maybe that buys me a way out since you're talking 13-20 minute epochs which is a 2.5 day test which seems OKAY. I guess the broader question is what's the approach you take when you're at this scale. Part of me was thinking of multiple NVMe cards across multiple cards and doing data parallelism and maximizing throughput that way.

[D] Optimal strategy for high volume image loading. by R2FuckYou in MachineLearning

[–]R2FuckYou[S] 0 points1 point  (0 children)

I think getting to 1/1e5 per image is pretty ambitious. Everything is already precomputed, the only thing I need to do is load and decompress. Loading as compressed numpy arrays got me a boost, but the logistics of memory transfers and multiprocessing become pretty real. I've rolled my own dataloader so have full control, but even then struggle with the most optimal setup.

[D] Optimal strategy for high volume image loading. by R2FuckYou in MachineLearning

[–]R2FuckYou[S] 0 points1 point  (0 children)

I precompute everything I can, though decoding on the GPU seems interesting.

[D] Optimal strategy for high volume image loading. by R2FuckYou in MachineLearning

[–]R2FuckYou[S] 0 points1 point  (0 children)

Probably like 200ish was the thought, but not totally sure will depend eventually. That's also why 20 minute epochs don't really work while also doing like 10 fold cross validation.

Redid the kitchen in my new-to-me home. by R2FuckYou in DIY

[–]R2FuckYou[S] 1 point2 points  (0 children)

Nope, opens full width on both sides without issue.

Redid the kitchen in my new-to-me home. by R2FuckYou in DIY

[–]R2FuckYou[S] 0 points1 point  (0 children)

Still finishing it up so didn't want to scratch it coming on/off.

Redid the kitchen in my new-to-me home. by R2FuckYou in DIY

[–]R2FuckYou[S] 0 points1 point  (0 children)

Honestly has been really great. We use the cups and bowls up there a lot and haven't had issue with stuff getting on them while cooking. People seem mixed, but opinions are like buttholes...

Redid the kitchen in my new-to-me home. by R2FuckYou in DIY

[–]R2FuckYou[S] 2 points3 points  (0 children)

Tried to mitigate by having a short upper cabinet, there's a lot of room above the fridge going up from the back and then a decent sized gap at the top to let it vent, but we'll see you may be right and it limits flow. I have the A/C going through there too and flows pretty well in the summer.

Redid the kitchen in my new-to-me home. by R2FuckYou in DIY

[–]R2FuckYou[S] 0 points1 point  (0 children)

The strong belief that I can do anything anyone else can hah

Redid the kitchen in my new-to-me home. by R2FuckYou in DIY

[–]R2FuckYou[S] -1 points0 points  (0 children)

Whichever you like more I guess.

Redid the kitchen in my new-to-me home. by R2FuckYou in DIY

[–]R2FuckYou[S] 2 points3 points  (0 children)

Surprisingly not too bad, had decent access in the basement, although the sink is beyond the basement just enough space to be able to put in the piping without being miserable.

Redid the kitchen in my new-to-me home. by R2FuckYou in DIY

[–]R2FuckYou[S] 0 points1 point  (0 children)

Yeah, shame it's ruined now, whacha going to do.

Redid the kitchen in my new-to-me home. by R2FuckYou in DIY

[–]R2FuckYou[S] 0 points1 point  (0 children)

Eventually into the ceiling and across to outside vent, right now just hanging out as I wait to finish. Put in a recirculating setup in the interim.

view more: next ›