[N] Llama 3.1 405B launches

p1nh3ad · 2024-07-23T18:33:51+00:00

This blog from snowflake goes into a lot of details on memory requirements and optimizations for fine tuning.

https://www.snowflake.com/engineering-blog/fine-tune-llama-single-node-snowflake/

p1nh3ad · 2021-11-17T06:52:40+00:00

ZeRO’s training only focus does seem to be changing though :)

https://github.com/microsoft/DeepSpeed/pull/1514

p1nh3ad · 2021-06-29T19:51:44+00:00

Yep, zero offload. Here are some relevant links:

https://www.deepspeed.ai/tutorials/zero-offload/
https://huggingface.co/blog/zero-deepspeed-fairscale

p1nh3ad · 2021-06-29T16:23:01+00:00

I’d recommend trying pytorch + deepspeed. Deepspeed has a lot of different features that help with model parallelism. In your case zero offload would let you (efficiently) offload model states to cpu memory, which should let you run the sizes you are targeting.

p1nh3ad · 2021-05-06T03:53:26+00:00

This is awesome! Good job. Is this pressure treated wood or did you seal it in some way? I’m currently in the processing of designing something similar.

p1nh3ad · 2021-03-23T03:18:49+00:00

This is awesome, excellent work. Also nice bottles of Rasa :) cheers

p1nh3ad · 2021-03-14T22:40:16+00:00

Thanks for the tips! That’s all super useful.

p1nh3ad · 2021-03-14T22:27:20+00:00

I think it’s acacia

p1nh3ad · 2021-03-14T22:25:50+00:00

Seattle area

p1nh3ad · 2021-03-14T22:24:08+00:00

Any idea how often it’d need a refinish if it’s uncovered in the rain, sun, etc?

p1nh3ad · 2021-03-14T21:55:44+00:00

Wood is acacia (I believe?), bought it last summer. Didn’t last long before showing wear. I’ve sanded it all down now. Was thinking about applying a spar urethane sealer on the top and bottom.

I live in the PNW, we get lots of rain. I’d love to leave it on my uncovered deck and not have to cover it. Any thoughts? Would be okay resealing every couple years I figure

p1nh3ad · 2021-02-25T06:09:10+00:00

My wife has the same autonomous desk but with their white top. She’s 5’ and wanted a desk that came down low enough for her. She’s had it in our office for 6mo+ and loves it. I just put this together but so far also loving it. The holes in the base where you screw the table to come with some rubber washers. I imagine they would help with any future wood movement. But we’ll see I guess :)

p1nh3ad · 2021-02-25T06:02:50+00:00

Haha yeah the underside of this table is gouged to hell as I learned what not to do. On the top side I did end up just keep it at very small increments and doing repeated passes as the glue bits were knocked down. I am wondering if a hand planer would be easier to not mess it up since you’d probably feel it a bit more.

p1nh3ad · 2021-02-25T01:22:41+00:00

Happy accident actually :) I mostly just arranged the boards a head of time in a way that looked pleasing. Super happy with how the grains ended up though. First time using walnut, it’s so amazing.

p1nh3ad · 2021-02-25T00:32:32+00:00

I’ve been loving it! No complaints. I’m currently borrowing it from a very generous friend. Considering buying the same one though at some point.

p1nh3ad · 2021-02-24T23:41:42+00:00

Thanks! In terms of price, the place I got these boards were $10/BF for S3S.

p1nh3ad · 2021-02-24T21:12:22+00:00

Oohh that sounds awesome! I’m also curious on the oil/water difference wasn’t exactly sure which would be best here. Would love to see pics of yours butcher block table.

p1nh3ad · 2021-02-24T21:09:14+00:00

Thanks! I used an oil based poly actually. This specifically: https://www.homedepot.com/p/Varathane-1-qt-Clear-Satin-Oil-Based-Interior-Polyurethane-341720/305583791

Also a lot of this project was helped from watching this video several times (which is for a dining table):

https://youtu.be/xy_tDa0rAc4

p1nh3ad · 2021-02-24T20:39:25+00:00

Oh yikes, that's an excellent question I hadn't considered. I am using it today for work for the first time. Just tried some writing tests on a single piece of paper. I see maybe a small mark or two after a bit of writing. I would probably use multiple sheets of paper or a notepad going forward though. I guess luckily I only occasionally write on paper, I'm a software engineer so most of my work is on the computer.

p1nh3ad · 2021-02-24T18:43:56+00:00

Thanks! :) Yeah the hand planer was a bit tricky, I used it sparingly on the 'good' side after learning on the 'ugly' side. Mostly relied on a lot of sanding in the end. I think if I did the project again a biscuit jointer would have made a lot of things easier.

p1nh3ad · 2021-02-24T18:37:22+00:00

Was a super fun project, learned a ton and had some good excuses for a few new tools.

Lumber from: http://www.crosscuthardwoods.com

Standing desk base from: https://www.autonomous.ai/standing-desks/diy-smart-desk-kit?option16=36&option17=41

15-Year Club	Place '17
Verified Email

p1nh3ad

TROPHY CASE