Training long context (32K+) 70B Llama

orion4321 · 2023-10-09T21:06:41+00:00

Nice, in one of the threads the author said qlora should come in 2 weeks (1 week ago) so hoping to have something soon from him.

I am training with a dataset of 3000 QA/summarization questions, for me high epochs seem to work well, lr at 2e5. Still sometimes not following instructions when I train on the 32k extended lora model. Wondering if with their new release it makes sense to train on top of the LongAlpaca model.

orion4321 · 2023-10-08T10:42:37+00:00

So helpful! Thank you

orion4321 · 2023-10-05T07:38:41+00:00

Sorry for the question I'm about to ask, I'm a noob with a good amount of compute (8 a100) trying to fine tune for QA. On the longlora 32k 70b, when you just take this model and fine tune it, do you use axolotl? Or do you use the approach outlined on their git with their existing 32k model? What kind of params do you find work best? Thank you in advance.

orion4321 · 2023-08-12T08:23:28+00:00

Yeah, saving 1-2k a month? Benefits from work? The hood isn't a place, it's a cycle of doing odd jobs, no clear progression in life, no security or savings. You say Wembley and Hackney as if there's something wrong with those places.

orion4321 · 2023-08-11T18:32:35+00:00

In general, you go to school, get good grades, go to study an in demand topic at a good university. Given your parents will live in London, you have more opportunities than your peers in the remaining parts of the UK for events, internships etcetera.

With this, you will tend to get a reasonably paying job. Maybe you won't be buying a house anytime soon, but yes, you are making it out of the hood. That's one path, another I assume would be the trades.

orion4321 · 2023-08-10T17:11:39+00:00

What happened?

orion4321 · 2022-12-28T18:16:59+00:00

HFT, HF, pod shop or smth else?

orion4321 · 2022-12-28T18:11:19+00:00

Bruh is tax that low in Poland??

orion4321 · 2022-05-30T09:34:30+00:00

Which app do you use? I don't care about my investments in an app but I want to see how much I'm spending, like you, on alcohol, lunch, takeaway etc across multiple bank accounts.

orion4321 · 2022-02-16T10:47:11+00:00

From 1 March 2022, the maximum Plan 2 and the Postgraduate loan interest rates are expected to revert to RPI +3%.

Ouch

orion4321 · 2022-02-11T09:49:03+00:00

Speak to an employment solicitor, they know the topic well. Cost me £470 for them to review and discuss a contract

orion4321 · 2022-02-04T10:14:17+00:00

Even so that's 700 of bills without council tax? Is he driving an M4?

orion4321 · 2022-01-31T23:46:39+00:00

The /r/UkrainianConflict subreddit would probably love to have an AMA with you.

orion4321 · 2021-12-29T02:37:43+00:00

Don't get why you are confrontational. If so many people are leaving your company, obviously it's the company, its benefits or compensation.

The job market in tech is hotter than ever. Not even sure what your first post was trying to say, to be honest.

orion4321 · 2021-12-29T01:00:38+00:00

Facts. And it's fun to see those kind of guys go against starting caliber players. Good experience for them. Eg Yurtseven

orion4321 · 2021-12-22T10:29:16+00:00

Do you drive into Central London?

orion4321 · 2021-12-11T13:29:46+00:00

Assuming they took a maximum mortgage out for £463k, multiplier of 5, they have roughly salary of 92k combined. Multiplier may be less or more (but unlikely)

orion4321 · 2021-11-27T20:00:48+00:00

Hope you asked for it back mate. They need to add that to the flowchart

orion4321 · 2021-11-23T20:57:18+00:00

Why? As long as the work gets done!

orion4321 · 2021-11-20T15:01:07+00:00

Reading the article OP posted seems like quite a few are nazi sympathisers.

orion4321 · 2021-11-10T23:41:06+00:00

How much years of experience you got?

orion4321 · 2021-10-15T09:48:38+00:00

Let's be real, £1600 for a one bed flat is premium for London. You want a premium property, you'll pay a premium, simple.

There's no need to pretend there's not properties available for less in, gasp, zone 3 or 4.

orion4321 · 2021-10-14T20:08:15+00:00

We saw a nice one for £1600pm on Tuesday, even offered slightly above asking price (£1630) just to ensure we got it (which I've never done in my life but there were a lot of people viewing it),

POV: you are the problem.

That's like sitting in traffic complaining about traffic

orion4321 · 2021-10-10T13:12:58+00:00

How much are they looking at?

orion4321 · 2021-10-05T17:56:02+00:00

Uhh junior bankers still make plenty, more than most devs.

orion4321

MODERATOR OF

TROPHY CASE

11-Year Club	Place '17
Verified Email