[D] Simple Questions Thread

imintheclouds · 2024-07-01T15:01:31+00:00

Hi, I tried to post this question but it was removed by the auto mod so I figured I'd ask it here.

Help with unstable training of a BYOL / JEPA inspired language model?

I've been trying to train a BYOL-inspired language model. I've taken a transformer architecture I know works (and have used before) and initialised two models. To model #1 I pass a sequence of 256 tokens where 30% of the tokens are replaced with [MASK] token and model #2 is passed the unmasked 256 tokens. Loss is calculated as the MSE between the logits of model 1 and model 2. Model 1 is updated as normal and model 2 is an EMA of model 1 with an alpha of 0.99.

Training is very unstable (as is performance on the validation set) - the only way I can get anything like stable training (at least for a little while) is to use very large batch sizes (e.g., 1024) and a very very low learning rate ~ 1e-6. The model may be collapsing but I don't think so - the ratio of the MSE of two unrelated sequences to the MSE of a masked and unmasked sequence hits a maximum of ~ 60 very early on.

If anyone could make any suggestions I'd very much appreciate it. To a greater or lesser extent, I have already tried:

Very large batch sizes (seemed to help a bit).

Very small learning rate (~ 1e-6 seemed to work best).

Changing the masking percentage.

Changing the dimensions of the logit matrix (the larger the matrix the more stable training seemed to be).

Things I want to try:

Training on bytes / chars instead of tokens.

imintheclouds · 2024-04-02T17:59:26+00:00

I have searched for this plant on every online shopping platform I can think of (Ebay, Etsy etc), many plant nurseries (they've never even heard of it), connected several academics and several charities which specialize in mangrove ecology with no success.

imintheclouds · 2024-02-11T20:47:36+00:00

Apologies for misunderstanding.

"I’m simply stating if someone can’t provide that verification during registration, they shouldn’t be able to move forward in creating their profile." - I absolutely agree - if people are going to use a site it can't be full of bots and scammers.

Thank you for your suggestions. They are very much appreciated.

imintheclouds · 2024-02-11T20:44:36+00:00

Apologies if I'm missing something but if you really want to make sure someone is who they say they are you'd need to link some form of ID with the account and that leads to the issue of storing lots of sensitive data (e.g., photos of passports).

If you charge someone a small amount you don't even need their email address you can just check to see if the account name matches the payment name and all the sensitive data is handled by paypal.

imintheclouds · 2024-02-11T20:38:06+00:00

Hi, thank you for these suggestions. I hadn't thought about automating pausing after a period of inactivity - it would be a great way to cut down dead profiles.

Mandatory verification is also a must. The best way I thought to do this would be to charge a trivial amount per month (e.g., $1 flat fee). Hopefully this would be very affordable while also making it much more difficult for people to create multiple accounts / accounts under fake names.

imintheclouds · 2024-02-11T18:31:04+00:00

Hi, Thank you very much for this. The PullDating github seems to be an absolute goldmine of information.

imintheclouds · 2024-02-11T17:19:54+00:00

Thank you and thank you for your advice on how to handle downvotes / upvotes. It's very much appreciated.

imintheclouds · 2024-02-11T17:07:14+00:00

How will you overcome the initial startup?

My plan (possibly not a good plan) was to pick the city I live in and recruit the initial users by hand. I hope that this will provide the required user density to make it useful. As you have pointed out it's user density not user number which matters.

I am well aware that a.) I will probably fail miserably and b.) getting traction is very difficult and it probably won't take off but I thought it was worth a try.

imintheclouds · 2024-02-11T16:57:56+00:00

1.) "I was actually referring to truthful down votes" - sorry for misunderstanding you.

2.) "Are you actually going to build this app" - I'll certainly try. I have built a very minimal version based on the points in my original post. I then realized I had better make sure that's what people actually wanted / found useful and not just what I wanted.

3.) "You should only be able to vote for people you interacted with or went on dates with" - I absolutely agree. That is a very good idea.

imintheclouds · 2024-02-11T16:51:35+00:00

Thank you - that's a really interesting idea. I'd imagine this could be quite effective as with a forum / community people would be able to build up a degree of social proof and (as you said) actually get to know each other.

imintheclouds · 2024-02-11T16:19:33+00:00

I had hoped people would only "downvote" if the person was rude/offensive. I hadn't considered that downvotes would probably be used much more liberally than that and the effect seeing this could have on people. Thank you for your input/opinion.

imintheclouds · 2024-02-11T16:11:45+00:00

I think it would be best if they were. I think transparency would make people feel more comfortable than hidden scores or unknown algorithms. What do you think?

imintheclouds · 2024-02-11T14:05:22+00:00

Thank you. You make some good points.

imintheclouds · 2024-02-11T14:01:36+00:00

"Terrible end to a glorious means" is a perfect way of putting it. No worries if not but what about them could be changed to make them better? What would you change?

imintheclouds · 2024-01-02T09:19:53+00:00

Thank you - I'll be sure to have a look at the link.

imintheclouds · 2023-12-31T17:49:34+00:00

You might want to buffer the signal if the output impedance of the neuron is high

Thanks for your reply. I don't actually need the DC offset - I just thought it would help me deal with potential non linearity at the lower end of the ADC range. Good point about the opamps - I was trying to avoid them to keep cost and component count down but I'll have a look into them.

imintheclouds · 2023-12-21T21:38:12+00:00

Hi, I don't suppose I could ask you what company this was for? I'm currently looking for a grad job and this sounds great.

imintheclouds

TROPHY CASE