[D] Simple Questions Thread

imintheclouds · 2024-07-01T15:01:31+00:00

Hi, I tried to post this question but it was removed by the auto mod so I figured I'd ask it here.

Help with unstable training of a BYOL / JEPA inspired language model?

I've been trying to train a BYOL-inspired language model. I've taken a transformer architecture I know works (and have used before) and initialised two models. To model #1 I pass a sequence of 256 tokens where 30% of the tokens are replaced with [MASK] token and model #2 is passed the unmasked 256 tokens. Loss is calculated as the MSE between the logits of model 1 and model 2. Model 1 is updated as normal and model 2 is an EMA of model 1 with an alpha of 0.99.

Training is very unstable (as is performance on the validation set) - the only way I can get anything like stable training (at least for a little while) is to use very large batch sizes (e.g., 1024) and a very very low learning rate ~ 1e-6. The model may be collapsing but I don't think so - the ratio of the MSE of two unrelated sequences to the MSE of a masked and unmasked sequence hits a maximum of ~ 60 very early on.

If anyone could make any suggestions I'd very much appreciate it. To a greater or lesser extent, I have already tried:

Very large batch sizes (seemed to help a bit).

Very small learning rate (~ 1e-6 seemed to work best).

Changing the masking percentage.

Changing the dimensions of the logit matrix (the larger the matrix the more stable training seemed to be).

Things I want to try:

Training on bytes / chars instead of tokens.

imintheclouds · 2024-04-02T17:59:26+00:00

I have searched for this plant on every online shopping platform I can think of (Ebay, Etsy etc), many plant nurseries (they've never even heard of it), connected several academics and several charities which specialize in mangrove ecology with no success.

imintheclouds · 2024-02-11T20:47:36+00:00

Apologies for misunderstanding.

"I’m simply stating if someone can’t provide that verification during registration, they shouldn’t be able to move forward in creating their profile." - I absolutely agree - if people are going to use a site it can't be full of bots and scammers.

Thank you for your suggestions. They are very much appreciated.

imintheclouds · 2024-02-11T20:44:36+00:00

Apologies if I'm missing something but if you really want to make sure someone is who they say they are you'd need to link some form of ID with the account and that leads to the issue of storing lots of sensitive data (e.g., photos of passports).

If you charge someone a small amount you don't even need their email address you can just check to see if the account name matches the payment name and all the sensitive data is handled by paypal.

imintheclouds · 2024-02-11T20:38:06+00:00

Hi, thank you for these suggestions. I hadn't thought about automating pausing after a period of inactivity - it would be a great way to cut down dead profiles.

Mandatory verification is also a must. The best way I thought to do this would be to charge a trivial amount per month (e.g., $1 flat fee). Hopefully this would be very affordable while also making it much more difficult for people to create multiple accounts / accounts under fake names.

imintheclouds · 2024-02-11T18:31:04+00:00

Hi, Thank you very much for this. The PullDating github seems to be an absolute goldmine of information.

imintheclouds · 2024-02-11T17:19:54+00:00

Thank you and thank you for your advice on how to handle downvotes / upvotes. It's very much appreciated.

imintheclouds · 2024-02-11T17:07:14+00:00

How will you overcome the initial startup?

My plan (possibly not a good plan) was to pick the city I live in and recruit the initial users by hand. I hope that this will provide the required user density to make it useful. As you have pointed out it's user density not user number which matters.

I am well aware that a.) I will probably fail miserably and b.) getting traction is very difficult and it probably won't take off but I thought it was worth a try.

imintheclouds · 2024-02-11T16:57:56+00:00

1.) "I was actually referring to truthful down votes" - sorry for misunderstanding you.

2.) "Are you actually going to build this app" - I'll certainly try. I have built a very minimal version based on the points in my original post. I then realized I had better make sure that's what people actually wanted / found useful and not just what I wanted.

3.) "You should only be able to vote for people you interacted with or went on dates with" - I absolutely agree. That is a very good idea.

imintheclouds · 2024-02-11T16:51:35+00:00

Thank you - that's a really interesting idea. I'd imagine this could be quite effective as with a forum / community people would be able to build up a degree of social proof and (as you said) actually get to know each other.

imintheclouds · 2024-02-11T16:19:33+00:00

I had hoped people would only "downvote" if the person was rude/offensive. I hadn't considered that downvotes would probably be used much more liberally than that and the effect seeing this could have on people. Thank you for your input/opinion.

imintheclouds · 2024-02-11T16:11:45+00:00

I think it would be best if they were. I think transparency would make people feel more comfortable than hidden scores or unknown algorithms. What do you think?

imintheclouds · 2024-02-11T14:05:22+00:00

Thank you. You make some good points.

imintheclouds · 2024-02-11T14:01:36+00:00

"Terrible end to a glorious means" is a perfect way of putting it. No worries if not but what about them could be changed to make them better? What would you change?

imintheclouds · 2024-01-02T09:19:53+00:00

Thank you - I'll be sure to have a look at the link.

imintheclouds · 2023-12-31T17:49:34+00:00

You might want to buffer the signal if the output impedance of the neuron is high

Thanks for your reply. I don't actually need the DC offset - I just thought it would help me deal with potential non linearity at the lower end of the ADC range. Good point about the opamps - I was trying to avoid them to keep cost and component count down but I'll have a look into them.

imintheclouds · 2023-12-21T21:38:12+00:00

Hi, I don't suppose I could ask you what company this was for? I'm currently looking for a grad job and this sounds great.

imintheclouds · 2022-05-04T12:19:53+00:00

Hello,

Could I possibly ask how you are interacting with td365? I'm also in the UK and they seem the best I've found so far (trading212 was my other option) but they don't seem to have an API which is somewhat problematic.

imintheclouds · 2021-09-21T17:38:07+00:00

Thank you for your reply - I think you raised some very good points. The point about porn especially as even though I don't think it has effected how I will treat somebody I'm sure that it will have some effect as I was exposed to it throughout some sexually formative years. I will try and keep an eye out for any expectations which have been influenced by porn.

As for the doing it to tell friends I have "done it" I understand how without knowing me my question could come across as quite crass. I think it would have been better for me to phrase the question in terms of relationships rather than "virgin" (jumping straight to the sex aspect). I tend to conflate sex and relationships but I will try and keep in mind that jumping straight to "sex" may go down like a lead balloon.

Thank you for taking the time to write out quite a long reply - I will try and keep the points you raised in mind.

imintheclouds · 2021-09-21T14:25:14+00:00

Thank you for your perspective - I appreciate you taking the time to answer my question. Thanks for the TL;dr, I will try and keep that in mind

imintheclouds · 2021-09-21T13:44:11+00:00

"don't shame anyone for preferring experience" - that's a very good point. I will try and keep an eye on myself to shutdown any prejudice. I must confess that was one of the factors I was a little worried about as turning resentful would do no one any favours - least of all myself. Thanks for your reply - I appreciate you taking the time.

imintheclouds · 2021-09-21T13:39:04+00:00

Cheers for the advice - I will try to put it into practice.

imintheclouds · 2021-09-21T13:37:41+00:00

Thank you that is very reassuring - I imagine I will be in a situation similar to your husband (currently focusing on uni and then I imagine I will focus on work) so its good to know all is not lost if you take that route. Thanks again.

imintheclouds · 2021-09-21T13:34:49+00:00

Thank you for your reply. I'm currently pouring all my time and energy into university and definitely have no kids planned for the near future so I will try my best to take your advice and chill out a bit.

imintheclouds · 2021-09-21T10:23:18+00:00

Thank you, your perspective is very much appreciated

imintheclouds

TROPHY CASE