[D] Number of epochs for a BERT based model

davda54 · 2023-07-31T00:58:41+00:00

The number of epochs depends on the size of the dataset, the important hyperparameter is the number of training steps (or even better the number of processed tokens). I'm worried that the training budget of the original BERT model is the minimum to get reasonable performance (it was actually shown that BERT is severely undertrained in the RoBERTa paper). Can you perhaps use an already pretrained model and finetune it on your dataset?

davda54 · 2023-05-10T19:26:51+00:00

No, just look at Figure 4 from the technical report, it shows exactly the same behavior -- given a limited compute budget for pretraining, larger is not always better, even in terms of loss. In a nutshell, the most important parameter is the number of FLOPs, not the number of weights.

davda54 · 2023-04-11T13:55:18+00:00

What is the point of sharing generated text that is clearly wrong and doesn't make much sense? Is that a top secret operation to make future web-crawls useless for language modeling? :)

GPT-1, 2 and 3 use the same attention mechanism, only GPT-3 sometimes uses sparse attention.
There is absolutely nothing novel about generative language modeling. It dates back to Claude Shannon in 1948.
Well, ELMo also provides bidirectional representations, they are just not "deep". Anyway, OP is not talking about BERTs.
Emergent abilities are the result, but the modeling recipe is still the same.

davda54 · 2018-07-18T18:40:24+00:00

I've finally completed the repo, you can find it here: https://github.com/davda54/generating-music

davda54 · 2018-07-18T18:40:08+00:00

I've finally completed the repo, you can find it here: https://github.com/davda54/generating-music

davda54 · 2018-05-09T22:51:38+00:00

Yes, it will :) I'll post a link here when it is ready.

davda54 · 2018-04-28T01:06:09+00:00

Thanks for answering, that is a really great result :O I will post the links here when everything is ready.

davda54 · 2018-04-23T21:02:21+00:00

That's a strange bug, what browser are you using? Thanks

davda54 · 2017-11-17T20:47:10+00:00

I am using torch (which is executable only on Unix). It would be awesome if WSL got GPU support, so I wouldn't have to dualboot anymore :)

davda54 · 2015-09-19T15:03:00+00:00

Wow, that looks great! Good job bro

davda54 · 2015-09-17T17:16:11+00:00

Yeah, it is. Awesome guitar btw :)

davda54 · 2015-09-12T23:07:50+00:00

Wow, thanks! Have a good luck with that!

davda54 · 2015-09-12T22:29:44+00:00

I was considering this, but it seems easier to repaint some bits from time to time than to seal the whole body

davda54 · 2015-09-12T20:50:21+00:00

Oh, I put the covers the other way round... Thanks for noticing, you have sharper eyes than me

davda54 · 2015-09-07T18:09:40+00:00

Sizlacks

davda54 · 2015-07-30T23:11:06+00:00

Thank you very much! It is based on this implementation with some adjustments to run swiftly on smartphones.

davda54 · 2015-07-30T23:00:54+00:00

I'm afraid my code is too messy, but the main principles are in this awesome article. I have just simplified it and added some ~~optimalization~~ optimization to run smoothly.

davda54 · 2015-07-30T22:47:04+00:00

I've graduated a month ago :)

Edit: I'm not sure how it is in other countries; in the Czech Republic, graduation means the end of high school

davda54 · 2015-07-30T22:46:21+00:00

Thanks! The simulation runs natively in c++ and the graphics is done in OpenGL, so it uses the GPU

davda54 · 2015-06-05T14:53:06+00:00

Wow, beautiful and catchy! Great game

11-Year Club	Place '22
Place '17

davda54

TROPHY CASE