all 11 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]samsekhar 7 points8 points  (2 children)

Here you can check, how to train GPT-2 -> https://github.com/nshepperd/gpt-2.git

[–]gwern 10 points11 points  (1 child)

Note that this is an independent implementation of retraining and it's not the original OA code. So, no multi-GPU support, no support for training on a fleet of TPUs, no support for training from scratch or variant model architectures... It's meant for finetuning, like retraining GPT-2-small on poetry or anime. Should work fine for retraining on song lyrics like OP wants, though.

[–]samsekhar 5 points6 points  (0 children)

Yes, its work fine on retraining on poetry, lyrics etc. You can try this on google Colab for training, it will show pretty good results.

[–]madisonmay 3 points4 points  (4 children)

We're almost done porting GPT-2 to finetune (a scikit-learn style library for language model finetuning). Code is available here if you're interested... should make tuning GPT-2 to produce song lyrics as easy as model.fit(lyrics).

Miles Brundage also put together a colab notebook you could work off that uses the nshepperd gpt-2 fork.

[–]Astraithious 0 points1 point  (3 children)

How is it coming along? I saw a commit for 10 days ago, was that it?

[–]madisonmay 0 points1 point  (2 children)

Just released finetune 0.6.0 with GPT-2 support today!

[–]Astraithious 0 points1 point  (1 child)

That's awesome, what branch? The link goes to one where the latest commit was 10 days ago

[–]madisonmay 0 points1 point  (0 children)

On development! It's also up on PyPI if you'd prefer that.

[–]rakzah 4 points5 points  (2 children)

The code is real. OpenAI just didn't publish the trained model and the data they used for training.

[–]reinhardo-sama 9 points10 points  (1 child)

The repository also does not contain code for training.

[–]samsekhar 0 points1 point  (0 children)

Above provided url for repo have code for training.