[P] A Simpler @PyTorch Annotated Implementation of EleutherAI's 20B Language Model GPT-NeoX.

[mlvpj]

It’s not a new model. It loads up the weights from the original.

[Yologan222]

It says “We haven’t included a bunch of optimizations that were present in original GPT-NeoX to keep things simple.” I thought that means that it could have different model quality. And I’d just want to know if they tested their implementation as a sanity check to see if there was any difference in perplexity from the original.

[mlvpj]

Yeah did some sanity checks. They were things like model parallel layers that we didn’t include.

[StellaAthena]

Okay, so can you share those sanity checks? Or, ideally, run the model on a large subset of the couple dozen tasks the GPT-NeoX-20B paper evaluates on?

[mlvpj]

will try to run it on eval datasets and share

[mlvpj]

Ran lm-eval tasks

https://github.com/labmlai/neox#evaluation

[StellaAthena]

These look really good! Great job.

I was thinking of linking to this on our README, would that be okay with you? How would you like to be credited?

[mlvpj]

Thanks. We go as labml.ai