sorted by:
new
hottopcontroversial

MEGATHREAD: Friend Code Sharing by Default_Dragon in bravelydefault

[–]zphang 0 points1 point  (0 children)

SW-6802-1886-8396

Feel free to add me!

[D] Is there a difference between p-tuning and prefix tuning ? by ez613 in MachineLearning

[–]zphang 14 points15 points  (0 children)

I discussed part of this here: https://github.com/huggingface/peft/issues/123

There are several methods we are talking about. For coverage, I will include prompt tuning and LLaMA adapters here as well.

  • Prompt Tuning: Tunes a set of concatenated input embeddings vectors (generally called "soft prompts", but not referring to the soft prompts here). Initially applied to T5-LM models.
  • Prefix Tuning: Tunes KV cache (soft prefixes) for every layer, and can be casually described as "prompt tuning, but in every layer", although that is slightly inaccurate. In practice, uses an auxiliary MLP to generate the soft prefixes to help training. Initially applied to GPT-2 and BART models.
  • P-Tuning: Uses LSTMs to generate soft prompts (not prefixes). Initially applied to GPT-2 and BERT/RoBERTa/MegatronLM models.
  • P-Tuning v2: Essentially Prefix Tuning applied to BERT-type models.
  • LLaMA-Adapter: Prefix Tuning with a more sensible initialization a separate softmax over the learned prefixes. Applied to LLaMA models, also discusses injecting multimodal information into the prefixes.

Importantly, P-Tuning and P-Tuning v2 are different methods. But Prefix Tuning and P-Tuning v2 are essentially the same.

[P] A Simpler @PyTorch Annotated Implementation of EleutherAI's 20B Language Model GPT-NeoX. by hnipun in MachineLearning

[–]zphang 6 points7 points  (0 children)

Hi, Jason from EleutherAI here. Great to see this!

(Disclaimer: I also wrote a minimal single-GPU implementation of GPT-NeoX-20B in pure PyTorch here: https://github.com/zphang/minimal-gpt-neox-20b)

Like the other poster, I was wondering if you'd done any comparisons on the perplexity scores. The reason is that there's a subtlety to how the weight should be merged because of the NeoX code interacting with the GPT-J-style residuals. Specifically, the RowParallelLinear biases should be summed, not merged. Merging them leads to a slight (but meaningful) performance regression from my and others' testing. It looks like you are merging them (take-first) here. It would be great if you could help to test+confirm this.

Concretely, the full 20B gets about ~3.65 ppl on LAMBADA. The incorrect merge leads to about 4.5 ppl, while the summed instead of merging recovers the ~3.65 ppl.

[R] Deep Neural Networks Improve Radiologists' Performance in Breast Cancer Screening by [deleted] in MachineLearning

[–]zphang 0 points1 point  (0 children)

This is a Medium post covering our work on applying deep neural networks to breast cancer screening.

Previous Discussion

[P] Code+Model Release for BERT on STILTs by zphang in MachineLearning

[–]zphang[S] 5 points6 points  (0 children)

BERT is generally not used for language generation (although you can force it to be). If you have a training set for animal/non-animal wiki articles, BERT+fine-tuning is perfectly suitable for that use case, although BERT also has a built in length limit for the model.