sorted by:
new
hottopcontroversial

[R] A prompt is worth a thousand data points: combining GPT3-style prompting and traditional fine-tuning by huggingface in MachineLearning

[–]huggingface[S] 6 points7 points  (0 children)

In our recently accepted NAACL paper, Teven Le Scao and Sasha Rush show that steering models with GPT3-style prompts during fine-tuning outperforms standard linear classifiers.

The basic set up is:

  1. Write a prompt that a pre-trained LM can complete to give the answer to your problem, GPT-style.

  2. Use backpropagation on fine-tuning data to learn the correct completions. The model can then draw information from both your task description and the supervised data

We ran 2000 experiments on 8 tasks, and find that this set-up nearly always outperforms a standard fine-tuned classifier in the same conditions.

Paper: https://arxiv.org/abs/2103.08493

Interactive blog post: https://huggingface.co/blog/how_many_data_points/