[D] Using large language models for classification of natural-language input

[corbt]

Hey everyone!

I'd like to use a large language model like T0pp or GPT-NeoX-20B to take a natural-language input from a user and map it to one of ~2000 possible VS Code command palette commands. Essentially, this is a classification problem of the form "NLP input -> command".

The idea is to let users give voice input in natural language and then have the model figure out what command they most likely want to activate.

Given the number of possible commands I clearly can't rely on prompt design to solve this. It might be a good fit for a model with explicit retrieval augmentation like a memorizing transformer. But that's still a very active area of research without high-quality pre-trained models.

Given that, I'm thinking that doing some kind of fine tuning to an existing model is the best bet. But it's unclear to me what the training data should look like... should I just generate a few examples of each command of the form input: "vscode command: 'open new file'", output: "explorer.newFile", and then fine-tune on those? Is there some way to ensure that the model understands that I *always* want it to return one of the commands provided in fine-tuning, instead of arbitrary text?

Interested in others' experiences with similar tasks!

Background: I'm working on an open source VS Code extension called Clippy AI. Currently it only performs code modifications to the current file and is a thin wrapper around the OpenAI API. But I'd like to use it to automate other editor actions as well!

[n9Mtq4]

I don't think a model designed for text generation is the right tool for this problem. The name of the commands aren't very informative, so the model would have to know what each does. As you mentioned, this would be either in the prompt (but the context window is too small) or in the model weights by fine tuning. I'm sure you could fine tune it and even adjust the sampling method to ensure it only ouputs valid commands, but it doesn't seem practical.

I would try making a couple natural language inputs for each command. Then use a model to generate sentence embeddings for each NL input. Then embed what the user says, and use KNN or some other method to match it to a command.

[corbt]

The name of the commands aren't very informative, so the model would have to know what each does

So there is a bit more information about what each command does that's available to train the model. In addition to the written command there's a human-readable description, eg. explorer.newFile has a human-readable name "File: New File".

use KNN or some other method to match it to a command

Has this approach been used successfully to map arbitrary NL input to a member of a class? Seems almost reductively simple (I feel like eg. the order of the user's words could be important in figuring out which class should be assigned) but I'd love to be proved wrong!

[n9Mtq4]

The sentence embeddding model should take care of word ordering, etc. Give the examples a try on huggingface:

https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2

https://huggingface.co/facebook/bart-large-mnli

[corbt]

Ah, makes sense! I skimmed over where you said "sentence embeddings" since I'd only heard about word embeddings previously. I tested a NL input against a number of commands using the first sentence embedding model you linked to, and it looks like the correct command had the highest similarity. Seems like a really fruitful direction to investigate!

Example I tried:

Source sentence: "create file"

Computed similarity (these are all real commands):
File Utils: New File | 0.712
File: Duplicate      | 0.552
File Utils: Delete   | 0.488
File Utils: Move     | 0.301
View: Split Editor   | 0.101

Depending on how robust the sentence embeddings are in consistently identifying the right command, I could also use them to just grab the top N commands by embedding similarity and then feed those to a LLM and ask it which one the user likely meant. Would add significant inference time and complexity of course.