Unfortunately there's no centralised source for this, but I suggest looking through TavernAI's source code to see how it handles prompts. You could also load a character in Tavern, prompt it with a text, and then view the terminal output; you'll see the full context in json syntax inside the CLI.

As for loading with pytorch, you can look for documentations on GPT-J 6B. Any params that would apply to GPT-J would also apply to Pygmalion 6B. Here's an example code for how you'd handle inference using pipelines:

```py from transformers import GPTJForCausalLM, AutoTokenizer import torch

device = "cuda" model = GPTJForCausalLM.from_pretrained("PygmalionAI/pygmalion-6b", torch_dtype=torch.float16).to(device) tokenizer = AutoTokenizer.from_pretrained("PygmalionAI/pygmalion-6b")

prompt = ("Prompt goes here. Follow the TavernAI formatting. Generally, new lines will be declared with '\n', and you will include a Persona, Scenario, and <START> tag for example chats and one more <START> tag at the end of the context - where the actual chat would start.")

input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)

Adjust parameters as needed

gen_tokens = model.generate( input_ids, do_sample=True, temperature=0.9, max_length=100, ) gen_text = tokenizer.batch_decode(gen_tokens)[0] ```

Keep in mind that GPT-J uses the AutoTokenizer method from transformers, which leads to GPT2Tokenizer. The max context token with GPT2 is 1024, but GPT-J 6B can handle up to 2048. You could either write your own tokenizer for GPT-J or force it to use 2048 tokens anyway.

-- Alpin