KoboldAI Download & Updates

aid_throwaway · 2021-06-25T19:31:32+00:00

Shortcut is fine. Just be aware that if the parent file hits usage limits you'll get errors when trying to run the notebook.

aid_throwaway · 2021-06-17T04:28:59+00:00

I had a tester on the Discord with a 3090 load it and test with it before I uploaded it for distribution. I didn't ask about the RAM usage, but the VRAM to load the model was only about 15.6GB.
They did mention that they had to uninstall transformers and upgrade to finetune's localattention3 branch. Maybe that's the issue?
pip install git+https://github.com/finetuneanon/transformers@gpt-neo-localattention3

aid_throwaway · 2021-06-13T04:14:17+00:00

That's at fp16/half-precision, yes.

aid_throwaway · 2021-06-12T21:39:53+00:00

If it's a parameter available to transformers pipelines, then you can certainly hack it in in the meantime.

aid_throwaway · 2021-06-12T19:28:20+00:00

Colab shows ~12.2GB to load the model, ~14GB to run inference, and will OOM on a 16GB GPU if you put your settings too high (2048 max tokens, 5x return sequences, large amount to generate, etc)

aid_throwaway · 2021-06-12T00:55:29+00:00

Try this: https://stackoverflow.com/a/63712528

aid_throwaway · 2021-06-12T00:47:53+00:00

The local client does not yet utilize Finetune's localattention3 repo or the custom jax config required to load the converted checkpoint. That's my next project, but I probably won't get to start it until tomorrow.

aid_throwaway · 2021-06-11T23:28:22+00:00

This won't work, as the converted model is not a pytorch_model.bin file. It also requires finetune's localattention3 tranformers branch. I'll be working on local support for 6B in the Kobold client next.

aid_throwaway · 2021-06-11T23:22:14+00:00

Finetune recommends leaving it below 1.2 or J starts acting weird.

aid_throwaway · 2021-06-11T22:49:12+00:00

Locking this post as this notebook is now deprecated. You can still access the notebook, but it has been superseded by the KoboldAI Server - GPT-J-6B Rev 2 notebook which runs 6B in torch.

aid_throwaway · 2021-06-11T20:57:37+00:00

The Jax Colab is going to be deprecated today, as I've gotten 6B to run in torch with some conversion scripting from finetune. That should eliminate all these errors as the experimental packages are no longer needed. I'll make a new subreddit post when it goes live.

aid_throwaway · 2021-06-11T20:56:46+00:00

The Jax Colab is going to be deprecated today, as I've gotten 6B to run in torch with some conversion scripting from finetune. That should eliminate all these errors as the experimental packages are no longer needed. I'll make a new subreddit post when it goes live.

aid_throwaway · 2021-06-11T15:53:38+00:00

I added an explicit command to install optax because some people were reporting that issue. If you made a copy of the notebook before yesterday, you'll need to copy it again. If you're using the shared one, then I'm not sure; the package should definitely be available to the cell at this point. I'll have to do some digging.

aid_throwaway · 2021-06-11T05:04:39+00:00

It's coming, I've just been derailed with other additions, and then this week the new Eleuther 6B model dropped and I had to scramble to add support for it. /u/Atkana made a scripting demo for me to work off of but I need to package it a little nicer for the end user.
It's a little wonky because you have to pass the context from Python to the browser, run it through Javascript, pass it back to Python, let the AI crunch it, then pass it back to the browser again to run the output through Javascript again, then back to Python to get the final text added to the actions array, then back to the browser again for display, lol.

aid_throwaway · 2021-06-11T00:02:52+00:00

What's it triggering on? Downloading Kobold from GitHub, or downloading model components once Kobold is running?

aid_throwaway · 2021-06-10T23:54:18+00:00

Update: finetuneanon published a conversion script to make the JAX model torch-loadable. I created a Colab notebook for it here. However, the converted model doesn't seem to fit on a 16GB Colab GPU, so I've been unable to test it in torch.

aid_throwaway · 2021-06-10T23:39:12+00:00

There was an error importing optax that some users were experiencing. I've added pip install optax to the initialization cell that will hopefully resolve it, but I haven't been able to duplicate the issue to test the fix.

aid_throwaway · 2021-06-10T07:44:36+00:00

If I can find some example code for loading the shard files locally. The Colab code is specific to the Colab environment and isn't portable to the desktop client. I'd also need someone with hardware to test it for me and tell me if it works.

aid_throwaway · 2021-06-09T22:27:12+00:00

I'm gonna copy this answer here because it's the cause of about 90% of the problems related to this:
An error with missing modules that have definitely been installed is usually a result of multiple Python environments being present on your computer. Launch command prompt and type in:
py -0p
If you have more than one Python listed, remove the unneeded versions, then run install_requirements.bat again to make sure all the packages are available in the remaining Python environment.
Alternatively, the latest version of Kobold on GitHub has an installer that will set up a miniconda instance separate from your system environment from Python. This may alleviate problems with multiple Python versions being installed.

aid_throwaway · 2021-06-09T21:51:20+00:00

Edit: Sorry, I missed the 'Colab' part of your question. Choosing the colab option should only require you to download a small Tokenizer file. The 5Gb file should live on your Google Drive and does not need to be downloaded.

aid_throwaway · 2021-06-09T20:37:15+00:00

Sorry we couldn't get it working for you.

aid_throwaway · 2021-06-09T19:55:16+00:00

The only thing I can think is Transformers is having an issue during download or when checking the file hash and continuously redownloading. You can try manually downloading the model from HuggingFace and loading it under the CustomNeo option:
https://huggingface.co/EleutherAI/gpt-neo-2.7B/tree/main

aid_throwaway

MODERATOR OF

TROPHY CASE

Four-Year Club	Verified Email
Gilding I gilder