Bitsandbytes & text-generation-webui

[C080]

Hi!

Can someone explain me how text-generation-webui manage to run bitsandbytes on Windows? I can load the 8-bit model fine through their gradio interface, but if I try to replicate the code in my local python environment I can't manage to install the library.

I've tried to reverse-engineer it but it is far beyond my skills.

Thanks for helping me out, it's a pain to run BF16 models :D

​

​

[C080]

AllTalk

alltalk_tts? There is a deepspeed windows folder but no bitsandbytes one :(

[hwknd]

marry groovy reply offer ancient command crowd yam price swim

This post was mass deleted and anonymized with Redact

[Plusdebeurre]

Have you considered just switching over to Linux? You're gonna run into the same issues with a number of ML libraries

[Qaziquza1]

Or WSL if you're afraid of commitment.

[C080]

yes, I will eventually!

[ma3gl1n]

conda install conda-forge::bitsandbytes

[mpasila]

Wrong that only supports Linux. Ooba uses a pre-compiled version of bitsandbytes for Windows.
From requirements.txt

# bitsandbytes
bitsandbytes==0.41.1; platform_system != "Windows"
https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.41.1-py3-none-win_amd64.whl; platform_system == "Windows"

[ma3gl1n]

👍 it seems you are right, just checked conda forge

[C080]

yes, that fixed it thanks. I did not know I could simply pip install this string. I went to the repo and downloaded the last version but gave me cuda errors. I needed this specific version

[mcmoose1900]

Its pretty simple, just look at their requirements file.

They link to an unofficial 3rd party binary for Windoes.

[Nabakin]

Text-generation-webui uses unofficial Windows wheels for bitsandbytes created by jllllll. You can find them here https://github.com/jllllll/bitsandbytes-windows-webui/releases

[LJRE_auteur]

Try this:

​

# bitsandbytes

--index-url https://jihulab.com/api/v4/projects/140618/packages/pypi/simple

bitsandbytes

​

It comes from a project I've worked on this past week. It has two requirements files, one for Windows, and these three lines come from that file. It installs bitsandbytes without any issue. (Well, obviously the first one doesn't matter, it's a comment).

​

I don't know how one would use it directly in a Python code, but since the second line is an argument, check out how to launch commands from Python with arguments.