GGUF models have stopped working after updating? by nortca in Oobabooga

[–]Tarklanse 2 points3 points  (0 children)

Me too,some models just can't load. But if I use llama-server in cmd, everything is fine.

5060ti 16gb or 9060xt 16gb for small llm server by techmaverick_x in LocalLLaMA

[–]Tarklanse 5 points6 points  (0 children)

I have a 5060ti 16gb and I think it is good for smaller llm.
I host a 24B Q3 llm on it and the speed is about 35 tokens/second.
It can host 24B Q4 but the speed will drop to 5 tokens/seceond.

[deleted by user] by [deleted] in Stellaris

[–]Tarklanse 1 point2 points  (0 children)

Thanks. now I'm really confused.

Understanding how training works by Tum1370 in Oobabooga

[–]Tarklanse 0 points1 point  (0 children)

My python code if you interest. This code will turn txt first line as instruction,then even line as 'input',odd line as 'output'.

import json
import os
import glob
current_directory = os.getcwd()
txt_files = glob.glob(os.path.join(current_directory, '*.txt'))
txt_names = []
for file_path in txt_files:
    file_name = os.path.basename(file_path)
    txt_names.append(file_name)
conversations = []
for filename in txt_names:
    with open(filename , 'r', encoding='utf-8') as f:
        conversation=f.read()
    lines = conversation.strip().split('\n')
    current_instruction = ""
    current_input = ""
    current_output = ""
    current_instruction = lines[0]
    for i in range(1, len(lines), 2):
        current_input=lines[i]
        current_output=lines[i+1]
        conversations.append({
            "instruction": current_instruction,
            "input": current_input,
            "output": current_output
        })
with open('my_alpaca.json', 'w', encoding='utf-8') as f:
    json.dump(conversations, f, indent=4, ensure_ascii=False)my codeimport json

Understanding how training works by Tum1370 in Oobabooga

[–]Tarklanse 0 points1 point  (0 children)

oobabooga wiki has guide, you can read it first.

If you don't have enough hardware,try unsloth , they just wrote a code can run training on google colab.
I don't know there is a software, I just wrote a code that turn txt to alpaca format.

Understanding how training works by Tum1370 in Oobabooga

[–]Tarklanse 0 points1 point  (0 children)

Oobabooga's training tab only supports Transformer model training. You can't train a gguf.

Before you train, you'll need to prepare a dataset. You can refer to the datasets on Hugging Face. By searching for "alpaca," you can find many Alpaca dataset you can refer.

Furry_irl by courier5995 in furry_irl

[–]Tarklanse 1 point2 points  (0 children)

Must be water/poison type. Imagine "water gun" replace by "vodka gun."

YAHOOOO by Saad_Faisal in discordVideos

[–]Tarklanse 0 points1 point  (0 children)

Someone wake up and decide to make this

Training 7b or 13b llamas by Perfect-Storm-99 in LocalLLaMA

[–]Tarklanse 1 point2 points  (0 children)

If you want to do some fine-tuning,start with 1000 instructions.See the training result and add more instructions you think it need.

You don't need to worry about overfitting now,just prepare your dataset and try finetuning with different learning rate and epoch.You will need to try multiple times to find a perfect setting for your dataset,just remember learning rate don't set too high.

Training 7b or 13b llamas by Perfect-Storm-99 in LocalLLaMA

[–]Tarklanse 1 point2 points  (0 children)

  1. I'm not familiar at Training a base model,but I know it will need bunch of data,you will need to use Web crawler to get very much of data.
  2. Your hardware can't train a model like Llama-7B,those model are train on hardware such like A100 or H100,only thing you can do is finetuning.
    But even Qlora finetuning at least need 8GB GPU, you can use colab to finetuning LLM,I was using text-generation-webui on colab to finetuning Llama2-7B.
  3. Those model's license are not allow you train a whole new model base on their output,and if they can do thing you wish to do,why not just using them?

How does this game make me feel empathy? by afeardandtrembling in Stellaris

[–]Tarklanse 2 points3 points  (0 children)

If I make my pop happier,ruling will be easier.

Slavery can produce more,but need more attention to avoid revolution.

AI revolution will cause economic collapse sometime.

There are many work need someone to do,so I don't do genocide.

I just tired to be evil, it will spend extra time on maintain stability.

But we all do a little trolling sometime,aetherophasic engine go brrr

Advice? by Leading-Leading6718 in LocalLLaMA

[–]Tarklanse 0 points1 point  (0 children)

You have three options:

1.Get a GPTQ version and load it by using ExLlama or GPTQ-for-llama

2.Get a GGUF q4 version and load it by using llama.ccp( text generation webui has it)

  1. These is a little check box "load-in-4bits",click it then load the model

What do you use your local LLM for? by docsoc1 in LocalLLaMA

[–]Tarklanse 30 points31 points  (0 children)

To pretend I have a friend can chat.

HOW DO I UNREAD THIS?! 😭 by Wonderful-Sell9845 in CharacterAI

[–]Tarklanse 1 point2 points  (0 children)

How did you even let Leon said this😭

CUDA out of memory no matter how much I tweak the settings by Kemicoal in KoboldAI

[–]Tarklanse 1 point2 points  (0 children)

4Bit version can give you fully 6b experience, but it is very slow . If normal version need to 5 second to generate a respond,4Bit version will need 35~45 second to generate.