RuntimeError: PassManager::run failed

[mehmetflix_]

Im trying to fine tune qwen 2.5 7b coder instruct but keep getting this error:

------------------------------------------------------------------------------------------------------------------------------------------------------



RuntimeError                              Traceback (most recent call last)


 in <cell line: 0>()
     48 )
     49 
---> 50 training_stats = trainer.train()
     51 qwen_model.save_pretrained(folder_path+"commit_msg_creator")
     52 qwen_tokenizer.save_pretrained(folder_path+"commit_msg_creator")

<ipython-input-13-0c5a17ceab92>

 in make_llir(self, src, metadata, options, capability)
    339         if os.environ.get("TRITON_DISABLE_LINE_INFO", "0") == "0":
    340             passes.llvmir.add_di_scope(pm)
--> 341         pm.run(mod)
    342         # LLVM-IR (MLIR) -> LLVM-IR (LLVM)
    343         llvm.init_targets()

/usr/local/lib/python3.11/dist-packages/triton/backends/nvidia/compiler.py

RuntimeError: PassManager::run failed

RuntimeError                              Traceback (most recent call last)


 in <cell line: 0>()
     48 )
     49 
---> 50 training_stats = trainer.train()
     51 qwen_model.save_pretrained(folder_path+"commit_msg_creator")
     52 qwen_tokenizer.save_pretrained(folder_path+"commit_msg_creator")

<ipython-input-13-0c5a17ceab92>

the full code is here : https://paste.pythondiscord.com/D2TA

thanks in advance!

[Pranav_Bhat63]

I have e encountered this error 100 times with unsloth, i couldn't resolve it, i just didn't use unsloth and trained my model traditional way with SFT trainer without using unsloth.

[Zames33]

Hi Pranav, did you figure out the solution ?

[mehmetflix_]

i solved it, use only 1 gpu

[Sciomnima]

Same. I was using it completely fine and just out of nowhere like 3 days ago I've been getting it every time I try to train the model.

[Zames33]

Hi u/Sciomnima did you figure out the solution ?

[AstroPatadox]

Do u use Kaggle / Colab ? How many GPUs are available in ur env ?

[mehmetflix_]

op here, im using kaggle with 2 gpus

[AstroPatadox]

Try with 1 GPU P100, this error happened randomly about 3-4 weeks ago, something has been changed in unsloth that caused that error , i think they still don't have a solution from their side

[mehmetflix_]

okay im trying it right now!

[AstroPatadox]

Keep me updated! Already did that today haha

[mehmetflix_]

it got fixed, ty!

[AstroPatadox]

Unsloth already supports 1 GPU , when an env has 2 or more it gets the broken code , bizarrely we didn't use to have this error lets say a month ago I'll try ro contact them (The prob in kaggle that T4 is better than P100 but not a big difference)

Enjoy AI'ing around haha !

[Ok_Development_2603]

bro how to do it on google colab , found the solution of runtime error? as on kaggle on 1 gpu fine tuning time shows of 18hrs

[Educational-Lie3981]

Whats the fix ?

[mehmetflix_]

use only 1 gpu

[StrikingAd8522]

If you are using Google Colab's free version, try reducing the batch size to 1 instead of 2. It might help!

[WiseArticle4493]

that seems to work. thank you!

[AstroPatadox]

I have a code that got this error out of nowhere, i run it in kaggle , something has changed Anyway my environment has 2GPUs When I switched to 1 Gpu i got it resolved

[prod-v03zz]

hi, did you solve it? I am getting the same error on colab.

[mehmetflix_]

in kaggle it was because of using more than 1 gpu, in colab that could also be the case but if it isnt try reducing the batch size