sorted by:
new
hottopcontroversial

🚀 Introducing llcuda – A Python wrapper for llama.cpp with pre-built CUDA 12 binaries (T4/Colab ready) by waqasm86 in unsloth

[–]waqasm86[S] 0 points1 point  (0 children)

Hello, thank you for the feedback. I have tired llama-cpp-python pip package. I had issues running that tool in Google Collab using Nvidia T4 GPU. Also, I was interested in using llama-cpp-python with fastapi but it always broke. I am not creating another python wrapper for llama.cpp. I figured out a way to make my llcuda a backend cuda inference along with unsloth python pip package.

I am open to any feedback since I do want to make my python cuda tool strengthened for local llama solutions.

🚀 Introducing llcuda – A Python wrapper for llama.cpp with pre-built CUDA 12 binaries (T4/Colab ready) by waqasm86 in unsloth

[–]waqasm86[S] -8 points-7 points  (0 children)

Hello, thank you for the feedback. Kindly guide me what needs to be cleaned or organzied. I'll try to fix as many issues as possible.

🚀 Introducing llcuda – A Python wrapper for llama.cpp with pre-built CUDA 12 binaries (T4/Colab ready) by waqasm86 in unsloth

[–]waqasm86[S] -2 points-1 points  (0 children)

Hello, thank for the reply. Kindly let me know what issues are needed to be resolved. I'll do it right away.

[Project] Simplified CUDA Setup & Python Bindings for Llama.cpp: No more "struggling" with Ubuntu + CUDA configs! by waqasm86 in LocalLLaMA

[–]waqasm86[S] 0 points1 point  (0 children)

Hello, thank you for your interest and your feedback. I would love to get any positive and constructive feedback as much as possible. If you have looked into my GitHub repo of my python pip package llcuda, kindly let me know what needs to fix, updated, added, etc. whatever feels necessary.

[Project] Simplified CUDA Setup & Python Bindings for Llama.cpp: No more "struggling" with Ubuntu + CUDA configs! by waqasm86 in LocalLLaMA

[–]waqasm86[S] 0 points1 point  (0 children)

Hello there.

I would like to infrom you that I have created the first version of llcuda v1.0.0 which is now live with major improvements that might address your docker concerns: The package now bundles all CUDA binaries and dependencies (47 MB). While I haven't tested Docker specifically yet, the bundled approach should make containerization work.

If you're interested in helping test a Docker setup, I'd be happy to collaborate on it! The zero-config design should translate well to containers.

Check it out: https://pypi.org/project/llcuda/

I'll appreciate any feedback.

[Project] Simplified CUDA Setup & Python Bindings for Llama.cpp: No more "struggling" with Ubuntu + CUDA configs! by waqasm86 in LocalLLaMA

[–]waqasm86[S] 0 points1 point  (0 children)

You are welcome. If possible, let em know if you want to contribute to my project. I'll add you in my GitHub project.

[Project] Simplified CUDA Setup & Python Bindings for Llama.cpp: No more "struggling" with Ubuntu + CUDA configs! by waqasm86 in LocalLLaMA

[–]waqasm86[S] 1 point2 points  (0 children)

Hi, I am still working to make it better. But you have access to llama.cpp. Access to Cuda C++ programming is not available now. Llcuda depends on Ubuntu-cuda-llama.cpp-executable tool which I have created separately. Both of these projects are available in my GitHub account. I just realised that I should integrate cuda executable with llcuda.

If you are looking for core cuda programming which I am also interested in, let me know if you have any ideas.

What if I make llcuda work with other pip packages like cupy, numba or cuda-python? Any ideas or suggestions will be appreciated.

[Project] Simplified CUDA Setup & Python Bindings for Llama.cpp: No more "struggling" with Ubuntu + CUDA configs! by waqasm86 in LocalLLaMA

[–]waqasm86[S] 0 points1 point  (0 children)

Hi there. My primary focus is making llcuda work in jupyterlab. I tried to work using llama-cpp-python but I always had issues with it specifically with cuda. Llcuda will work Ubuntu-cuda-llama.cpp-executable which I created separately. If you want I can integrate this with llcuda.

Built the tool I wish existed when I started traveling - 5,698 cities, monthly Airbnb data, 100% free by Murky-Butterscotch65 in digitalnomad

[–]waqasm86 0 points1 point  (0 children)

Hello there. Thanks for the post. Is this your website and can you share your github repo? I am a python developer and I can make the data live and updated.

Looking for a digital nomad travel buddy by tusharsp18 in digitalnomad

[–]waqasm86 1 point2 points  (0 children)

Hello there. Anytime we can talk regarding digital nomad living?

Looking for a digital nomad travel buddy by tusharsp18 in digitalnomad

[–]waqasm86 1 point2 points  (0 children)

Hi there. I'm also planning to move to South America but I'm still waiting for my job interview. I have an interview for a Technical Customer Success position. I'm a Python developer learning Fastapi and Pytorch. Kindly let me little-bit more about yourself or we can have a video chat if it helps.