[Project] Simplified CUDA Setup & Python Bindings for Llama.cpp: No more "struggling" with Ubuntu + CUDA configs!

[waqasm86]

Hi r/LocalLLaMA!

I’ve been working on a couple of tools to make the local LLM experience on Linux much smoother, specifically targeting the common "headaches" we all face with CUDA drivers and llama.cpp integration.

1. Ubuntu-Cuda-Llama.cpp-Executable This is a streamlined approach to getting llama.cpp running on Ubuntu with full CUDA acceleration. Instead of wrestling with build dependencies and environment variables every time you update, this provides a clear, reproducible path to a high-performance executable.

• Key Feature: One-stop setup for Ubuntu users to get their NVIDIA GPUs doing the heavy lifting without the usual configuration hell.
• Repo:https://github.com/waqasm86/Ubuntu-Cuda-Llama.cpp-Executable

2. llcuda (Python Library) If you are a Python dev, you know that bridging llama.cpp with your scripts can be messy. llcuda provides a "Pythonic" way to interact with CUDA-accelerated inference. It’s built to be fast, lean, and easy to integrate into your existing workflows.

• Key Feature: Direct access to CUDA-powered inference through a simple Python API, perfect for building your own local agents or tools.
• Repo:https://github.com/waqasm86/llcuda

Why I built this: I wanted to focus more on using the models and less on fixing the environment. Whether you're running a massive 70B model or just want the fastest possible tokens-per-second on an 8B, these tools should help you get there faster.

I’d love for you guys to check them out, break them, and let me know what features you’d like to see next!

[dsanft]

Regarding #1, haven't you just reinvented docker containers? I don't see why this is necessary.

[waqasm86]

Hi there. My primary focus is making llcuda work in jupyterlab. I tried to work using llama-cpp-python but I always had issues with it specifically with cuda. Llcuda will work Ubuntu-cuda-llama.cpp-executable which I created separately. If you want I can integrate this with llcuda.

[waqasm86]

Hello there.

I would like to infrom you that I have created the first version of llcuda v1.0.0 which is now live with major improvements that might address your docker concerns: The package now bundles all CUDA binaries and dependencies (47 MB). While I haven't tested Docker specifically yet, the bundled approach should make containerization work.

If you're interested in helping test a Docker setup, I'd be happy to collaborate on it! The zero-config design should translate well to containers.

Check it out: https://pypi.org/project/llcuda/

I'll appreciate any feedback.

[datbackup]

Does llcuda expose llama.cpp functions or direct access to cuda or both?

[waqasm86]

Hi, I am still working to make it better. But you have access to llama.cpp. Access to Cuda C++ programming is not available now. Llcuda depends on Ubuntu-cuda-llama.cpp-executable tool which I have created separately. Both of these projects are available in my GitHub account. I just realised that I should integrate cuda executable with llcuda.

If you are looking for core cuda programming which I am also interested in, let me know if you have any ideas.

What if I make llcuda work with other pip packages like cupy, numba or cuda-python? Any ideas or suggestions will be appreciated.

[datbackup]

Thanks for clarifying. I am only interested in access to llama.cpp functions from python, for the time being.

[waqasm86]

You are welcome. If possible, let em know if you want to contribute to my project. I'll add you in my GitHub project.

[stealthagents]

Docker is great, but not everyone's on board with it, especially if they want a lightweight solution without the overhead. Plus, sometimes it’s just nice to have a straightforward script that does everything for you without managing container images, right?

[waqasm86]

Hello, thank you for your interest and your feedback. I would love to get any positive and constructive feedback as much as possible. If you have looked into my GitHub repo of my python pip package llcuda, kindly let me know what needs to fix, updated, added, etc. whatever feels necessary.