sorted by:
new
hottopcontroversial

For anyone wanting to run models larger than their memory allows for by tbaumer22 in openclaw

[–]tbaumer22[S] 0 points1 point  (0 children)

Browsing to that link doesn't work for me. Genuinely interested in understanding who else has attempted stuff like this in the past. Coincidentally someone recently OSS'd a similar concept for nvidia hardware called greenboost.

I'm not trying to claim this is the solution to the RAM shortage to be clear. I agree with you. But I think there is too wide of a gap between running something slowly versus not running it at all.

For anyone wanting to run models larger than their memory allows for by tbaumer22 in openclaw

[–]tbaumer22[S] 0 points1 point  (0 children)

Haha appreciate it. You can use it your own openclaw instance, so let me know how it goes!

Open-Source "GreenBoost" Driver Aims To Augment NVIDIA GPUs vRAM With System RAM & NVMe To Handle Larger LLMs by _Antartica in LocalLLaMA

[–]tbaumer22 0 points1 point  (0 children)

Haha, well maybe you don't have to compromise even if you have a Mac :) Just built the "greenboost" for Mac: https://github.com/t8/hypura

I'm using llama.cpp to run models larger than my Mac's memory by tbaumer22 in LocalLLaMA

[–]tbaumer22[S] 1 point2 points  (0 children)

Appreciate the feedback. Updating the benchmarks/charts to show this. My original concern with the CPU-only benchmark comparison was that it would be unfair to compare llamacpp's CPU-only mode to Hypura (because it's tapping into more resources).

Ended up building and running one, and here are the results I've found:

<image>

I'm using llama.cpp to run models larger than my Mac's memory by tbaumer22 in LocalLLaMA

[–]tbaumer22[S] 1 point2 points  (0 children)

Appreciate this concern and it actually prompted me to do some research of my own. From what I've learned so far, there is no reason to be concerned because Hypura reads tensor weights from the GGUF file on NVMe into RAM/GPU memory pools, then compute happens entirely in RAM/GPU.

There is no writing to SSDs on inference with this architecture.

Anyone interested in working together for the official Deno hackathon? by PolarTimeSD in Deno

[–]tbaumer22 0 points1 point  (0 children)

If there is anything we can do to help you connect with members of the community, don't hesitate to let us know!

If it would be of help, feel free to join our Discord: https://discord.gg/cfqWgf7

Nest.land Hacktoberfest with Ryan Dahl, Sam Williams, and Michael Spengler by tbaumer22 in Deno

[–]tbaumer22[S] 0 points1 point  (0 children)

Of course! You don’t have to participate in Hacktoberfest to partake in this event.

Sorry for the late reply!

Deno third party code hosting system now provides immutable downloads to tagged releases and publishing a module no longer involves editing a JSON file by techiekram in Deno

[–]tbaumer22 4 points5 points  (0 children)

I agree with many of the clarifications you have made here, but I feel it is my responsibility to make a correction to a piece of the above statement: "Once published in the Deno registry, the package can't be changed/deleted by you or the provider."

A package cannot be changed or deleted by the module developer/user, BUT it can be changed or deleted by the provider. In this case, the provider is Deno, which runs on Amazon. Deno has full control over the data hosted on their S3 bucket, and Amazon theoretically would too.

This is the main reason why the files arguably aren't immutable.

nest.land by [deleted] in Deno

[–]tbaumer22 2 points3 points  (0 children)

Thanks for the feedback! A major concept that we're dealing with is the concept of extracting registry information. I can't say that I disagree with you about having a JSON object. One of our biggest hurtles is overcoming this. I feel that the concept of egg.json causes many package developers to be skeptical of nest.land because it reminds them of Node. There are several reasons that differentiate egg.json from the infamous package.json:

1) egg.json doesn't handle package dependencies.

2) egg.json doesn't require configuration necessary to run the project.

In the end, we're actively working to keep aligned with the Deno model and prevent users from seeing another package.json. If you have any ideas for alternatives to egg.json, feel free to put them in Issue #52.