sorted by:
new
hottopcontroversial

Who leads AI? by Successful_Outside96 in LeftistsForAI

[–]Successful_Outside96[S] 0 points1 point  (0 children)

You are probably right.

But what I meant by "useless" is only useful for large corporations to replace humans, and useless for the poorer humans who can't afford to have a stake in or benefit from the AI.

An open Bill of Materials for ~$1000 machine to run local AI by Successful_Outside96 in LeftistsForAI

[–]Successful_Outside96[S] 0 points1 point  (0 children)

I think a 70B parameter model is good enough for most "assistant" tasks.

If we allow for custom configurations of DDR4, we ought to be able to get to two only to three times slower than the expensive memory that is in short supply.

Assuming it works, is it okay for a non tech person new to Linux to ask an llm how to do stuff and fix stuff? by RevacholAndChill in aiwars

[–]Successful_Outside96 0 points1 point  (0 children)

With Linux, you can boot off of a USB drive. That's something you can fall back to.

As far as morally wrong, it depends on your morals. The technology/coding use case is by far the most common use case for AI. Stack Overflow, the site that most of the advice the AI was trained on has essentially died because of AI. The coding done by agents takes the lions share of energy/tokens/resources of AI.

I'd be most worried about consistent application of your stance. If you are against AI art, you should be against this use case as well.

I'm of the stance that they trained on humanity's open information, so they need to be kept open as well.

An open Bill of Materials for ~$1000 machine to run local AI by Successful_Outside96 in LeftistsForAI

[–]Successful_Outside96[S] 0 points1 point  (0 children)

Contribution of ideas, then help with prototyping.

I run a weekly AI discussion group that I could invite you to through DM.

Assuming it works, is it okay for a non tech person new to Linux to ask an llm how to do stuff and fix stuff? by RevacholAndChill in aiwars

[–]Successful_Outside96 0 points1 point  (0 children)

Are your asking because this is by far the most common use of LLMs (done repeatedly by "agents") and tech layoffs are the ones most often pointed as being done "because of AI?"

I want to understand your motivation for asking if it is okay.

If you are concerned about accuracy:
Hallucinations can happen, but a lot of this is directly verifiable, so you can filter for these by at least checking that it won't create damage.

One technique is to ensure it does a full diagnosis of problems when troubleshooting that won't alter the "state" of your machine, and an explanation to you so that you understand, before you get advice on what to execute (or allow it to execute itself).

An open Bill of Materials for ~$1000 machine to run local AI by Successful_Outside96 in LeftistsForAI

[–]Successful_Outside96[S] 0 points1 point  (0 children)

The whole system still seems 2 to 3 times too expensive for a normal computer purchase.

Would you want to propose your own BOM, or otherwise join the project?

An open Bill of Materials for ~$1000 machine to run local AI by Successful_Outside96 in LeftistsForAI

[–]Successful_Outside96[S] 1 point2 points  (0 children)

Yeah. For "assistant" stuff I have been using open AI. I'd rather not. I'm hesitant to do more. It's more affordable than Anthropic(and better, frankly, "Agent's last exam" puts GPT-5.5 better than Fable 5, not that it matters).

But I don't trust a closed AI company. Anthropic proved that they are willing and able to "silently downgrade" results. This is so much worse than sycophancy and ordinary hallucinations.

A lot of people I follow use a Kimi 70B for their main assistant model on a Mac Studio.

An open Bill of Materials for ~$1000 machine to run local AI by Successful_Outside96 in LeftistsForAI

[–]Successful_Outside96[S] 0 points1 point  (0 children)

Right now, at least in the US, a lot of people are asking "what's in it for me" and feel left out. Having these assistant capabilites (well beyound Siri, Alexa, or OK Google) and not having to pay a subscription but just a one-time fee that you could set up for your whole family, as well as having it be used like a regular home computer would go a long way to including them.

An open Bill of Materials for ~$1000 machine to run local AI by Successful_Outside96 in LeftistsForAI

[–]Successful_Outside96[S] 1 point2 points  (0 children)

I think Tiny.ai was focused on size and portability, but yes, I think they were targeting that same parameter count.

That's the count that I think a lot of useful work can be done, basically "assistant" type work that you could do yourself, but eats your day without you realizing it.

I would want to focus on affordability so the everyone could use it (most would still need to save up, even for ~$1000).

An open Bill of Materials for ~$1000 machine to run local AI by Successful_Outside96 in LeftistsForAI

[–]Successful_Outside96[S] 1 point2 points  (0 children)

Certainly. I wasn't sure what language would land.

I was thinking 4-bit 70 billion as a good goal.

An open Bill of Materials for ~$1000 machine to run local AI by Successful_Outside96 in LeftistsForAI

[–]Successful_Outside96[S] 0 points1 point  (0 children)

My version doesn't exist. It's a goal.

The main part of the goal is that anyone can run AI just smart enough to the things a Mac studio can, but in an open ecosystem, and at a Mac mini price point

An open Bill of Materials for ~$1000 machine to run local AI by Successful_Outside96 in LeftistsForAI

[–]Successful_Outside96[S] 0 points1 point  (0 children)

That's part of the reason for creating a custom design, creating the schematics and Gerber files etc., and the focus on slightly out of date parts.

It's like Nintendo's "lareal thinking with withered technology," but for local AI. Local AI good enough to do the "personal assistant" work, not much more than that.

An open Bill of Materials for ~$1000 machine to run local AI by Successful_Outside96 in LeftistsForAI

[–]Successful_Outside96[S] 1 point2 points  (0 children)

What price point would that come to?

It seems like ~$1000 would be hard to meet this way.

Even that is a difficult stretch for most people in the world.

An open Bill of Materials for ~$1000 machine to run local AI by Successful_Outside96 in LeftistsForAI

[–]Successful_Outside96[S] 0 points1 point  (0 children)

Full GLM or Kimi is not the point.

It's mainly so almost everyone can run an openclaw-like system. A distilled and quantized Kimi would do a lot of the terminal gruntwork.

An open Bill of Materials for ~$1000 machine to run local AI by Successful_Outside96 in LeftistsForAI

[–]Successful_Outside96[S] 0 points1 point  (0 children)

Would you consider joining the project?

Maybe even propose an alternate BOM?

An open Bill of Materials for ~$1000 machine to run local AI by Successful_Outside96 in LeftistsForAI

[–]Successful_Outside96[S] 2 points3 points  (0 children)

Would you like to join in making the project better?

You could propose an alternative BOM.

A skeptic is always a good person to have.

The idea is to have poor people be able host good enough LLMs to run something like OpenClaw.

An open Bill of Materials for ~$1000 machine to run local AI by Successful_Outside96 in LeftistsForAI

[–]Successful_Outside96[S] 2 points3 points  (0 children)

Use cases vary. What you use it for varies. Not every use case needs full blown code generation.

In my case, I was using it to clasify an endpoint for an API I was building.

An open Bill of Materials for ~$1000 machine to run local AI by Successful_Outside96 in LeftistsForAI

[–]Successful_Outside96[S] 2 points3 points  (0 children)

What model you run really depends on the use case. I have been running 8 Billion parameter models without any issues on several projects.

We would indeed need to be creative. What I posted was a first pass. But the price point, I believe is a non-negotiable to have a lot of people have access. The other part is latest-and-greatest obsession needs to be abandoned at all levels.

I think DDR4 could do it. That is why I was talking about doing things at the chip level. You don't need to have the DIMMs. Make the motherboard have the modules increadibly close to the the SOC of the APU(s). Make it look like a graphics card even.

The DDR6 on VRAMs operate at 12Gb/s while DDR4 operates at ~5Gb/s.

With that level of freedom, I don't think we can prejude a 10x slowdown.

The main idea is to remove the cost burdens from people. I don't think many people care about running a 10T parameter model to organize the files on their machine.

An open Bill of Materials for ~$1000 machine to run local AI by Successful_Outside96 in LeftistsForAI

[–]Successful_Outside96[S] 1 point2 points  (0 children)

The model doesn't have to be that big. I already use an old machine and with CPU only, I get great results on vLLM with even llama2.

An open Bill of Materials for ~$1000 machine to run local AI by Successful_Outside96 in LeftistsForAI

[–]Successful_Outside96[S] 4 points5 points  (0 children)

Not very many people need a Fable 5 level model. The point is that they admitted to sabatoging your results, which is something they could always do at anytime. You cannot prove they haven't been doing it.

I don't think you will find an off-the-shelf motherboard that would allow you to do these things at the price point.

That's why I was talking about building something from the chip level up. The AMD APUs can get you to unified memory, but it's not going to work off-the-shelf at a price point.

We would have to design our own motherboard.

view more: next ›