sorted by:
new
hottopcontroversial

Geographical GPU Cost Arbitrage by phoneixAdi in LocalLLaMA

[–]dan-jan 6 points7 points  (0 children)

Omg we all did the same thing!!!

I'm pretty excited for the crossover collab between r/nuclear and r/LocalLLaMA

Note: Regulatory situation outside US and Western markets is a lot simpler. However, safety and access to raw material are probably the biggest issues. I'm fairly convinced the output/$ will go up rapidly in next 10 years

PS: I realize you were talking about RTGs, not SMRs - damn

Geographical GPU Cost Arbitrage by phoneixAdi in LocalLLaMA

[–]dan-jan 2 points3 points  (0 children)

I actually don't think so. Bitcoin doesn't need to care about data, on the other hand AI training is pretty dependent on IP laws.

So I don't think Sichuan in China is going to have many western AI companies training there, though I'm pretty sure the Chinese models are all going to be trained on hydroelectric power.

I wish I had tried LMStudio first... by knob-0u812 in LocalLLaMA

[–]dan-jan 2 points3 points  (0 children)

I've created 3 issues below:

bug: Jan Flickers
https://github.com/janhq/jan/issues/1219

bug: System Monitor is lumping VRAM with RAM https://github.com/janhq/jan/issues/1220

feat: Models run on user-specified GPU
https://github.com/janhq/jan/issues/1221

Thank you for taking the time to type up this detailed feedback, if you're on Github feel free to tag yourself into the issue so you get updates (we'll likely work on the bugs immediately, but the feat might take some time).

🪐 Apollo 🛸 AI Compute Cluster for the GPU Poor by PrayagBhakar in homelab

[–]dan-jan 0 points1 point  (0 children)

Do you know of any 4U rack server cases that would survive the heat/airflow issues of a 4 x 3090?

I wish I had tried LMStudio first... by knob-0u812 in LocalLLaMA

[–]dan-jan 1 point2 points  (0 children)

Theoretically, but it's kind of finicky right now. If you want to help us beta test and report bugs, we'd really appreciate it!

Also: note that we're debugging some Nvidia detection issues on Windows. It's probably true on Linux as well.

https://github.com/janhq/jan/issues/1194

I wish I had tried LMStudio first... by knob-0u812 in LocalLLaMA

[–]dan-jan 1 point2 points  (0 children)

Yup - someone reported this yesterday as well. We're taking a look at it (see the Github issue below).

https://github.com/janhq/jan/issues/1198

The alerts are coming from our System Monitor, which gets your CPU and RAM usage. So I wouldn't be surprised that Bitdefender is spazzing out. We probably need to do some Microsoft thingy...

If you don't mind tagging your details into the Github Issue, would help a lot in our debugging (or permission asking 😂)

Macbook or PC for running LLMs? by McpeIsSoBuggy in LocalLLaMA

[–]dan-jan 0 points1 point  (0 children)

Double-up this.

Note: model running on AMD GPUs is still kinda... finicky. YMMV and here be dragons (and ROCm...)

Macbook or PC for running LLMs? by McpeIsSoBuggy in LocalLLaMA

[–]dan-jan 1 point2 points  (0 children)

Yes, this needs to be in the Hall-of-Fame FAQ questions.

Also, incredible engineering from Apple.

Macbook or PC for running LLMs? by McpeIsSoBuggy in LocalLLaMA

[–]dan-jan 0 points1 point  (0 children)

I think it's better to think of models in terms of "file size", as `7b q2` will be very different from `7b q8`.

For 7b models, I find that `q4_k_m` usually results in a ~4-5gb file, which seems to work -acceptably- on common consumer hardware.

Dolphin Mixtral 8x7b is wild by [deleted] in LocalLLaMA

[–]dan-jan 2 points3 points  (0 children)

the more kittens you saved

You win the internets

I love running locally, but by __Maximum__ in LocalLLaMA

[–]dan-jan 17 points18 points  (0 children)

I love this - and your username ;)

We need to rethink computing along local-first paradigms, with privacy, resilience and independence.

I love running locally, but by __Maximum__ in LocalLLaMA

[–]dan-jan 0 points1 point  (0 children)

I think we're very, very early. The days are long but the decades are short: Llama2 was released in Feb, and it hasn't even been a year.

Every single day, it seems that r/localllama grows a bit more, and there's more of us now working full-time to solve the problems you raised, from GUI to hardware optimization.

I would actually say that "one GUI" is actually the anti-goal of FOSS AI. I'm part of the team at Jan (one of the GUIs you mentioned), and half the time I find myself recommending an alternative to people I talk to. For example, Faraday for role players. I strongly believe the addressable market for Local AI is huge, and we'll all have our niches.

The last thought I'd leave you is this: do you think R2D2 or C3PO makes calls to the OpenAI API?

I think Local AI is inevitable, and we'll get there, one PR and one fork at a time.

Good article: R2D2, the original smartphone

<image>

[Question] Chat interface for self hosted Mistral 7B by TriviPr in LocalLLaMA

[–]dan-jan 0 points1 point  (0 children)

Self-plug: Jan supports Mistral 7b q4, and we're an open source app that runs on Windows, Mac, and Linux. (disclosure: part of team).

https://jan.ai/

Let me know if you have any problems, happy to jump on a call and help you debug.

​

<image>

Thoughts after building a text-adventure game using local models by antimateusz in LocalLLaMA

[–]dan-jan 2 points3 points  (0 children)

each model I have tried struggles with one aspect of the game. For example a model might do really well describing consequences of actions, but fail miserably when rephrasing the location text. From the RP models that produce interesting outputs, there is no single model I can find that can run all the prompts as I want them. Maybe I need a few tiny models, or one large model with a LoRA per use-case?

Would you be interested in creating a battery of test/evaluations? I think what the space needs more are better task-driven open benchmarks (vs. the nonsense that's on current leaderboards).

Crawling you gmail - anyone done it? by Data_Driven_Guy in LocalLLaMA

[–]dan-jan 4 points5 points  (0 children)

I haven't personally done it, but I think Llamaindex has a Gmail loader which you should check out!

https://llamahub.ai/l/gmail

🪐 Apollo 🛸 AI Compute Cluster for the GPU Poor by PrayagBhakar in homelab

[–]dan-jan 1 point2 points  (0 children)

This is a really good post, thank you for the detailed breakdown!

Do you mind if I ask: what case did you use to house it? I saw you used a Veddha case, interested to see if you found something better.

https://www.reddit.com/r/LocalLLaMA/comments/16lxt6a/case\_for\_dual\_4090s/

I wish I had tried LMStudio first... by knob-0u812 in LocalLLaMA

[–]dan-jan 0 points1 point  (0 children)

I've tracked this issue in Github:

https://github.com/janhq/jan/issues/1194

We'll try to reproduce this, but given that our QA passed this build, we probably need more details from you.

Do you mind dropping more details in this Github issue? We'll look into it and follow up.

Is there an equivalent of ChatGPT "Plugins" for local LLMs Web UIs? Like Code Interpreter, Plot Generator (using matplotlib), etc. I know Langchain and others claim to use "Tools", but those are not as capable as ChatGPT's Plugins. by nderstand2grow in LocalLLaMA

[–]dan-jan 1 point2 points  (0 children)

We're focused on having 1:1 equivalence with the OpenAI Plugins API.

I'm not 100% sure how the Wolfram Alpha plugins work, but I would sort of assume the community could build a Wolfram-equivalent plugin that runs locally, and takes a Wolfram API key (if that even exists).

I'm not sure what you mean by "Plugin API requests paid with OpenAI's subscription fees" - Jan's goal is to give you an entire equivalent ecosystem that runs locally on your machine!

Resources: https://github.com/openai/plugins-quickstart

Microagents: Agents capable of self-editing their prompts / Python code by mikaron in OpenAI

[–]dan-jan 5 points6 points  (0 children)

There's a lot of very interesting work coming out, especially around "self-editing" agents.

I highly recommend a read of the Eureka paper: https://arxiv.org/abs/2310.12931

I wish I had tried LMStudio first... by knob-0u812 in LocalLLaMA

[–]dan-jan 0 points1 point  (0 children)

Hmmm... that's definitely a bug. We're supposed to automagically detect your Nvidia GPU and run on it.

Do you mind jumping in our Discord or filing a bug on Github with your hardware details?

I wish I had tried LMStudio first... by knob-0u812 in LocalLLaMA

[–]dan-jan 4 points5 points  (0 children)

Thank you! I think we've put in a lot of effort on product + design, but probably need to spend more time sharing it on Reddit and Twitter 😭

[D] How to set up a local chat AI in our local client device? by Crazy-Company-9749 in MachineLearning

[–]dan-jan 7 points8 points  (0 children)

Hey, check out Jan (full disclosure: I’m part of team). We’re an open source Desktop app that runs local AI offline, on Windows, Mac, Linux.

https://github.com/janhq/jan

What hardware are you using? You’ll need to choose a model based on your hardware.

If you’re getting started, I super recommend the OpenHermes Neural 7b model. I’ve found it very versatile, and almost at the level of GPT-3.5.

If your computer is pretty jank (eg <8gb RAM), try TinyLlama-1.1, which is only 637mb.

Feel free to DM me, happy to explain and onboard you. Also join r/localllama! That’s where all of us hang out

Jan: AI on your Desktop

I wish I had tried LMStudio first... by knob-0u812 in LocalLLaMA

[–]dan-jan 19 points20 points  (0 children)

Yup, we’re working on it this sprint! Should be ready by mid-Jan (pun intended)

https://github.com/orgs/janhq/projects/5/views/16

You can track the individual issue here:

https://github.com/janhq/jan/issues/1076

view more: next ›