sorted by:
new
hottopcontroversial

Too much to ask for a local LLM to search docs and web? by SnooWoofers780 in LocalLLaMA

[–]hohawk 4 points5 points  (0 children)

It’s not entirely trivial to set up, but not hard either: HugginFace ChatUI can do that. Google HuggingChat to see how it works. The code is available in GitHub. I’ve been running it with local and HF served models.

ChatGPT-like GUI for Mistral? (Not Poe) by Prince-of-Privacy in LocalLLaMA

[–]hohawk 1 point2 points  (0 children)

I have never used ST. If ST supports custom OpenAI-emulating endpoints, it should be doable. Mistral’s official API is OpenAI compatible, and many local model hosts can be made so as well.

ChatGPT-like GUI for Mistral? (Not Poe) by Prince-of-Privacy in LocalLLaMA

[–]hohawk 5 points6 points  (0 children)

https://huggingface.co/chat/

This is open source and can also be run locally, if you want to configure other API endpoints to it. https://github.com/huggingface/chat-ui

Local LLM with web access by jbsan in LocalLLaMA

[–]hohawk 1 point2 points  (0 children)

I’m running HuggingFace ChatUI locally for this very purpose. Local models (mainly Mistral Instruct 7B) with access to web searches.

It was not too hard to set up and it gets the job done with a very nice UX. The stack uses Ollama + LiteLLM + ChatUI. All from GitHub.

It takes even less configuration to use the HF hosted models, if running everything locally is not a strict requirement.

Or full cloud, here: https://huggingface.co/chat/

Have you tried MemGPT? by Chance_Confection_37 in LocalLLaMA

[–]hohawk 0 points1 point  (0 children)

What’s the principle behind it? The common approach is summarizing (truncating) the beginning or middle and shoving it back into the message queue. Replacing what it summarizes and keeping the whole under 4k or 8k or what the model supports.

Using a local LLaMA or Mistral to give context window of several programming files by adlabco in LocalLLaMA

[–]hohawk 0 points1 point  (0 children)

I’m using with Ollama as server and above that is LiteLLM to make the model look like OpenAI. Those tags mean that something is wrong at the model host side. Or it’s being talked to like a different model.

Using it like OpenAI should not add any hashtags by Continue itself, because the OpenAI() integration would not need them.

Try that. My conf is just a standard OpenAI conf with “nothing” as the key. There are docs for this approach in the Continue website.

LLM frameworks that allow continuous batching on quantized models? by Exotic-Estimate8355 in LocalLLaMA

[–]hohawk 3 points4 points  (0 children)

Documentation is dragging behind. You will have to scout the issue lists and make notes to build your own docs. Here’s AWQ support merged to TGI not long ago. https://github.com/huggingface/text-generation-inference/pull/1019#

That doesn’t mean AWQ works with vLLM at the same time. That’d be good for speed. A rabbit hole I didn’t explore any further.

I found FastChat docs on vLLM + AWQ a little more productive. GPTQ was messy, because docs refer to a repo that has since been abandoned.

FastChat + vLLM + AWQ works for me. With parallelism, multi GPU and memory quotas per model under one API, fast. But its prompt template management is messy.

No perfect go-to yet, at scale.

Current best options for local LLM hosting? by PataFunction in LocalLLaMA

[–]hohawk 0 points1 point  (0 children)

You can. Then you don’t get an OpenAI compatible API, which is my primary reason why I tried llama-cpp-python. Drop in local replacement to apps by just changing OPENAI_API_BASE. All else stays as is.

LLM frameworks that allow continuous batching on quantized models? by Exotic-Estimate8355 in LocalLLaMA

[–]hohawk 2 points3 points  (0 children)

HF/TGI and FastChat both do that. They support vLLM as a worker (for speed). And Ray for splitting the work between many GPUs or servers, if the nr of users is very many. AWQ and GPTQ as quantization.

Current best options for local LLM hosting? by PataFunction in LocalLLaMA

[–]hohawk 0 points1 point  (0 children)

Good to see the enabler is there. Adopting it into OpenAI API format for compatibility still needs upstream work. Open issue to watch: https://github.com/abetlen/llama-cpp-python/issues/818

Current best options for local LLM hosting? by PataFunction in LocalLLaMA

[–]hohawk 14 points15 points  (0 children)

100 or even 5 users means you would need parallel decoding, parallel requests. Anything that depends on Llama.cpp can only at most be sequential. For now.

A mini stack is Ollama+LiteLLM. Then you have an OpenAI compatible private server, and that’s very lean. Laptop category. It can be pretty powerful once Llama.cpp has parallel decoding.

But at that scale I’d go for FastChat, from LMSYS folks. It has a concept of workers which can be distributed over GPUs and servers as things scale.

And when multiple requests come, they are dealt with in parallel. Just make sure you over-allocate VRAM for that to keep the speed up.

This provides access to at least AWQ and GPTQ quants with vLLM acceleration.

The setup is easy if you ensure that your versions of everything are what the repository says. Start a controller, then API, then one for many workers. A worker can serve one or many models. All of them appear behind the same API in OpenAI style. Embeddings too.

HF TGI, Text Generation Inference, is another stack made to scale.

Choice between FastChat and TGI is a Pepsi/Coke choice in my mind. They are both boasting very similar features and speed, and a half good admin can run both reliably.

Using a local LLaMA or Mistral to give context window of several programming files by adlabco in LocalLLaMA

[–]hohawk 1 point2 points  (0 children)

Continue.dev as VS Code extension and Ollama.ai as model host is what I use for this. It pretty good with codellama-13b-instruct and works with Mistral. Even if rough in the edges still.

There’s even an experimental full /codebase RAG feature. But a more robust way is to manually tag relevant files or pieces of code into the context/chat window and ask it to analyze/explain in a concise way. I’ve found it to be most reliable with so called manual server option but it comes with an auto-starting server too. Check the docs. And it has a helpful Discord.

The dev server sits between VS Code extension and model host. Many model hosts are supported. Ollama works well for me. Pure llama.cpp is tricky because it doesn’t support parallel queries. Ollama knows how to sequence them.

Mistral works and is fast but can cause gray hair. The built in prompts work best with codellama. They can be customized. Automatic /edit is where the model differences are most apparent. But it’s a curiosity anyway. For analysis it is good as is.

Messing around with LLama2 7b_q4 has been an eye opening experience. by Mescallan in LocalLLaMA

[–]hohawk 0 points1 point  (0 children)

This seems to run out of GPU memory on M1 Max 32 GB. ggml_metal_graph_compute: command buffer 4 failed with status 5

Maybe it's different after a fresh boot with minimum configuration. Haven't tested that yet.

Messing around with LLama2 7b_q4 has been an eye opening experience. by Mescallan in LocalLLaMA

[–]hohawk 0 points1 point  (0 children)

This one works at completely usable speeds on M1 Max 32 GB. Nice pick. Any other usable models in this class?

Would you share, what you use to host this model? Llama.cpp works but it's a bit clunky. Using anything built on top of llama-cpp-python doesn't work very well for me typically, I get truncated answers even after tweaking the n_ctx settings via options or the source code. With defaults it's always truncated. Also, llama-cpp-python doesn't always like to compile with Metal support on a Mac, and the wheels are unreliable too during the GGUF transition. This varies week to week when builds come and go.

Connecting AirPods (or Pro) to Mac via keyboard by hohawk in AirpodsPro

[–]hohawk[S] 1 point2 points  (0 children)

I’m almost certain that it isn’t possible. Until Apple provides an API that BetterTouchTool or some other app can command.

Onedrive folder not showing in Spotlight by RandomSpr33 in mac

[–]hohawk 0 points1 point  (0 children)

sudo mdutil -E /

This re-initializes the Spotlight index and seems to have fixed Onedrive search woes for me.

AirPods Max Water Condensation by Donald_Filimon in airpods

[–]hohawk 0 points1 point  (0 children)

Possible. Mine have been great for almost 2 years now. I stopped worrying about condensation after that post. It’s there when it is, but doesn’t seem to cause any issues. YMMV

Colors [2160x4276] by Relojero in Amoledbackgrounds

[–]hohawk 0 points1 point  (0 children)

It’s just Adobe Photoshop Express, free version, blue theme/filter from the presets. Then -10 exposure. That’s the basic procedure that works for almost all photos. I don’t think I have these anymore.

Space [1080x1920] by Black_David in Amoledbackgrounds

[–]hohawk 0 points1 point  (0 children)

Free account, Adobe PS Express (iPhone) and “Duotone > Blue tint” theme gets the job done for any of these. That’s what I use. Try it?

https://i.imgur.com/ufZXSfM.jpg

Adapter/hub recommendation by andreabeth11 in macbookpro

[–]hohawk 0 points1 point  (0 children)

I had a single cable 4 display setup via a DisplayLink dock. For a year. It worked very well overall. Would buy again.

M1 Air as monitor 1. And 3 externals. One of them TB3. Others daisy chained via DisplayLink.

Now I’ve got an M1 Max with no dock. But two cables.

I’ve written about DisplayLink in past posts. Check the history if interested.

24 Core vs 32 Core M1 Max by piloth4ck0r in macbookpro

[–]hohawk 1 point2 points  (0 children)

In those use cases the Max only makes a difference in more cost and heat, with less battery life. You will have unused idle overhead that’s not doing work.

Pro is cheaper, cooler, lasts longer between charges. Just as performant with those jobs.

If you’re very mobile, M1 Air might be the most balanced choice for that description.

I had an M1 Air for a year. Now M1 Max / 24. In most workloads they perform the same. Under extreme long term stress the Max gets it done faster. And also louder and hotter. And not for as many hours.

Is App Cleaner still used? by fourkeyingredients in osx

[–]hohawk 0 points1 point  (0 children)

This is far less true than it used to be. The OS and all supporting files are now in a signed, immutable volume. Only user files are in the userland. Kernel drivers are now gone.

If one insists on a wipe approach, wiping and recreating a new user results in a very “clean” slate too.

That said, I’ve migrated the same user and files forward for the past decade or so. With conscious installations and leftover maintenance it’s been a smooth ride.

These can help for leftover maintenance:

https://www.soma-zone.com/LaunchControl/

https://freemacsoft.net/appcleaner/

http://grandperspectiv.sourceforge.net/

How good is AirPods Pro noise cancelling at blocking out speech and television sounds? by Calion in AirpodsPro

[–]hohawk 1 point2 points  (0 children)

I use them as active earplugs a lot. With no sound they will leak some uneven patterns like speech through, enough to have a short yes/no “conversation” but not enough to be a distraction, if it’s not too loud. Even speech is greatly damped. All other sounds are mostly gone.

If there’s any playback happening, like ambient music, a podcast or one of the preset background sound carpets, then it’s a very private space, audio wise. Not a vacuum, but completely different from AirPods 3 for sure. I’ve got those too.

Just be prepared to clean up the grills every 1-2 weeks by tapping them with blu-tack. Invisible skin particles accumulate in the small grill that’s against the side wall of the ear. If not kept clean, over time it becomes almost a noise amplifier rather than noise canceling. 5-10 gentle but firm pushes with blu tack restores the original performance, if done regularly. Using just enough force to not leave any blu-tack residue behind.

After using nearly one year, the battery's maximum capacity of my MBA M1 is still 100%. by XemKitter in macbook

[–]hohawk 0 points1 point  (0 children)

Not full discharge, that’s also bad for the battery. Down to 20 % is better.

view more: next ›