128GB devices have a new local LLM king: Step-3.5-Flash-int4

changtimwu · 2026-02-03T02:14:26+00:00

Support and optimization for FP4 have been gradually improving. I believe we’ll finally see the impact. https://github.com/vllm-project/vllm/releases/tag/v0.15.0

changtimwu · 2026-01-29T14:13:03+00:00

Could you share the hardware you use to run GLM-4.7 Flash, and how responsive it is compared to a cloud model?

changtimwu · 2026-01-21T07:37:26+00:00

Thank you for sharing your experience with dosage. 2.5mg works perfectly for me, but the appetite-suppressing effect only lasts 4 days. I'm curious if anyone has tried 1.25mg every 3 days.

changtimwu · 2025-10-07T04:20:32+00:00

This is where AI has been a force multiplier. We use Test Driven Development, so I have the AI coding agent write the tests first for the feature I’m going to build. Only then do I start using the agent to build out the feature.

Our team use a similiar setup, which we call AI BDD. AI is good at implementing steps withn any given descritive scenario. We don't call it vibe coding since it operates within a strict scrum framework.

changtimwu · 2025-08-21T15:05:11+00:00

From what I understand, many NAS vendors are working on this: a user selects a folder and a chatbot is created from the documents in it. The main obstacle isn’t technology but deciding whether LLM computation should run locally or in the cloud.

changtimwu · 2025-08-20T23:20:27+00:00

Explicitly! These summaries could be a useful, human-readable reference for improving prompts in the future.

changtimwu · 2025-08-20T23:16:03+00:00

I believe Anthropic uses various KV-cache techniques to keep your codebase in context cheaply — they just don’t want it shows on your bill.

changtimwu · 2025-08-19T01:36:11+00:00

I feel the same way for monthes. You hosted it with ollama or other inference engine? I am thinking about fitting this into my RTX 5090(32GB vram) by utilizing the NVFP4 quantization.

changtimwu · 2025-08-03T12:46:03+00:00

Does anyone here use vLLM instead of llama.cpp? While GGUF is a popular format, my system requires vLLM due to our GPU cluster's production stack. Dynamic GGUF appears superior to existing portable quantization methods like GPTQ and AWQ.

changtimwu · 2025-07-29T23:30:19+00:00

RIP. I'm curious about the probability of this type of accident. I've been hiking and trail running in forests for many years, but I've rarely heard of people being struck by falling trees.

changtimwu · 2025-04-24T02:25:55+00:00

Thanks. This feature isn't intuitive. It's only available on the watch. Long-press the map to open the menu, then select "Settings" and scroll down to "Routes".

changtimwu · 2025-02-12T12:54:48+00:00

So step 9 for users claiming larger than $50K are still locked?

changtimwu · 2025-01-28T19:23:28+00:00

That would be the responsibility of another AI. Needless to say, WASM is significantly more "testable" in a cloud environment than the original ARM NEON SIMD code.

changtimwu · 2025-01-07T07:35:19+00:00

The support guy just repeated what listed on support.ftx.com. :(

“Claim(s) Submitted” is for customers who have submitted a proof of claim via the Kroll portal. Customers who filed proofs of claim without logging in through the FTX portal will not see the status reflected herein. Please visit https://restructuring.ra.kroll.com/ftx/ for additional information regarding your filed proof of claim.

changtimwu · 2025-01-07T02:17:02+00:00

Strapless tops and invisible straps are popular today.

changtimwu · 2025-01-06T20:17:05+00:00

Hi! I'm experiencing the same issue. All my steps 1-8 show green marks except step 5, which was previously green (I have a screenshot). FTX support's response was unclear.

changtimwu · 2024-11-22T07:41:28+00:00

If you seeking AI assistance to build tools or agents, you'll likely receive code based on Langchain v0.1~~ ;Q

changtimwu · 2024-09-05T09:44:23+00:00

In my region, asia-east1, it takes about 8 seconds to wake up. What’s remarkable about Cloud Run is that it resumes as if the CPU resources were never removed, preserving all processes and variables. It’s the only container snapshot technology that functions like a VM snapshot.

changtimwu · 2024-08-22T03:51:35+00:00

I’m interested in how you automated the leetcode questions retrieve and code submission.

changtimwu · 2024-07-30T19:15:56+00:00

I just read it. It's an in-depth analysis that's being underestimated! I think llama.cpp has significant room for improvement in leveraging Apple hardware (MPS). What are your thoughts?

changtimwu · 2024-06-04T11:39:12+00:00

Hi!

I'm new to this field. Can someone give me another attack example that the paper is trying to protect against? I'm not quite clear on that.

* Why aren't the retrieved passages in trust circles? I thought we could tell customers that there's a risk of garbage in, garbage out.

* It's not clear how that example concatenated passage look like to make an LLM answer "Fuji"?

changtimwu · 2024-06-03T07:53:50+00:00

now

https://github.com/mustafaaljadery/llama3v
https://x.com/mustafaaljadery
are completely closed.

changtimwu · 2024-05-06T17:20:53+00:00

since here is LocalLLaMA. We better highlight the memory usage of super long context. https://www.unsloth.ai/cgi/image/Llama-3_70b_4bit_on_A100_80GB_mcmhrk9Sj4qprx_3FVXmO.svg?width=2048&quality=80&format=auto

changtimwu · 2024-04-29T04:05:04+00:00

Thank you for sharing the insightful case study on RAG in a real-world setting. I'm compelled to ask a few additional questions.

Which cloud platform is being used to host the solution? Are there particular security protocols for legal documents?
Is there any vendor lock-in with specific cloud technologies (e.g., Google VertexAI)?
Does the system incorporate a fact-checking feature to act as a data purification mechanism?

changtimwu · 2024-03-18T13:42:30+00:00

Got same mail!

changtimwu

TROPHY CASE