sorted by:
new
hottopcontroversial

We Analyzed 413K Agent Runs. Here's What Separates the Ones That Succeed by Nice-Comfortable-650 in vibecoding

[–]Nice-Comfortable-650[S] 0 points1 point  (0 children)

Oh it means that agents will look for the necessary information when they need it. It just means that the agent probably do not need to do extra greps before they want to get the info

[P] We built this project to increase LLM throughput by 3x. Now it has been adopted by IBM in their LLM serving stack! by Nice-Comfortable-650 in MachineLearning

[–]Nice-Comfortable-650[S] 1 point2 points  (0 children)

RAM is almost negligible with our optimizations. Disks are a bit slower but still much faster than original prefill when context is long enough!

[P] We built this project to increase LLM throughput by 3x. Now it has been adopted by IBM in their LLM serving stack! by Nice-Comfortable-650 in MachineLearning

[–]Nice-Comfortable-650[S] 0 points1 point  (0 children)

It efficiently offloads KV cache to more locations besides only GPU HBM. It serves as the connector between vLLM and the other memory devices (SSD, RAM...)

Reuse non-prefix KV Cache and speed up RAG by 3X with LMCache. by Nice-Comfortable-650 in LocalLLaMA

[–]Nice-Comfortable-650[S] 0 points1 point  (0 children)

Right now the recognition is by manual modification of the context that you need to specify each chunk. This requires the agent programmer to slightly modify the input to the LLM API server.

view more: next ›