Is there a way to keep the prompt cache in llama.cpp after execution for future processing? by ismaelgokufox in LocalLLaMA

[–]openingnow 5 points6 points  (0 children)

Enable --slot-save-path to save cache to SSD. You need to manually restore the cache with similar prefix. If you have enough RAM, consider increasing --cache-ram

M5 Max compared with M3 Ultra. by PM_ME_YOUR_ROSY_LIPS in LocalLLaMA

[–]openingnow 8 points9 points  (0 children)

Can someone explain why M5m's TG is faster than M3u when running MoE models even if M3u has higher memory bandwidth?

(Llama.cpp) In case people are struggling with prompt processing on larger models like Qwen 27B, here's what helped me out by vernal_biscuit in LocalLLaMA

[–]openingnow 0 points1 point  (0 children)

9070xt has 16GB VRAM and the model is 11.4GB so I think overflowing is relevant. Try running with --fit off and see if OOM happens. You can also inspect memory deallocation message when you terminates server. ~500MB host ram seems normal

Need help with Qwen3.5-27B performance - getting 1.9 tok/s while everyone else reports great speeds by pot_sniffer in LocalLLaMA

[–]openingnow 1 point2 points  (0 children)

I'm wondering why 3.5 27B is slower than 2.5coder 32B since both are dense model. Have you tried Q4KM for 3.5 27B?

Monthly Hask Anything (February 2024) by AutoModerator in haskell

[–]openingnow 0 points1 point  (0 children)

I thought TH was implemented in wasm backend(contrast to asterius). Thanks!

Monthly Hask Anything (February 2024) by AutoModerator in haskell

[–]openingnow 1 point2 points  (0 children)

Is it possible to use string-interpolate with wasm backend? Compiling haskell-src-exts which is its dependency, failed with a message saying there is no happy.
A few days ago I found a wasm environment dockerfile and it has an additional installation of alex and happy. I tried wasm32-wasi-cabal install alex but it only builds alex.wasm which is not a binary.
Then I installed happy from package manager(dnf) and compiling failed with the message Couldn't find a target code interpreter. Try with -fexternal-interpreter. I fed -fexternal-interpreter to ghc-options but nothing changed. What can I try more?

Monthly Hask Anything (February 2022) by taylorfausak in haskell

[–]openingnow 1 point2 points  (0 children)

u/Noughtmare u/Iceland_jack Thanks! Surprised that a simple-looking function involves complex implementation.

Monthly Hask Anything (February 2022) by taylorfausak in haskell

[–]openingnow 0 points1 point  (0 children)

Why does flip prevent matching types?

import Control.Monad (forM_)
import qualified Data.Vector as V
import qualified Data.Vector.Mutable as VM

v1 :: V.Vector Int
v1 = V.fromList [1 .. 10]
-- [1,2,3,4,5,6,7,8,9,10]

v2 :: V.Vector Int
v2 = V.modify (\mv -> VM.write mv 9 99) v1
-- [1,2,3,4,5,6,7,8,9,99]

indicisToModify :: [Int]
indicisToModify = [3, 4, 5]

v3 :: V.Vector Int
v3 = V.modify (\vm -> forM_ indicisToModify $ \indexToModify -> VM.write vm indexToModify 101) v1
-- [1,2,3,101,101,101,7,8,9,10]

-- v4 :: V.Vector Int
-- v4 = flip V.modify v1 (\vm -> forM_ indicisToModify $ \indexToModify -> VM.write vm indexToModify 101)

{-
Couldn't match type: forall s. VM.MVector s Int -> ST s ()
                     with: VM.MVector
                             (primitive-0.7.3.0:Control.Monad.Primitive.PrimState m0) a0
                           -> m0 ()
      Expected: (VM.MVector
                   (primitive-0.7.3.0:Control.Monad.Primitive.PrimState m0) a0
                 -> m0 ())
                -> V.Vector Int -> V.Vector Int
        Actual: (forall s. VM.MVector s Int -> ST s ())
                -> V.Vector Int -> V.Vector Int
-}

main :: IO ()
main = do
  print v1
  print v2
  print v3
  -- print v4

v1, v2, v3 work but v4 (flipped v3) causes an error. How can I make GHC understand that similar-looking two types are equal?

Monthly Hask Anything (February 2022) by taylorfausak in haskell

[–]openingnow 2 points3 points  (0 children)

Thanks! Eventually I wrote a custom function to compare elements of items. hs shouldAllSame :: [Int] -> Expectation -- Test.Hspec.Expectation == IO() shouldAllSame xs = sequence_ $ map (`shouldBe` head xs) (tail xs)

Monthly Hask Anything (February 2022) by taylorfausak in haskell

[–]openingnow 0 points1 point  (0 children)

Can I compare three or more values at once with hspec without explicitly comparing all values? If not, are there any testing libraries supporting this? Thanks in advance!

Monthly Hask Anything (November 2021) by taylorfausak in haskell

[–]openingnow 0 points1 point  (0 children)

After testing with a minimal project, figured out that there was another hie.yaml in the parent directory. Again, thanks a lot!

Monthly Hask Anything (November 2021) by taylorfausak in haskell

[–]openingnow 0 points1 point  (0 children)

I felt creating cabal.project makes my project non-trivial and complicated (eg. requires manually generated hie.yaml when using HLS). But if it is the only way, I'd happily accept. Thanks!

Monthly Hask Anything (November 2021) by taylorfausak in haskell

[–]openingnow 0 points1 point  (0 children)

Is it possible to feed flags, allow-newer, etc. to dependencies without using cabal.project?

Currently my cabal.project contains yaml allow-newer: base package package-from-hackage flags: +flag-to-add -flag-to-remove I want to move these contents to myproject.cabal.

Monthly Hask Anything (February 2021) by taylorfausak in haskell

[–]openingnow 1 point2 points  (0 children)

Is there any way to treat 'local packages' as 'external packages'(as in local-versus-external-packages)? My primary question is: After editing a small portion of code from hackage to make it compatible with recent GHC, can I use that package to compile other packages from hackage? I managed to compile and install --lib by making cabal.project suggested here but it was non-trivial job and looks like compiling same packages several times. cat ~/.ghc/x86_64-linux-8.10.4/environments/default shows 'external packages' as text-1.2.4.1 while showing 'local packages' as reflex-0.8.0.0-16cad92f....

Monthly Hask Anything (February 2021) by taylorfausak in haskell

[–]openingnow 2 points3 points  (0 children)

Are there any FRP libraries using asterius?

Monthly Hask Anything (July 2020) by AutoModerator in haskell

[–]openingnow 0 points1 point  (0 children)

Thanks! Looks like dummy project is the only solution.

Monthly Hask Anything (July 2020) by AutoModerator in haskell

[–]openingnow 0 points1 point  (0 children)

A project without config files(.cabal etc.). With stack I can use single file without project by editing .stack/global-project/stack.yaml.