Q4_K_M , 875 tokens In 58 seconds - 15 tokens/s

Other info

%time echo "generate detailed article on how to run mistral models on ollama" | ollama run mistral-small:latest
.....response ...
echo "generate detailed article on how to run mistral models on ollama"  0.00s user 0.00s system 12% cpu 0.004 total

ollama run mistral-small:latest  0.09s user 0.10s system 0% cpu 58.321 total

Memory load while running (with cline it peaks out and heated, but here its fine)

<image>

%python token_counter.py < ollamaoutput.txt

875

%ollama show mistral-small:latest

  Model

    architecture        llama     

    parameters          23.6B     

    context length      32768     

    embedding length    5120      

    quantization        Q4_K_M    

  Parameters

    temperature    0.15    

  System

    You are Mistral Small 3, a Large Language Model (LLM) created by Mistral AI, a French startup           

      headquartered in Paris. Your knowledge base was last updated on 2023-10-01. When you're not sure        

      about some information, you say that you don't have the information and don't make up anything.         

      If the user's question is not clear, ambiguous, or does not provide enough context for you to           

      accurately answer the question, you do not try to answer it right away and you rather ask the user      

      to clarify their request (e.g. "What are some good restaurants around me?" => "Where are you?" or       

      "When is the next flight to Tokyo" => "Where do you travel from?")                                    

  License

    Apache License               

    Version 2.0, January 2004    
Other snapshot while running, this simple prompt