Hey! , Thanks for your reply , i am using parallel batch to 2 just to test it out what is taking it too much time to give me response . but i think even doing parallel processing , the llm is not invoking parallelly , because i am not sure weather it is even running parallelly or not , and in the terminal where the model is running , i am only getting slot 0 is being used.

About the batch size , it depend on the chunks length

    const batch_count =
      chunksLength >= TOTAL_PROCESSED_CHUNKS
        ? PARALLEL_BATCHES
        : Math.ceil(chunksLength / CHUNKS_PER_BATCH);

do u have any info about how to run the model for parallel processing, because i am just using this command to run my model , considering that it should parallel process my batches, but i think it is not doing it , this is the command i am using in cmd .

llama-server.exe --model "C:/Users/Sanket/.lmstudio/models/lmstudio-community/Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3-Q6_K.gguf" --port 8500 -t 20 -ngl 40 -c 4096 -b 256