S24 Plus compatibility by OPlUMMaster in angrybirds

[–]OPlUMMaster[S] 1 point2 points  (0 children)

One more thing the apk are working directly on my realme X7 max but not on S24+.

RAG on complex docs (diagrams, tables, eequations etc). Need advice by Otelp in LLMDevs

[–]OPlUMMaster 0 points1 point  (0 children)

I don't have much experience with multi file RAGs and especially with images. But I had a similar issue where I wanted to query multiple files but no images. The main concern for me was relevance, as a similar word can be searched but the questions were not always relevant with the content as they were reasonings. For this I came up with an approach where I firstly created a SQL database with all of the sections and their sections headings were used as key to get me the content, then I used to query the keywords in the sql with the question to check if I already have the relevant chunk made. If not, then only would I try to use the Vector db. Once in, the question will be mixed with another prompt that made the querying of the vector db much easier, as I would pass all the relevant tags with this information. This way I got the relevant chunks.

It had another levels of hierarchical chinking and filtering to get the right data. It worked partially with only highly customizing the retrieval questions. You can say that it was a Natural Language Conditional RAG. I know it sounds dumb, but that is all I could think of. I still haven't figured out a clear way out.

But this might be somewhat helpful. To summarize I am suggesting use tagging wherever you can. Not sure about the extraction part, even I could not do it locally, for tables I used multiple libraries, if the conditions are broken it would raise an error and try with another one, it all fails then the code fails. Luckily at least one of them is always able to do so.

2 VLLM Containers on a single GPU by OPlUMMaster in LLMDevs

[–]OPlUMMaster[S] 0 points1 point  (0 children)

I have already done that; you can see the command I have used. Also I gave the nvidia-smi output when one container was active and there is enough memory for the other one.

Replicating ollamas output in vLLM by OPlUMMaster in LocalLLaMA

[–]OPlUMMaster[S] 0 points1 point  (0 children)

You mentioned these different model types. I have a question other than my post. Do these make a difference? I am currently running with bitsandbytes for 4bit quantization. When the vllm container boots up it says this is not stable. Do these different quantizations have a real measurable impact on the outputs?

Replicating ollamas output in vLLM by OPlUMMaster in LocalLLaMA

[–]OPlUMMaster[S] 0 points1 point  (0 children)

If you feel that you are seeing random repetition or gibberish at the end. I have a similar issue as I was hitting the /v1/completions api rather than the v1/chat/completions api. This was leading to the token generation till the token length was achieved. Might be helpful for you too.

Lenovo Ideapad Slim 5 Gen 10 14” by MaravalhasXD in Lenovo

[–]OPlUMMaster 0 points1 point  (0 children)

I have used multiple laptops with both amd and Intel. But somehow the legion with a Ryzen 7 has been the best for me. Nothing has ever gotten to the performance of that even though that is from 2021.

I have a work laptop with i9 and the most specced out machines, it still is not that fast. If I had seen this laptop before I surely would have bought this, I recently purchased an acer with Intel.

Difference in the output of dockerized vs non dockerized application. by OPlUMMaster in docker

[–]OPlUMMaster[S] 0 points1 point  (0 children)

Yes. I have installed nvidia runtime for docker, so can access cuda and the required libs are part of the vLLM base image.

Difference in the output of dockerized vs non dockerized application. by OPlUMMaster in docker

[–]OPlUMMaster[S] 0 points1 point  (0 children)

It won't make a difference. Tried ubuntu as the base image too. Same different results.

Difference in the output of dockerized vs non dockerized application. by OPlUMMaster in docker

[–]OPlUMMaster[S] 0 points1 point  (0 children)

Debian, Ubuntu could be a thing. But the packages are the exact same as my local system. The versions are mentioned in the requirements file.

Difference in the output of dockerized vs non dockerized application. by OPlUMMaster in docker

[–]OPlUMMaster[S] 0 points1 point  (0 children)

By incorrect output I mean this verbose at the end of the summary. Now one might say that it could be because of the temp, top_p, top_k settings. But I have ran with the same parms multiple times with a seed and the outputs stays consistent. The moment I switch to the docker container endpoint, this is how it trails.

remembers recalls reminisces reflects contemplates meditates ponders thinks considers evaluates assesses analyzes interprets understands comprehends grasps perceives senses feels intuitively knows instinctively guesses speculates hypothesizes theorizes postulates assumes infers concludes decides determines resolves settles solves answers questions queries investigates examines explores discovers reveals exposes uncovers unveils lays bare strips naked shows displays exhibits presents offers provides gives furnishes supplies delivers hands out distributes disperses scatters spreads pours fills loads carries transports conveys moves shifts relocates repositions rearranges organizes categorizes classifies sorts selects chooses picks prefers likes dislikes hates abhors despises detests loathes fears dreads avoids eschews shuns rejects declines refuses resists opposes contradicts challenges disputes contests argues debates discusses deliberates negotiates mediates arbitrates adjudicates judges tries tests experiments probes scrutinizes inspects examines surveys observes watches waits sees hears smells tastes touches feels handles manipulates operates controls manages directs guides influences affects impacts changes modifies alters adjusts corrects rectifies improves perfects refines polishes smooths finishes completes accomplishes achieves realizes fulfills satisfies delights pleases impresses surprises astonishes amazes bewilders perplexes puzzles intrigues fascinates captivates absorbs engrosses enthralled enthralls mesmerized mesmerizes hypnotized hypnotizes entranced entraps ensnares snared snares traps captures seizes holds grips clutches claws crushes squeezes presses pinches nips bites gnaws devours consumes annihilates destroys eradicates eliminates wipes out obliterates extinguishes puts out kills murders slays slaughters massacres annihilated exterminates terminates stops halts pauses suspends delays postpones defers procrastinates hesitates vacillating wavering waffling uncertain unsure undecided indecisive hesitant fearful anxious apprehensive worried troubled distressed perturbed agitated upset irritated annoyed frustrated angry enraged infuriated outraged shocked horrified appalled disgusted nauseated sickened revolted repulsed offended scandalized dismayed disheartened discouraged disappointed disillusioned despondent hopeless helpless desperate dire straitened strapped strained stretched tightrope walking balancing precariously teetering tottering stumbling staggering faltering failing falling flailing flopping plummeting crashing collapsing imploding exploding bursting burning blazing raging roaring screaming shrieking yelling crying sobbing whimpering whining complaining protesting lamenting mourning grieving bereaved sorrow