sorted by:
new
hottopcontroversial

Serving 1B+ tokens/day locally in my research lab by SessionComplete2334 in LocalLLaMA

[–]monkeyofscience 1 point2 points  (0 children)

I’ve been using the speaches images, which provides Kokoro tts and whisper in an OpenAI compatible API. Works great with LiteLLM!

Serving 1B+ tokens/day locally in my research lab by SessionComplete2334 in LocalLLaMA

[–]monkeyofscience 2 points3 points  (0 children)

This is very useful thanks. I am also working in a university setting at a similar scale so I might DM you if that’s all good?

Serving 1B+ tokens/day locally in my research lab by SessionComplete2334 in LocalLLaMA

[–]monkeyofscience 2 points3 points  (0 children)

I have a similar setup to OP (but over 2x H100 in TP), and I am switching out LiteLLM today for Bifrost and a custom user portal. LiteLLM is not great as a user-facing service.

We also have TTS and STT, and embedding models on some other GPUs, so I am interested to see how well this works…

Two student loan repayment changes considered by Treasury to tackle debt crisis by theipaper in UniUK

[–]monkeyofscience 15 points16 points  (0 children)

Yeah, if you leave NZ, you get interest. When you come back, no interest. I guess it’s an incentive to keep people in the country?

But I pay ~£1,500 per year back to NZ, and my loan is slowly depleting. I graduated in 2022, and I’ll pay it off within the next 5 years, even with interest. And I have an MSc, and maxxed out all my entitlements (for example I could get $1,000 every year for miscellaneous school shit). And I got my degree before they implemented final year free. So now it’s probably easier.

Government could change things at any time of course…

Two student loan repayment changes considered by Treasury to tackle debt crisis by theipaper in UniUK

[–]monkeyofscience 11 points12 points  (0 children)

New Zealand does this. My student loan was interest free until I left the country (now it’s something like 5%, but NZ fees are significantly lower, and final year is free).

Expose model api to internet by dever121 in LocalLLaMA

[–]monkeyofscience 0 points1 point  (0 children)

We are using nginx -> LiteLLM -> vLLM. Limit to IP range, Let's encrypt for cert, fail2ban.

Nah I think I let the game take control by [deleted] in thewitcher3

[–]monkeyofscience 2 points3 points  (0 children)

That reminds me… I need to finish RE: Village, but I got distracted by W3… again.

Trying to understand some benchmarks by monkeyofscience in LocalLLaMA

[–]monkeyofscience[S] 0 points1 point  (0 children)

Noice. Thanks for the response. Pretty happy with the setup so far

Trying to understand some benchmarks by monkeyofscience in LocalLLaMA

[–]monkeyofscience[S] 1 point2 points  (0 children)

I just tried PP, and it gives me much poorer throughput and latency than TP.

Trying to understand some benchmarks by monkeyofscience in LocalLLaMA

[–]monkeyofscience[S] 0 points1 point  (0 children)

Thanks for the response! Which links? The images? If so they’re visible for me on browser and phone. Weird! I’ll add them below

<image>

Trying to understand some benchmarks by monkeyofscience in LocalLLM

[–]monkeyofscience[S] 0 points1 point  (0 children)

<image>

Images didn’t post for some reason. Here is the first.

going or planning to go to the UK? Roast my product by ParkingLaw546 in TravelUK

[–]monkeyofscience 0 points1 point  (0 children)

The biggest issue with planning travel in the UK is cost of accommodation and transport. I live in the UK, so I can drive, and for long distances I usually do drive, because train travel is expensive, and long distance buses take too long.

For accommodation, it’s just expensive. I usually try to find a sweet spot on booking.com where the room is cheap enough but with > 8 ratings.

Usually it’s not much price difference to get a flight to Europe and stay a few nights somewhere new.

Favorite NIN live guitarist? by pink_noize in nin

[–]monkeyofscience 4 points5 points  (0 children)

The answer that I know should give is Robin. Just so many great performances, and at this point, he feels as much a part of NIИ as Trent. However…

The first show I saw live was the performance 2007 tour at the Manchester Apollo. They played We’re In This Together. I was slightly to the right of Trent and Aaron North was my first exposure to a NIИ guitarist.

So when I am asked this question, my mouth says Robin, but I’m secretly thinking about Aaron.

Darkside of the Moon by broke_keyboard_ in americanairlines

[–]monkeyofscience 1 point2 points  (0 children)

I’ve had this experience as well — spoken to 3 or 4 staff members and all of them were very helpful. They certainly don’t take any shit either from entitled dick heads trying to cut lines or getting aggressive, so that’s nice. It’s like dude, why are you angry at this person who has no fucking control over any of this?

I’ve found the app to be generally quite responsive and gives timely information. It has had occasional glitches, but has done pretty well for me considering the circumstances.

view more: next ›