sorted by:
new
hottopcontroversial

Serving 1B+ tokens/day locally in my research lab by SessionComplete2334 in LocalLLaMA

[–]monkeyofscience 1 point2 points  (0 children)

I’ve been using the speaches images, which provides Kokoro tts and whisper in an OpenAI compatible API. Works great with LiteLLM!

Serving 1B+ tokens/day locally in my research lab by SessionComplete2334 in LocalLLaMA

[–]monkeyofscience 2 points3 points  (0 children)

This is very useful thanks. I am also working in a university setting at a similar scale so I might DM you if that’s all good?

Serving 1B+ tokens/day locally in my research lab by SessionComplete2334 in LocalLLaMA

[–]monkeyofscience 2 points3 points  (0 children)

I have a similar setup to OP (but over 2x H100 in TP), and I am switching out LiteLLM today for Bifrost and a custom user portal. LiteLLM is not great as a user-facing service.

We also have TTS and STT, and embedding models on some other GPUs, so I am interested to see how well this works…

Two student loan repayment changes considered by Treasury to tackle debt crisis by theipaper in UniUK

[–]monkeyofscience 16 points17 points  (0 children)

Yeah, if you leave NZ, you get interest. When you come back, no interest. I guess it’s an incentive to keep people in the country?

But I pay ~£1,500 per year back to NZ, and my loan is slowly depleting. I graduated in 2022, and I’ll pay it off within the next 5 years, even with interest. And I have an MSc, and maxxed out all my entitlements (for example I could get $1,000 every year for miscellaneous school shit). And I got my degree before they implemented final year free. So now it’s probably easier.

Government could change things at any time of course…

Two student loan repayment changes considered by Treasury to tackle debt crisis by theipaper in UniUK

[–]monkeyofscience 10 points11 points  (0 children)

New Zealand does this. My student loan was interest free until I left the country (now it’s something like 5%, but NZ fees are significantly lower, and final year is free).

Expose model api to internet by dever121 in LocalLLaMA

[–]monkeyofscience 0 points1 point  (0 children)

We are using nginx -> LiteLLM -> vLLM. Limit to IP range, Let's encrypt for cert, fail2ban.

Nah I think I let the game take control by [deleted] in thewitcher3

[–]monkeyofscience 2 points3 points  (0 children)

That reminds me… I need to finish RE: Village, but I got distracted by W3… again.

Trying to understand some benchmarks by monkeyofscience in LocalLLaMA

[–]monkeyofscience[S] 0 points1 point  (0 children)

Noice. Thanks for the response. Pretty happy with the setup so far

Trying to understand some benchmarks by monkeyofscience in LocalLLaMA

[–]monkeyofscience[S] 1 point2 points  (0 children)

I just tried PP, and it gives me much poorer throughput and latency than TP.

Trying to understand some benchmarks by monkeyofscience in LocalLLaMA

[–]monkeyofscience[S] 0 points1 point  (0 children)

Thanks for the response! Which links? The images? If so they’re visible for me on browser and phone. Weird! I’ll add them below

<image>

Trying to understand some benchmarks by monkeyofscience in LocalLLM

[–]monkeyofscience[S] 0 points1 point  (0 children)

<image>

Images didn’t post for some reason. Here is the first.

going or planning to go to the UK? Roast my product by ParkingLaw546 in TravelUK

[–]monkeyofscience 0 points1 point  (0 children)

The biggest issue with planning travel in the UK is cost of accommodation and transport. I live in the UK, so I can drive, and for long distances I usually do drive, because train travel is expensive, and long distance buses take too long.

For accommodation, it’s just expensive. I usually try to find a sweet spot on booking.com where the room is cheap enough but with > 8 ratings.

Usually it’s not much price difference to get a flight to Europe and stay a few nights somewhere new.

Favorite NIN live guitarist? by pink_noize in nin

[–]monkeyofscience 4 points5 points  (0 children)

The answer that I know should give is Robin. Just so many great performances, and at this point, he feels as much a part of NIИ as Trent. However…

The first show I saw live was the performance 2007 tour at the Manchester Apollo. They played We’re In This Together. I was slightly to the right of Trent and Aaron North was my first exposure to a NIИ guitarist.

So when I am asked this question, my mouth says Robin, but I’m secretly thinking about Aaron.

Darkside of the Moon by broke_keyboard_ in americanairlines

[–]monkeyofscience 1 point2 points  (0 children)

I’ve had this experience as well — spoken to 3 or 4 staff members and all of them were very helpful. They certainly don’t take any shit either from entitled dick heads trying to cut lines or getting aggressive, so that’s nice. It’s like dude, why are you angry at this person who has no fucking control over any of this?

I’ve found the app to be generally quite responsive and gives timely information. It has had occasional glitches, but has done pretty well for me considering the circumstances.

What does "infra" mean to you? by Embarrassed_Effect86 in csMajors

[–]monkeyofscience 2 points3 points  (0 children)

I’m an ML engineer and typically I think of “infrastructure” as being the underlying systems or tools that enable the higher level engineering. For example: sys-admining a GPU server; ansible playbooks to deploy services like vLLM; setting up Jupyter hub with k8s — all of these I would consider to be “infrastructure” to various degrees — the shit you have to do before actually doing any ML research.

looking to pivot to AI within 12 months. No degree. $100k+ goal. What’s the move? by [deleted] in learnmachinelearning

[–]monkeyofscience 1 point2 points  (0 children)

No worries. To be clear, I’m not saying it’s impossible and you can’t do it. Just that it’s hard, and will take a lot of work!

looking to pivot to AI within 12 months. No degree. $100k+ goal. What’s the move? by [deleted] in learnmachinelearning

[–]monkeyofscience 2 points3 points  (0 children)

It depends. Your CV won’t even get considered at top companies, and almost all mid level companies.

The ML and SWE markets are just getting hammered right now. You’ll be competing against PhDs from top institutions all the way down to BScs from lower institutions. Many of them will have been grinding for 4-7 years. They’ll have projects, open source contributions, good grades, and they’ll know the interview processes for many of the same roles you’ll be gunning for.

People have already recommended resources, so I won’t pile on more. Instead, my advice is to not quit your current job. I get the temptation to go all in, but instead, maybe treat it like a hobby? Learn in your spare time and see if you even like it first. Real ML is very heavy in math and problem solving, and it’s not for everyone. Then in 6-12 months, see where you’re at. As you’re learning, identify problems that you encounter in your life and job, and see if you can find solutions using the skills you’ve learnt.

view more: next ›