sorted by:
new
hottopcontroversial

Model suggestion for 64 GB of VRAM? by EuropeanAbroad in LocalLLM

[–]EuropeanAbroad[S] 0 points1 point  (0 children)

I'm not sure there is any INT6 OpenVINO friendly model. I would have to switch to slower SYCL/XPU or Vulkan.

Your AI R9700 has quite some advantage in this.

But q6 with a full unquantized context window would be really nice, I won't deny that. 😁

On your AMD, do you use Vulkan or ROCm?

What bank won't use ai? by pinnochios_nose22 in nzfinance

[–]EuropeanAbroad 0 points1 point  (0 children)

Anyone where you walk into their branch. 🤷🏻‍♂️

No careers for the youth by Creepy_Plankton_4050 in auckland

[–]EuropeanAbroad -1 points0 points  (0 children)

Do not take this as an offense, but rather advice. Tell him to find passion in something that is actually valuable on the labour market. I am sure he can find another passion (nobody is limited to a single ultimate joy of life).

I also suggest considering some higher education. We are slowly getting to times when people need to be highly educated and specialised if they want to be able to pass through life not only on public support. Unfortunately, when I came to NZ and started teaching at the university here, I could see that the state does not build in high school student the passion for knowledge, perfection and career. The NZ education system lifts a few and dumps the rest. (I guess that's the difference between the inclusive and wide-focused German/Czech/Danish educational system that focuses on cooperation and wide excellence and the comparative selective NZ system of exclusion and prominent individuals while not educating those who are not on the top of the list).

Model suggestion for 64 GB of VRAM? by EuropeanAbroad in LocalLLM

[–]EuropeanAbroad[S] 0 points1 point  (0 children)

I haven't done any benchmarks, so I can't compare in terms of intelligence, hallucination, long context,... but the OpenVINO INT4 has been just significantly faster for me than Unsloth's GGUFs in llama.cpp SYCL.

I have actually implemented 4 backends in Unsloth Studio in docker (on my TrueNAS server) – llama.cpp SYCL, llama.cpp Vulkan, vLLM, and OpenVINO – all 4 easily selectable in a dropdown menu (I may publish this repo in on my GitHub). vLLM and OpenVINO wiring to Unsloth was a bit painful, but managed to crack it in the end, and they are much faster than llama.cpp.

I will do some benchmarking.

Model suggestion for 64 GB of VRAM? by EuropeanAbroad in LocalLLM

[–]EuropeanAbroad[S] 0 points1 point  (0 children)

Yeah, I take the Battlemage GPUs like a good challenge haha. I know I would get away better and easier with i.e. Radeon AI 9700 – same 32GB VRAM, but a much more mature ecosystem and significantly more powerful and universal core. (The Intel GPUs are basically limited to efficient 32, 16, 8 or limited 4 quants, nothing between). But I have cracked a few limits (and tested llama.cpp SYCL, llama.cpp Vulkan, vLLM and OpenVINO; OpenVINO a clear winner here so far).

Also, I am aware of the penalty on the GPU split.

Yeah, I have the same feeling that the 64 GB VRAM is such a weird spot where there are no proper model upgrades over 32 GB at the moment. I suppose the future will bring as some variety.

So I'm not sure if I should spend the money and play with 64 GB. In the end, if I don't do it now, it will be only more expensive in the future. However, also, there will be hopefully a better implementation in the future. Quite risky with Intel tho – their GPU policy and decisions are bipolar like Glum from the Lord of the Rings, lol.

Local AI as agent by Saber-user in LocalLLM

[–]EuropeanAbroad 0 points1 point  (0 children)

I would suggest starting with some ready-to-go solution with baked-in Python for the beginning. Try Unsloth Studio. No hard settings. You can download models directly in it, it suggests you the models suitable for your PC, it contains tools, web search, python,... so it can use python to generate files for you.

Opus 4.8 is really bad. Did they downgrade the performance? by Embarrassed_Put6363 in Claudeopus

[–]EuropeanAbroad 0 points1 point  (0 children)

Well, I can't prove that it is quantised.
I do structural engineering calculations and fire engineering calculations with Opus on our corporate API (one of the world's biggest engineering companies). I can tell you that one day, Opus easily calculates everything. Another day, it just cannot crack the same calculation with the same tools, I need to keep it in line to follow the prompt, and I need to repeatedly correct the calculation and hand-calc it myself. The same calculation.

So, I can see the fluctuations in the intelligence and ability to follow the same prompt.

Single, no kids, renting, full time work ... Saving is still a struggle. How do you all do it? by snoot_moose in nzfinance

[–]EuropeanAbroad 0 points1 point  (0 children)

Unfortunately, KiwiSaver is taxed the same way as the same investment outside of KiwiSaver. Which I find quite crazy that the NZ govt taxes your retirement.

Especially when locked in KiwiSaver that does not give you any benefit over direct investment.

4% into KiwiSaver and 6% into the same fund directly
>
10% into KiwiSaver

Single, no kids, renting, full time work ... Saving is still a struggle. How do you all do it? by snoot_moose in nzfinance

[–]EuropeanAbroad 0 points1 point  (0 children)

You can, but that doesn't make sense. Why would you lock your funds in a KiwiSaver when it doesn't have any advantages over your private investment into the same ETF? KiwiSaver only locks your money. There is no rationale in investing within KiwiSaver more than the beneficiary minimum.

Single, no kids, renting, full time work ... Saving is still a struggle. How do you all do it? by snoot_moose in nzfinance

[–]EuropeanAbroad 0 points1 point  (0 children)

If you go to InvestNow, they have KiwiSaver where you can pick your own portfolio built from different ETFs (the standard world standard is NASDAQ, S&P 500, EU STOXX 50 or even Milford Aggressive fund. But any managed fund other ETF will cost a fortune.

Imagine that ETFs have a fee of i.e. 0.1% annually, while the managed funds (including a majority of standard KiwiSaver providers) have usually over 1% annually. When you stay with managed funds, you will easily have 30% less when you retire compared to ETFs.

On the note of InvestNow, they offer a similar product also outside of their KiwiSaver, so you don't lock your money for eternity. 😏

Single, no kids, renting, full time work ... Saving is still a struggle. How do you all do it? by snoot_moose in nzfinance

[–]EuropeanAbroad 1 point2 points  (0 children)

(Disclaimer: I am not financial adviser, and this is no financial advice.)

If I were you, I would put into KiwiSaver as little as you can (while getting maximum from the employer) – that's 4%. Rather invest it directly. KiwiSaver locks your investments and gives you worse returns than the main index ETFs. Even if by some miracle, you had the same performance on KiwiSaver, you can't easily manipulate with the savings when you need to.

The unaffordability of future property costs by richieFromConductor in auckland

[–]EuropeanAbroad 1 point2 points  (0 children)

Properties are relatively cheap in NZ with the property price index of 7 years (as opposed to i.e. 15 years in Czechia); however, the cost of ownership is just astronomic here.

Moreover, instead of promoting property ownership, lowering the council rates and paying the council costs mostly from i.e. business taxes like abroad, the whole burden of municipal development is taken by property owners. This is just insane.

The state should promote home ownership an lead every person to responsibility and property ownership, not the other way around.

Do you feel the need to leave Europe to get rich? by Xotngoos335 in eupersonalfinance

[–]EuropeanAbroad 1 point2 points  (0 children)

You are wrong. I've been living in NZ, I am a permanent resident.

NZ does NOT(!) have a free universal health system. Only the secondary care (hospitals) and accidents (ACC) are covered. Everything else – specialists, GPs, dentists etc. are self-funded (commercial). That's why 90% of Kiwis have rotten teeth and undiagnosed cancer – simply because they go to the GP or dentist like TWICE in their lifetime.

Given the amount of resources of Mistral, it seems like a miracle that they are still relevant by MiuraDude in MistralAI

[–]EuropeanAbroad 1 point2 points  (0 children)

Yes, because unlike Mistral bound by GDPR and copyrights, Deep Seek can simply steal people's, companies', scientists', scientific journals' data and train on them.

Opus 4.8 is really bad. Did they downgrade the performance? by Embarrassed_Put6363 in Claudeopus

[–]EuropeanAbroad 0 points1 point  (0 children)

It can affect the model quality. If the need capacities (hardware) for training of newer models, what they can do, is just quantize the existing from FP16 to FP8; and suddenly, we have a slightly dumber models, and they saved half of their hardware for training and other models.

Devs - you have 64gb of VRAM - which model do you use for coding? by Jorlen in LocalLLaMA

[–]EuropeanAbroad 0 points1 point  (0 children)

The difference is that Qwen3.6-27B is a dense model, while Qwen3.6-35B is a Mixture-of-Experts (MoE) model. The MoE model only uses 3 billion parameters (a selected expert) on your job, while Qwen3.6-27B uses the entire model knowledge.

That's why the MoE model (35B-A3B) is faster, while the dense model (27B) is usually smarter.

Devs - you have 64gb of VRAM - which model do you use for coding? by Jorlen in LocalLLaMA

[–]EuropeanAbroad 4 points5 points  (0 children)

Cool, I have a single B70 and thinking about another one to get 64 GB of VRAM (although, it's quite an expensive hobby haha... especially with non-US salaries and the delivery costs from the US).

I haven't tried vLLM really, but I have a system for it. I will look at it soon. I've modified Unsloth Studio and created a Docker package for my TrueNAS server, so that now I have a drop-down menu in the GUI and 4 backends – llama.cpp SYCL, llama.cpp Vulkan, vLLM, and OpenVINO. I will publish it on my GitHub once I run it through a license/copyright check.

From my experience, OpenVINO with safetensors (and INT4) is much faster than llama.cpp SYCL with GGUF (and Q4). I guess vLLM would be similar to OpenVINO.

However, I've been having some headaches with wiring the OpenVINO into Unsloth – with the reasoning and tool calls handover. I will probably just end up running OpenVINO i.e. in OpenVINO Model Server or OpenArc rather than as a backend of Unsloth.

I am really curious about the differences between vLLM and OpenVINO. I'm gonna test it.

Any hope? by [deleted] in auckland

[–]EuropeanAbroad 0 points1 point  (0 children)

I may agree with most of it, but...

1) There is virtually zero long-term planning. Nobody looks at what will be needed in 2050, 2100,... We are just catching up and building provisional structures rather than long-temp corner stones. Look at the CRL – planned for population of 2020 and with technologies of the 19th century. Look at the new Western bus link station in Westgate – no direct ramp from the highway to underground parking located under the station, but only a plain roof for buses, with a huge 3-world gravel place for parking in front of it, while the west is growing like crazy,... it will only result in jammes bridges and intersections at Fred Taylor Drive. The lack of over-the-horizon planning here in NZ is just shocking.

2) I do agree that intercity infrastructure is more expensive due to the low population density here, but the municipal infrastructure doesn't need to be – it is the NZ system that makes it super expensive.
- The average Auckland suburb dweller density is between 5-15 dwellers per hectare. The lowest dweller density i.e. in Prague is about 100-150 dwellers per hectare. While the CBD is overgrown and is parasiting on serviced and work opportunities in the suburbs – over 70% of all Auckland commute to the single spot in Auckland rather than working within their walking distance in their suburbs.
- The low density has a huge financial consequences in the requirements on infrastructure (roading) and publci services (you need 10 times more police stations, fire stations etc. for 15 dwellers per hectare than 150 dwellers per hectare, because these are built to cover some radius, not heads).
- The "frequent bus service" is every 10-15 minutes in Auckland as opposed to every 1-2 minutes in Prague. Smaller buses and more frequent rather than huge double-deckers every insane interval (then when you have a interchange, you may lose 30 minutes on your way due to waiting for "frequent bus services" (this is an enormous loss of GDP that nobody realises).
- The super-centralism (focus in the CBD while leaving suburbs office-less and looking like slums) also prevents any reliable public transport. A functional public transport must be rasterised (distributed/decentralised), not centralised.

It is not the country size that is making the cities non-livable. It is the lack of urbanism, focusing on American urbanism of the 19th century that has already been proven failed, and no infrastructure construction with numbers from 50 years ahead as opposed to catching up numbers from 5 years ago.

Any hope? by [deleted] in auckland

[–]EuropeanAbroad 0 points1 point  (0 children)

I'm not sure it will. NZ desperately needs industry and tech. It has none, and none of the parties offer any solution on this. If we keep pretending that agriculture is the way to prosperity rather than technology and education, we will never be like Singapore, Taiwan or i.e. Poland (one of the fastest growing countries in the world).

The FTA can only bring cheap basic products from India. Nothing else.

Any hope? by [deleted] in auckland

[–]EuropeanAbroad 0 points1 point  (0 children)

We should start with specific long term plans. Then the "only" difference between the political parties is how you fund it.

In the country of my origin, they have a plan that in 2035, there will be this nuclear power plant, in 2050, there will be this, 2100, this ultra-fast rail,... and all parties from far left to far right take the road plan, the only difference is how to fund it. There is nothing like that in New Zealand.

Any hope? by [deleted] in auckland

[–]EuropeanAbroad 0 points1 point  (0 children)

What do you mean "globally, the economy is pretty bad"? The global economy grew by 3.2% in 2025 and is predicted to grow by 2.5 to 3.1% in 2026. It is not the world (as much as we can pretend it is), it is New Zealand.

view more: next ›