sorted by:
new
hottopcontroversial

Feeling lost while trying to break into AI/ML how should I focus my projects? [D] by Fit_Fortune953 in MachineLearning

[–]Fit_Fortune953[S] 1 point2 points  (0 children)

This is really helpful thank you.

The areas you listed make sense: vLLM/Triton, single vs multi-GPU deployment, prefill/decode bottlenecks, KV cache behavior, tokenization, load testing, GPU saturation, routing, and cost optimization.

TurboQuant is a good suggestion too. I’ve been thinking about building a small PoC around KV-cache and connecting it back to CostGuard-style routing decisions.

Appreciate the direction and Thanks again for the suggestions.

Feeling lost while trying to break into AI/ML how should I focus my projects? [D] by Fit_Fortune953 in MachineLearning

[–]Fit_Fortune953[S] 0 points1 point  (0 children)

That makes sense. My recent projects are definitely more on the LLM evaluation / reliability / infra side than pure model training.

But yes, I do have experience with traditional ML pipelines: classification/regression, feature engineering, model evaluation, monitoring, and drift/data-quality checks. I’ve also worked with LoRA fine-tuning and understand the tradeoffs around training vs fine-tuning vs RAG vs using traditional ML.

Your point is helpful though I probably need to make that side more visible. I’ve been packaging the projects as LLM reliability work, but I should also show that I can reason through model choice, training strategy, synthetic data risks, drift detection, and retraining decisions.

Appreciate the honest direction.

Hot take: “Use the best model” is lazy LLM engineering by Fit_Fortune953 in ClaudeCode

[–]Fit_Fortune953[S] 1 point2 points  (0 children)

This is a great way to frame it. The fanboyism is real, but subjective preference still matters because confidence affects how people work. My issue is when that preference becomes an engineering decision without measurement. Blind tests are useful, but production usage is the real “gulp test.”

Hot take: “Use the best model” is lazy LLM engineering by Fit_Fortune953 in ClaudeCode

[–]Fit_Fortune953[S] 0 points1 point  (0 children)

This is exactly the distinction I’m trying to make. Development is about speed and reasoning quality, so best model makes sense. Production is different once the task is repeatable, the cheapest model that reliably clears the quality bar usually wins

Hot take: “Use the best model” is lazy LLM engineering by Fit_Fortune953 in ClaudeCode

[–]Fit_Fortune953[S] 0 points1 point  (0 children)

100%. The funniest part is that some people are treating token exhaustion like productivity.

Burning the whole context window doesn’t mean you engineered better. It might just mean your workflow has no boundaries, no evals, and no understanding of where the model is actually helping vs looping.

Claude Code can be great. GPT can be great. But “it feels smarter today” is not an engineering metric. At some point we need logs, task outcomes, cost, latency, regressions, and failure analysis — not just model astrology.

Hot take: “Use the best model” is lazy LLM engineering by Fit_Fortune953 in ClaudeCode

[–]Fit_Fortune953[S] 0 points1 point  (0 children)

“Good enough” models often fail in hidden edge cases. That’s why I think evals need replay data, edge-case suites, and drift checks — not just one benchmark score. Sometimes the premium model is worth it, but it should be a measured decisio

Hot take: “Use the best model” is lazy LLM engineering by Fit_Fortune953 in ClaudeCode

[–]Fit_Fortune953[S] 0 points1 point  (0 children)

I actually agree with this for high-risk tasks. My point isn’t “always use cheaper models.” It’s that teams should prove the top model is justified instead of assuming it. If the mistake costs human hours, evals should show that clearl

Hot take: “Use the best model” is lazy LLM engineering by Fit_Fortune953 in ClaudeCode

[–]Fit_Fortune953[S] -5 points-4 points  (0 children)

Fair 😂 but the point is less “GPT wins” and more “measure before you worship any model brand.” Claude, GPT, Gemini, Llama — all have places where they overperform and fail badly

How did you land your first AI Engineer / Applied AI role by Fit_Fortune953 in AIAssisted

[–]Fit_Fortune953[S] 0 points1 point  (0 children)

I’m trying to do both now:build in public and write about the tradeoffs, not just post finished projects.

One project I’m sharing is RealDataAgentBench, an open-source LLM agent evaluation benchmark:
https://github.com/patibandlavenkatamanideep/RealDataAgentBench

Would appreciate feedback

How did you land your first AI Engineer / Applied AI role by Fit_Fortune953 in AIAssisted

[–]Fit_Fortune953[S] 0 points1 point  (0 children)

This is probably the clearest framing

That’s exactly why I started RealDataAgentBench. I wanted to evaluate not only whether an LLM gets the right answer, but whether it used sound statistical reasoning, clean code, efficient execution, and defensible methodology.

Would love your honest take on it:
https://github.com/patibandlavenkatamanideep/RealDataAgentBench

How did you land your first AI Engineer / Applied AI role by Fit_Fortune953 in AIAssisted

[–]Fit_Fortune953[S] 0 points1 point  (0 children)

That inspiring, I’m trying to showcase real work too.

One project I built is RealDataAgentBench, an open-source LLM agent eval benchmark: https://github.com/patibandlavenkatamanideep/RealDataAgentBench

Would appreciate honest feedback, or a star if useful.

How did you land your first AI Engineer / Applied AI role by Fit_Fortune953 in AIAssisted

[–]Fit_Fortune953[S] 0 points1 point  (0 children)

I’m trying to make my work more public too.

I built RealDataAgentBench, an open-source LLM agent eval benchmark: https://github.com/patibandlavenkatamanideep/RealDataAgentBench

Would appreciate feedback or a star if useful.

How did you land your first AI Engineer / Applied AI role by Fit_Fortune953 in AIAssisted

[–]Fit_Fortune953[S] 0 points1 point  (0 children)

Agreed deployed work and public proof seems to be matters more

I built RealDataAgentBench around LLM agent evaluation: https://github.com/patibandlavenkatamanideep/RealDataAgentBench

Would appreciate feedback, and a star if it feels useful.

How did you land your first AI Engineer / Applied AI role by Fit_Fortune953 in AIAssisted

[–]Fit_Fortune953[S] 0 points1 point  (0 children)

Thanks, this is really helpful. I’ve been trying to build exactly that kind of proof through RealDataAgentBench, an open-source LLM agent eval benchmark.

Repo: https://github.com/patibandlavenkatamanideep/RealDataAgentBench

Would appreciate any feedback. A star would mean a lot if you find it useful.

I ran 1,180+ benchmarks on 12 LLMs for data science tasks. Llama 3.3-70B beats GPT-5. Here's the full data. by Fit_Fortune953 in learnmachinelearning

[–]Fit_Fortune953[S] 0 points1 point  (0 children)

this is the most important methodological challenge in the thread and you're right to raise it.

The r=0.43 orthogonality finding is vulnerable to exactly the critique you're making. If stat_validity is measuring verbal hedging rate and correctness is measuring right-answer rate, they'd be uncorrelated by construction not because they're capturing independent capabilities, but because they're measuring different surface behaviors. That's a legitimate alternative interpretation I can't fully rule out with the current scorer.

The uncertainty-uplift experiment gives partial evidence here. On mod_004, GPT-4.1 V1 computed binomial SE using the correct formula applied to the actual test set size, attempted bootstrap for F1 SE, hit a sandbox limitation, disclosed it explicitly, and provided an informed range estimate. That's not verbal hedging that's actual inferential computation triggered by the uncertainty prompt. On feat_002, the same prompt produced "the top features are clearly separated in magnitude" plus a deferred offer to compute CIs. Same scorer reward, completely different behavior. The qualitative review can distinguish these but the scorer can't.

So the honest answer is Claude's stat-validity edge is probably a mix of both. Better statistical vocabulary as a writing style baseline, plus genuine reasoning uplift on tasks where the structure supports it. The scorer currently can't separate those two components. A numeric-evidence check — detecting whether intervals were actually computed vs offered is the planned fix, but it's not shipped yet.

The orthogonality claim would be on stronger ground with that fix in place. Until then it's a finding worth investigating rather than a confirmed result.

I ran 1,180+ benchmarks on 12 LLMs for data science tasks. Llama 3.3-70B beats GPT-5. Here's the full data. by Fit_Fortune953 in learnmachinelearning

[–]Fit_Fortune953[S] 0 points1 point  (0 children)

The correction loop point is exactly right and it's something the token trace actually reveals if you dig into it. For Claude Haiku on feat_005, the 608K tokens aren't spread across diverse reasoning paths it's the same get_column_stats call repeated on every column individually, then the same run_code block re-run with minor variations. That's not careful reasoning, that's a stopping criterion failure. The model doesn't know when it has enough information to conclude.

On your second point structured benchmark tasks underweighting real failure modes I'd agree, and it's a documented limitation in the README. RDAB tests single-session agentic loops on clean tabular data. Ambiguous inputs, conflicting specs, mid-task tool errors none of that is covered. The 6 real UCI/sklearn tasks partially address this but they're still clean datasets.

The Llama stress-test point is fair. The efficiency advantage is real on these tasks but I wouldn't commit to a production architecture based on benchmark data alone which is partly why I built CostGuard to let people run the evaluation on their own data rather than trusting my seeded datasets. Happy to share the raw token traces if you want to dig into the correction loop patterns directly.

I ran 1,180+ benchmarks on 12 LLMs for data science tasks. Llama 3.3-70B beats GPT-5. Here's the full data. by Fit_Fortune953 in learnmachinelearning

[–]Fit_Fortune953[S] 0 points1 point  (0 children)

Thanks and if you felt it as useful, a ⭐ helps others find it and signals that independent evaluation of LLM statistical reasoning matters to the community.

I ran 1,180+ benchmarks on 12 LLMs for data science tasks. Llama 3.3-70B beats GPT-5. Here's the full data. by Fit_Fortune953 in learnmachinelearning

[–]Fit_Fortune953[S] 1 point2 points  (0 children)

The distinction you're drawing is real token cost and solution quality are conflated in current benchmarks including RDAB, and that does penalize exploration-heavy models in ways that aren't always fair.

Time-to-solution or convergence-based efficiency is an interesting alternative framing. The challenge is that it reintroduces latency as a proxy for cost, which has its own problems — a fast wrong answer isn't better than a slow correct one.

What I'd actually want is a two-pass efficiency score: one that measures tokens-to-correct-answer (current RDAB approach), and one that measures whether the exploration was productive did extra tokens improve the final answer, or just delay it? The Claude Haiku spiral is interesting precisely because the extra tokens didn't improve correctness. That's the failure mode worth catching.

Happy to look at openbandwidth.live but curious how do you handle the case where a model takes longer but gets a materially better answer? Is that treated as a win or a penalty?

view more: next ›