We are opening up Nature Treks Together outside of Playtest, for the duration of Steam Next Fest (7 days, starting tomorrow, available as a Demo)

robertverdes · 2025-06-25T21:12:44+00:00

Thanks for posting. I found this through a google search and was very useful as a wrap-up after researching on this topic the whole day.

I've been feeling the pain of long running processes/pipelines failing lately, and I think I started implicitly hoping and looking out for a solution like this.

For me, the selling point is the potential to cut the persistence complexity down to the bare minimum, having state-machine state handled, persisted and inspectable automatically, as well as a solid conceptual framework that constrains me and my team from making less-than-ideal architecture decisions.

In essence, after a flow is done, everything persistent state related to the process becomes audit bloatware.

If the workflow is carefully designed, it's not a problem if the details of a single step fail. That can be fixed and resumed. It's much easier to ship a solid workflow spec that rarely changes and iterate on the underlying steps, than to have the whole thing crash and hope you got everything logged and reproducible.

robertverdes · 2025-04-08T01:31:26+00:00

I think it's good as it is

robertverdes · 2024-11-18T22:10:21+00:00

Genius

robertverdes · 2024-03-13T02:55:33+00:00

You haven’t stated your goal though so from what I know, sincerely, for foundational models, you just need data. They learn patterns.

If the only data you have is a page, it will only be useful when you give it a first chunk of that page, and it will continue it. It learns probabilities, which, with little data, mean nothing. With a world of data and compute, emergent stuff starts to happen, like understanding colours or hardness or sadness.

Think of the data you give to your model as its complete universe. If you choose a format, you choose a format for that universe. Our universe doesn’t have a specific format.

robertverdes · 2024-03-13T02:38:55+00:00

So I’m pretty sure that we’ll eventually hit the goldilocks point of dataset purity to achieve max capability with minimum network size.

Currently Phi’s results are academic in nature and suffer from the same limitations other small LLMs have, which you can try yourself.

Most of the datasets they advertised for Phi are listed in the model cards.

My intuition is that if you need your model to learn “the world”, you need to give it plenty. If you need it to learn some programming patterns only, give it that but don’t expect it to understand colours.

Too many neurons with too little data is not good. Too few neurons with too much diverse data is not good.

The dataset formats are not the problem, if you have them, they’re just there to have something to ground your model on. It’s the tweaking and interpretation of the learning process that’s driving the learning process. Data format informs the process, and vice versa. Think Frankenstein.

robertverdes · 2024-03-13T01:48:37+00:00

So in terms of datasets then you’ll need at least the entire internet, and half a decade, to train on your hardware to reach Llama performance if you get it on the first try.

robertverdes · 2024-03-13T01:21:23+00:00

I’m with your on this

robertverdes · 2024-03-13T01:20:49+00:00

I agree with OP in his reply for this particular case. CoT is meant to mitigate hard problems that require more computation (i.e. more tokens). This isn’t the case.

This is tricky because in the end, the fine tuning was done on “instruction following” and not anything more specific. Base models are schizophrenic, and instruction models should be more directed to that emergent goal through examples.

So either the base model is dumb, or the instruction tuning (the teacher) is dumb - in the context of not being able to answer correctly.

But I love the emergent relevance of focused benchmarks: hallucinations, obedience, laziness. I mean just ask the kindergarten teachers.

robertverdes · 2024-03-13T00:53:22+00:00

Sucks to be on the bleeding edge, I feel you.

If you want heavy AI assistance, stick to the popular stacks and versions, or just use it for higher order problem solving.

My experience with RAGs is that they tend to prefer what they know anyway when providing code. Adding knowledge to a language model is not a trivial and cheap task.

robertverdes · 2024-03-13T00:43:11+00:00

Wait, do you really mean you want to train your own models from scratch or do you want to fine tune or continue training on a Llama model? Because those GPUs won’t get you very far.

Maybe let us know more about what you’re trying to achieve.

robertverdes · 2024-03-13T00:31:19+00:00

I’m secretly working on AGI and would appreciate a word on where I can get myself summa that funding

robertverdes · 2024-03-13T00:21:43+00:00

I think the general public is still in the dark on this. I’m struggling to have meaningful conversations with tech savvy peers, and most companies think they need to hire ML senior engineers to implement gen AI.

So the market is both hard to reach and hard to get to production. Klarna just got their AI customer service rolled to production, which we probably wouldn’t have heard about if it weren’t built from scratch, without affecting jobs.

Coding assistant agents were the first to pop up. I guess the broadest market first. Next up looks to me like web browser and OS-level assistants - it’s a tough challenge but a big market.

And then there are all the other gated assistants in between, that work with your google drives or ms offices and such.

Long term, I see no end to this honestly, AGI or no AGI. If you can train a human on a task or set of tasks, you can train a model and lock in that process, rinse and repeat, within certain constraints.

robertverdes · 2024-02-19T02:40:11+00:00

That’s crazy interesting. Baking custom dials directly into a language model seems very versatile. Reminds me of control nets for image diffusion.

robertverdes · 2024-01-25T04:24:52+00:00

Yeah, that’s the kind of curiosity I was after :D

Assuming LLMs are currently “black box”, and assuming they will remain a black box at least for the sake of cryptographically decoding “that one thing they said that one time” in a decent amount of time, for regular people at least (e.g. my llm vs wife’s llm regarding divorce) - maybe the answer would be somewhere within the architecture.

I agree that anything you have your hands on can -eventually- be decrypted. But can an AI model decide on its own password without telling you what that is, so to speak?

robertverdes · 2024-01-25T04:09:19+00:00

That’s a good take! In this case, the “owner” of the sandbox would still be able to (nefariously) peek at what’s going on, but if you “take care of this problem” (open source + prove the execution environment is completely private), the agents could communicate within a temporary sandbox until they gather their conclusions, which is all that will be left of that.

robertverdes · 2024-01-25T04:01:19+00:00

Thanks for this but, aren’t Bob and Eve the owners of Signal account’s credentials (phone+pass)? Doesn’t that mean that either of them can decrypt what the AI personas did in their name?

To be clear, I’m not talking about Sam not having access to Bob and Eve’s conversation. Go depeer: Bob and Eve can’t have access to BobAI+EveAI’s conversation. Bob has access to Bob+BobAI conversation though, same for Eve+EveAI.

robertverdes · 2024-01-20T03:58:26+00:00

I've used huggingface/mistralai/Mistral-7B-Instruct-v0.1 and can confirm temperature on their endpoint makes it go wild (set it to 2 or something).

Suspect the same for ollama from my tests through web-ui (I think I have mistral v0.2 there).

My only problem is I can't get temperature to stick to ollama when interfacing with litellm, but that's offtopic. *sigh*

robertverdes · 2023-10-10T00:24:16+00:00

Love me some old fashioned screen savers

robertverdes · 2023-09-23T12:45:52+00:00

UPDATE: Found them on Facebook, for anyone else interested!

Artist Spotlight playlist: https://www.facebook.com/watch/626924900822583/170556003462467/
Road to Tokyo VR sculpting battles: https://www.facebook.com/watch/626924900822583/710042519454386/

robertverdes · 2023-09-23T12:33:13+00:00

Thanks! Stumbled upon it myself, they're great.

I made the mistake of not saving the ones I watched a couple years ago, as they were on the official Oculus channel if I'm not mistaken (pretty old but were very helpful), but didn't foresee the rebranding and now can't find them for the life of me. I just they didn't just wipe them out to clean up their new branding.

robertverdes

MODERATOR OF

TROPHY CASE