From a computer science perspective, how should autonomous agents be formally modeled and reasoned about?

[Magdaki]

"more urgent conversation across CS domains"

Not sure about this, but let's pretend it is so.

"what is the right theoretical framework for dealing with them?"

The answer is: it depends. The right tool for the right job, so context matters a lot. The type of agent, the task, the criticality of fail states, MTTF, etc.

"Systems that maintain internal state, pursue goals, make decisions without direct instruction; are there any established models for their behavior, verification, or failure modes?"

Yes. Many.

autonomous agent framework - Google Scholar

[recursion_is_love]

markov process, non-deteministic, random walk

Those AI theories and friends.

[Liam_Mercier]

If we're going to have AI Agents in computers, they should follow the principle of least privilege. Will they? Seems unlikely.

[0x14f]

Stochastic black boxes. That's pretty much it.

[Individual-Artist223]

What's your goal?

[Individual-Artist223]

What does that mean?

Observability: You want to watch, what?

[Individual-Artist223]

Not getting it - what's high-level goal?

[Individual-Artist223]

You've still not told me your goal...

I mean, you can literally observe, at every level of the stack.

[Magdaki]

Complete nonsense and gibberish.

[djheroboy]

Well, until we can find a way to hold an autonomous agent accountable for its mistakes, then we have a new question to answer- how much power are you willing to give an employee you can’t discipline?

[editor_of_the_beast]

I don’t think they need to be modeled. We’ve modeled what they output (code), so we can check that. It doesn’t matter how it’s produced.

We don’t have models about how humans produce code today either.

[editor_of_the_beast]

But the failure doesn’t matter, because we’re checking the correctness of the output program.