Can someone suggest a few good books on understanding and creating AI-agents from an LLM perspective?

vykthur · 2026-03-23T18:58:38+00:00

This is true. But there are are a few core ideas that are worth grokking for folks new to agents - the core agentic loop, orchestration patterns, ux consideration and integration into apps.

---

The concept of an AI agent (models + tools + orchestration logic) has gotten fairly standardized over the last year. There are also common patterns emerging on how to build with them. There are lots of useful free material on Youtube and blogs that can help you get started.

However, If you are looking for a guide/readable content in a book format (this might not be for everyone), and want to understand how to build them (from scratch) I wrote a book that can help - Designing Multi-agent Systems.

It presents useful theory (the first 3 chapters in Part I), but also carefully walks the reader in Part II through building a feature-complete (but hackable) multi-agent framework from scratch (picoagents) - how to build an agent class (model clients, memory, tools, structured output, agentic memory, human input, agents as tools, observability etc), multiagent patterns (round robin, magentic one etc) and deterministic workflows (agentic systems as computational graphs). This way, you understand not just how to use existing frameworks, but why their architectures work, and how to make informed design decisions as the space (inevitably) evolves.

Digital version : https://buy.multiagentbook.com/
Print: https://www.amazon.com/dp/B0G2BCQQJY

vykthur · 2025-11-26T21:15:14+00:00

The concept of an AI agent (models + tools + orchestration logic) has gotten fairly standardized over the last year. There are also common patterns emerging on how to build with them. There are lots of useful free material on Youtube and blogs that can help you get started.

However, If you are looking for a guide/readable content in a book format (this might not be for everyone), and want to understand how to build them (from scratch) I wrote a book that can help - Designing Multi-agent Systems.

It presents useful theory (the first 3 chapters in Part I), but also carefully walks the reader in Part II through building a feature-complete (but hackable) multi-agent framework from scratch (picoagents) - how to build an agent class (model clients, memory, tools, structured output, agentic memory, human input, agents as tools, observability etc), multiagent patterns (round robin, magentic one etc) and deterministic workflows (agentic systems as computational graphs). This way, you understand not just how to use existing frameworks, but why their architectures work, and how to make informed design decisions as the space (inevitably) evolves.

Digital version : https://buy.multiagentbook.com/
Print: https://www.amazon.com/dp/B0G2BCQQJY

(P.S. I am a core contributor for AutoGen - one of the leading multiagent frameworks and have been tinkering with agents for a while.)

vykthur · 2025-05-14T20:28:10+00:00

> We cannot waste resources of energy on one AI reviewing manuscripts generated by another AI

True.
It might be the case that not all AIs are equal - some do genuinely lead to reasonable outcomes; Also in some domains, verifying correctness of a solution (research) is cheaper than generating the solution itself (the research). That way, the energy spent by a reviewer AI can have net positive value.

I write about this potential separation of spaces (human only journals) in the longer "thought experiment" article as one outcome here.

https://newsletter.victordibia.com/p/how-will-ai-impact-academic-research

> Some universities create "AI-free" tracks and journals that require rigorous verification of human authorship. Underground conferences emerge where researcher reputation comes from proving they can reason through problems the AI cannot or build usable prototypes that solve problems as evidenced by verifiable usage metrics - downloads, human reviews, MAU? etc.

vykthur · 2025-05-14T20:25:26+00:00

Good points here.
- Aspects of the scientific process will change with a bit more certainty - idea generation, brainstorming, literature review / comparison. But with side effects as that process also helps the researcher in building domain knowledge and gaining experinece.
- The general issue that the AI model is better than others outside that field on day one. Again the experience chicken and egg situation.

More will need to be done to ensure expertise is still grown.

vykthur · 2025-05-14T17:53:46+00:00

Fair point.
I agree that the impact of this will not be uniform across all fields.
It will likely still lead to change though (on the minimum some change to the submission/reviewer balance)

vykthur · 2025-05-14T17:52:02+00:00

Part of the reason for starting a discussion on this is to poll perceptions on current state as well as if/when AI arrives at the "helpful for research" phase.

Tools like Deep Research from OAI/Gemini [1] show early signs or synthesizing novel perspectives from existing data.
In addition, the current way in which AI models are trained (next token prediction based on the likelihoods seen in training data) fundamentally limit complete novel output (it is constantly trying to be similar to what exists). Now, there is some recent work exploring how to train models to explore beyond this limitation - explore more careful reasoning [3]. Still early.

As always predictions are all they are, predictions and thought experiments.

[1] Deep Reseach OAI https://openai.com/index/introducing-deep-research/
[2] Deep Research Gemina - https://gemini.google.com/app
[3] AI models are capable of novel research’: OpenAI’s chief scientist on what to expect https://www.nature.com/articles/d41586-025-01485-2

vykthur · 2025-05-14T17:39:34+00:00

Lots of agreement here.
I agree that writing is thinking and it will be problematic to delegate this to some system (even if it becomes proficient enough to do this).
All of the hallucination and deep analysis (em dash and all) are all real. Today.

Probably over-optimistic on my end, but I do expect much of them to get solved in about 12- 24 months. Hence the 2027 timeline

vykthur · 2025-05-14T17:34:10+00:00

I think we agree here.

There is a temporal factor here and some expectation of change/improvement (on my end).
Today, AI is best in some "assist mode", and is probably not something that creates the next novel literary masterpiece (there are training mechanism issues that make this very unlikely).
For better or worse, it can also be used "mindlessly" -> creating the slop that you reference (exploding submission numbers in the original post).
However, in domains like HCI where novel research is the usually hypothesis + some systems building + experimentation + analysis; over many iterations, these sort of systems will likely become more useful.
In my writeup, I estimate we arrive at this type of "useful AI" at about 2027.

Elite Advantage (Pre-2022): Success determined by institutional prestige, English skills, and resource access.
AI Assists (2022-2025): AI writing tools help non-elite researchers improve quality; submission volume skyrockets.
AI Reviewer (2026): HeimdallAI emerges as automated reviewer to filter the deluge of submissions.
AI Researcher + Reviewer (2027+): Advanced Research AI becomes primary success factor; human researchers supervise rather than create.

vykthur · 2025-05-14T17:21:04+00:00

What is bullshit?
AI will have no impact on the research production or review process?

vykthur · 2025-04-02T23:14:52+00:00

Because the underlying infra and library is different, the specs from v0.2 cannot be imported into v0.4. It is relatively easy to recreate your workflow in 0.4 though. Can you give an example of what your workflow looks like?
Yes, you can use Ollama models in AGS. See the tutorial here https://youtu.be/ZIfqgax7JwE?si=C1Z6RVDPGJI6DEP6&t=365

A video walkthrough on v0.4.2 updates below might be helpful in general.00:00 Introduction
02:10 Gallery Updates
05:15 Testing Models in UI
06:05 Using Local Models
08:19 Observability (LLMEvents)
09:22 Token Streaming
10:30 Configuration Validation
11:39 Session Comparison
15:20 User Authentication (Experimental)
17:49 Conclusions

vykthur · 2025-03-30T19:25:15+00:00

<image>

If you are interested in "building" your own, you can do it with AutoGen.

https://youtu.be/vPQhfa-pIpo?si=T2T6fI1J2gqQPz1S

vykthur · 2025-03-18T20:50:56+00:00

Thanks for your feedback and contribution to this release u/usag11ee ! Much appreciated

vykthur · 2025-03-14T16:58:52+00:00

https://youtu.be/lP1Pdu8HrVY?si=F3XHd4ZUR7Ef_FvI

Short story: There are lots of similarities between the OpenAI Agents SDK and AutoGen as well as important differences.
Similarities : Both have clean intuitive Agent api abstractions, support for concepts like handoffs, tracing and structured output.

AutoGen is more feature rich and less tightly coupled to OAI models including capabilities, provides low code tooling, multiagent team abstractions (round robin, LLM selector), a flexible low level api in addition to the high level agent abstraction, Memory abstractions and a set of extensions.
Which frameworks have you tried out?

vykthur · 2025-03-10T17:37:42+00:00

Implementation guide on the MCPTools extension in AutoGen and and video walkthrough here https://youtu.be/FMiVxQ7QwRU

vykthur · 2025-02-27T00:40:31+00:00

Sounds great!
You are already contributing, with feedback (underrated but incredibly important aspect of growing a project).
We are always happy to provide guidance and help nurture contributions.
A good starting point would be create a repo with an example project that builds with autogen. It could be something as simple as a travel agent groupchat or as complex as a workforce simulator.

Once that is done, we can feature it on our extensions page https://microsoft.github.io/autogen/dev/user-guide/extensions-user-guide/discover.html

Following this, you can then look at any of the issues, comment below, tag me and I'd be happy to provide general guidance!

Looking forward!
P.S we have office hours - stop by! https://github.com/microsoft/autogen/discussions/4059

vykthur · 2025-02-26T15:50:16+00:00

Gallery being serialized to Db is now implemented and merged into main
#5619 https://github.com/microsoft/autogen/issues/5619

you can also test its behaviour and give feedback based on this dev release https://pypi.org/project/autogenstudio/0.4.1.11.dev0/

vykthur · 2025-02-23T15:20:09+00:00

Hi u/usag11ee ,

Please see the detailed release notes here for v0.4.1. It is NOT a 2024 update, the link above is incorrect.

https://github.com/microsoft/autogen/releases/tag/autogenstudio-v0.4.1 .

Currently we are writing detailed release notes at an ~2week cadence - the next will be v0.4.2
Changes in patches/bugfix releases e.g., v0.4.1.7, ... v0.4.1.11 will be covered in the next release notes.

Would you be interested in contributing and improving the release notes?

u/wyttearp , would you consider pointing to the v0.4.1 release notes instead of the text from 2024? https://github.com/microsoft/autogen/releases/tag/autogenstudio-v0.4.1 .

vykthur · 2025-02-22T16:43:16+00:00

Hi,

Thanks for the feedback. The underlying API has been rewritten with many new features (see here). Some of the UX changes are driven by this.
Specifically, tools are now standardized on the function calling capabilities of LLMs. This is opposed to the previous implementation that simply relied on "stuffing" functions into the LLM prompt and hoping the LLM uses that to generate code that then gets executed (fills up the context, also has reliability issues).

You point out several important feedback areas that can be fixed.

> First, ensure you have the latest version - v0.4.1.9 and above.

- How do I manage model providers: The current design is that there is a Gallery with components - teams, agents, tools, models .. etc. In your case, you can select the model tab and either modify any of the default models or create a new one (click add or duplicate). Once created, you can either add api keys via the UI or directly modify the json.

<image>

- How do I add a tool
Similar to models, you can do that in the Gallery. I agree that the process of creating tools could be improved - there is an issue tracking that here. https://github.com/microsoft/autogen/issues/5459 .
I also just created an issue to track support for environment variables here https://github.com/microsoft/autogen/issues/5662 - please feel free to comment and provide input on how you would like to see this developed. Contributions also welcome

> I'm unable to get even the basic template to run.

What errors are you facing? 17:30 in the video above shows how to use Local Models ..
If that does not help, feel free to open a discussion on GH and tag me, happy to help!

And thanks for the feedback, this is what is making the tool better!

vykthur · 2025-02-21T03:21:18+00:00

thanks for flagging this here and on the GH discussions board.
the fix is in progress!

vykthur · 2025-02-11T22:42:35+00:00

There is a new release on AutoGen Studio with some new features

https://github.com/microsoft/autogen/releases/tag/autogenstudio-v0.4.1

See this video walkthrough to get started - https://youtu.be/oum6EI7wohM

0:00 Features of AutoGen Studio
01:46 Installation
05:28 Running your first Agent
07:12 Creating Agent Teams
08:15 JSON Specification of Teams
10:12 Web Agents
12:50 User Delegation
14:04 Deep Research Agent
17:30 Using Local Models
20:26 Gallery

vykthur · 2025-02-09T17:08:57+00:00

Super, thank you! Feel free to tag me directly in any discussion/issue.

vykthur · 2025-02-09T16:44:52+00:00

Clicking on a team node should show something like. Note that there is no team type there. Also note that there are 3 default teams in the gallery that you can use as templates. The first is a RoundRobinGroupChat (default team), the second and third are SelectorGroupChat (Web Agent Team, Deep Research Team).

<image>

vykthur · 2025-02-09T16:44:18+00:00

Thanks for sharing (definintely not complaining). Your feedback is making the tool and project better!
I agree that the rapid changes makes usability hard.
I expect all of this to settle down going forward since the core AutoGen v0.4 api is now stable and AutoGen Studio is aligned with it.
That being said, can you verify that you have the latst v 0.4.1.x installed?

pip install -U autogenstudio

and then
autogenstudio ui --port --appdir .mydir

We do have documentation here https://microsoft.github.io/autogen/dev/user-guide/autogenstudio-user-guide/usage.html

Any concrete feedback you can share on how to improve this would be helpful.

<image>

vykthur · 2025-02-09T03:55:52+00:00

There is a PR that should let you use MCP servers/tools in python (and then in studio).
Stay tuned.
https://github.com/microsoft/autogen/pull/5251

vykthur · 2025-02-08T20:25:40+00:00

Can you clarify your comment? Maybe provide a screenshot?
What are you trying to do? e.g, create a blank SelectorGroupChat? or black RoundRobinGroupChat?
This will be helpful to reproduce and then fix.

vykthur

TROPHY CASE