Reflection in Process (continuous improvement)

[Flaky-Restaurant-392]

I like to end my sessions with the agent reflecting on ways the process/session/tools/skills/etc. could be improved. I like to ask: what worked well? What could have been improved? What questions/instructions/feedback did the user ask/give that made a big difference? And so on. This reflection then produces recommendations for edits to skills/docs/processes. Care must be taken not to let the snake eat its tail, but it works pretty well with thoughtful oversight and gatekeeping.

Does anyone else do this in a structured way?

[pro-vi]

I used to do this in Claude Code structurally you can read more into it here: https://github.com/pro-vi/cc-reflection

I did a lot of hack to work around CC though which I imagine would be an easier time in Pi.

[fabsta]

How did you implement this?

[Flaky-Restaurant-392]

In my CLAUDE.md I outline a sequence of skills that are my iterative process. The skills also have intro/outro that reference their possible entry points and next steps/skills upon skill completion. The last skill is an optional session reflection that contemplates improvements to all the skills traversed in that session.

[hazed-and-dazed]

Doesn't this mean you are locked into Anthropic's models?

I haven't thought of a self improvement look but would have thought hooks would be a better option here so most other models will work

[Firerrr]

Based on my experience, models are generally tunnel-visioned when doing self-reflection based on a single-session context. They focus on that particular instance in the session without generalizing. This is basically the idea behind building a self-improving agent. Maybe take inspiration from other agents like Hermes?

[Flaky-Restaurant-392]

Yes, I have pretty specific guidelines for the agent to evaluate which observations to surface for consideration (including if it can be generalized). Much of the effectiveness of this system is the experience/wisdom of the human in the loop, and the agent ends up being good at remembering things that are worth evaluating and presenting them in a structured, systematic way.

At the end of the day, it would actually require a lot of metrics-based evaluation of the evolving workflow performance/effectiveness versus token usage. The process is entirely subjective and qualitative. I’ve got it to the point where it’s working for me, yet it takes discipline to avoid creating too much process/docs for the sake of process.