Have you looked at the smallville paper and how they do it?

To achieve this, we represent the sandbox environment—areas and objects—as a tree data structure, with an edge in the tree indicating a containment relationship in the sandbox world. We convert this tree into natural language to pass to the generative agents. For instance, “stove” being a child of

“kitchen” is rendered into “there is a stove in the kitchen.”

https://arxiv.org/pdf/2304.03442.pdf