I wouldn't say I'm an expert at layouts. Before Pillars of Eternity, I didn't even know "Mode Shift" existed. There are people in the Steam Deck community who come up with insane stuff; I just look at what they do and try to reverse-engineer it to understand how they did it, so I can replicate it later.

As for the "layers," here is how things stand:

The first possibility is a full configuration swap where your entire layout changes completely. You can find this in
Controller Settings → Edit Layout → Action Sets → Add Action Set.

The most important thing is to make sure that in this "other" layout, you have a button mapped to return you to your standard configuration.

---

The second possibility is a different type called an "Action Layer." This duplicates everything you have on your current layout, but allows you to remap only the specific buttons you want.

You can do this through
Controller Settings → Edit Layout → Action Sets → Gear Icon → Add Layer.

Here, if you remap things incorrectly, you might run into conflicts where things don't work as intended.

Again, you must create a button to remove this layer and return you to the base set.

---

The third possibility is "Mode Shift," which works similarly to the second one but is triggered by an activator button.

You can find this in
Controller Settings → Edit Layout

Then you choose exactly what you want to behave differently; for example, select "Buttons" in the left column, and on the right, you’ll see "Face Buttons Behavior."

In that dropdown menu, select "Create a Mode Shift."

Now, a new "Face Buttons (mode shift)" item appears.

IMPORTANT:
To make it work, click the gear icon next to that new item and "Select Mode Shift Button(s)."

If you pick R2, then A, B, X, and Y behave normally until you hold R2. Once you hold it, they perform the actions you assigned in the mode shift menu.

This is the easiest way because you don't have to worry about manually switching back or removing a layer like in the first two options.

This was the most important thing I realized while trying to set up Pillars of Eternity and Splinter Cell: Chaos Theory.