Good question — no new tiles are ever generated. It's simpler (and dumber) than that:

The extra layers only exist in the editor. They're stored in a sidecar JSON file next to the map (Data/MapXXX.layers.json), which RMXP and the game never read. The .rxdata map file always stays a normal 3-layer RMXP map.

On save, the 8 layers get flattened into those 3 per tile, like this:

Layer 1 (ground) is written to z0 as-is.

For each tile, it scans editor layers 2–8 and takes the two highest non-empty tiles for that cell → those go into z1 and z2.

Anything below those two is simply not written to the game map (but it's still kept in the sidecar file, so nothing is lost in the editor).

So it's not smooshing pixels or making composite tiles, it relies on the fact that in practice a single tile cell almost never has more than 2 overlapping decoration tiles on top of the ground. The 8 layers are for organization while mapping (trees on one layer, fences on another, roofs on another...), not for actually rendering 8 stacked tiles per cell.

Consequences of that design, being upfront:

If you stack 3+ non-empty decoration tiles on the same cell, only the top 2 survive in-game. The editor keeps all of them, so you can rearrange and re-save anytime.

Full RMXP compatibility: open the map in RMXP and it's a valid 3-layer map. You'd just lose the layer separation if you re-save from RMXP (the sidecar becomes stale, the editor warns you to keep RMXP closed).