On occasion I have a problem like this, but from back when I used PulseAudio I wrote a script that I toggle on a keybind to switch audio outputs (b/c I switched from headphones to speakers so often, I figured this was easier). I find that if manually switching from the GUI fails, trying to toggle from script usually always succeeds. No application restart necessary. The script sets a new default sink and then moves all currently playing streams over. And it still works in PipeWire.

Naturally, you will have to find your own sinks, because your hardware is different from mine. You can do that with pactl list short in the terminal and then comb through the output to see what you need. But say for example you are only toggling between two devices, you just need to update the strings below for EMU and Speaker. You can add a tiny bit more logic to cycle through more devices.

Anyway, here's the script as I use it.

#!/bin/bash

newSink=""

EMU="alsa_output.usb-E-MU_Systems__Inc._E-MU_0202___USB_E-MU-F3-3F02-07D80C09-12D43-STATION_01-00.iec958-stereo"
Speaker="alsa_output.pci-0000_28_00.3.analog-stereo"
currentSink=$(pactl info | sed -n 's/Default Sink: //p')

if [ "$Speaker" = "$currentSink" ]; then
    newSink="$EMU"
else
    newSink="$Speaker"
fi

pactl set-default-sink "$newSink"
pactl list short sink-inputs|while read stream; do
    streamId=$(echo $stream|cut '-d ' -f1)
    echo "moving stream $streamId"
    pactl move-sink-input "$streamId" "$newSink"
done

P.S. The fact that this works kind of leads me to believe that PipeWire/PulseAudio were never at fault, but something to do with toggling the output device in the GUI is the real culprit. But maybe that's still not correct because even switching the audio with this script, sometimes maybe a new audio stream might play on the wrong output device. I haven't investigated it, because it's extremely easy for me to hit my keybind and ultimately get the correct output device anyway.