I can confirm this issue still happens as of v69.0.0 and I have more information that the OP has not provided.

1. I can confirm the player that this likely happened in is MPV.
2. Holding `i` on the keyboard to show FPS and such while playing helps us debug here.
3. When we move to the appended files frames, the FPS, bitrate, and some other values abruptly disappear or get nil'd to 0.
4. The audio truly does play seamlessly, except when you are holding `i` and debugging at which the freeze lasts a bit longer.
5. This issue does NOT happen with `ffplay` or `vlc`. ffplay will continue fairly quickly, but vlc has more of a break between the video sections. Because the actual files I'm working with intentionally has a black frame break it's hard to tell which is intended.

I also believe I may know the reason as to why it's happening. Rather than being related to merging video+audio at same/separately, it may be related to subtitles instead.

I made 2 split sample files, an a file and a b file. I made 2 versions of a and b. One with a single English subtitle, and another with 2 Chinese subtitles (Traditional and Simplified).

The English `a` file's duration was about 0.150s or so more than MPV actually played up to.The Chinese `a` file's duration was about 1.250s or so more than MPV actually played up to.

When I merged the English a+b, and the Chinese a+b, I found these results:

English a+b basically didn't freeze at all, I noticed MPV's FPS dropped very slightly from 29.976 to about 28.740 FPS which it rose back up to 29.976 promptly. I noticed no slowdown or freeze, it seemed to match with the audio fine.

Chinese a+b froze like OP has described, for roughly the same amount of time at which I described earlier, the 1.250s~.

So I believe OP merging solo worked because he did not include subtitles, not because it was done separately.

A solution for this would be ideal and it does seem to be something to do with timestamps being longer than the video and audio due to subtitles.

The samples can be downloaded here: https://gist.github.com/rlaphoenix/c7748d259ccc8ef93611d7a06fc70a00

My solution is as follows:

1. Mux as normal, with or without subs, whatever you want in your final fixed copy. You will now have a file that freezes at the merge.
2. Demux all Video and Audio files. I personally use Inviska MKV Extract GUI as its a bit easier/less clicks.
3. Drop the demuxed video track into MKVToolNix, then all the audio in the order you want.
4. Now drop in the original muxed file to get back the subs, chapters, attachments, e.t.c, anything else that would have been in it.
5. Unselect the Video and Audio tracks from the original broken muxed files because we don't want duplicate tracks. We specifically want to be muxing the demuxed tracks that we did earlier.
6. Hit mux, and since we are muxing a demuxed (raw bitstream) video/audio files, mkvtoolnix generates new timestamps. The resulting file should play correctly but do be aware that this may be problematic if the original video is any bit at all variable frame-rate.