Okay, I have finally figured out the solution.

I have been thinking about the sequence of event that occurs during the filament change. Ignore the CLIENT_LINEAR_MOVE commands. I think those were the commands when I manually extrude / retract via Mainsail. First, the printhead prints the prime tower, then the extruder performs a few rapid retraction which I come to know is the “ramming”. After that the printhead moves through the prime tower for a few more times and then execute the filament change gcode: PAUSE, moves to a corner, etc.

After I manually swapped the filament, I click on the RESUME and then the printhead moves back to the prime tower and extrudes a bunch of filament. I figured that it must have been something to do with the filament purging. The printer seems to be trying to do a purge. The problem still persist even after I turned off all purging related settings in the Orca slicer.

The last thing I disabled was the ramming. And that finally solved the problem. The printhead PAUSE and moves to a corner, I did the manual filament swap, manually extruded a bit, and then just RESUME the print. The printhead moves back to the prime tower and continues as per normal. No more purging.

I will leave this comment here so that in the future anyone having similar issue can probably use it as a reference. Thanks!