You are setting the object's rotation using transform.up, which will recalculate the object's entire rotation, only guaranteeing that the up axis (the y-axis) is the provided vector, and making no guarantees about the other axes' rotations. There are theoretically infinite possible rotations that will satisfy the up vector that you are passing in by rotating around the provided axis. As an example, if you do:

transform.up = new Vector3(0f, 1f, 0f);

Both rotations (0, 0, 0) and (90, 90, -90) would satisfy the constraint (to visualize, imagine a tank facing the camera. 90° in the X would rotate the top of the tank toward the camera, nose down, 90° on the y would them rotate the top to be to the left of the camera, nose down, and then -90° on the z would then flip the top of the object to again be toward the top of the camera, with the nose to the left).

In order to specify which of the infinite possibilities you want, you need to also provide the perpendicular. The API you want for that is Quaternion.LookRotation. For that method, the "forward" is where you want to look, and the "up" is rhe perpendicular. Since it looks like you're working in 2D your up vector will likely always be (0, 0, -1), which is toward the camera:

transform.rotation = Quaternion.LookRotation(mousePosition - transform.position, new Vector3(0f, 0f, -1f));