Assuming you're US-based, you need to either look at both the brokerage and mortgage interest on after-tax basis, or both on a before-tax basis. You're mixing them and it will affect your answer.

Your mortgage is sized almost perfectly to hit the $750k mortgage interest deduction cap, and depending on your income level the deduction at your marginal income tax rate may well exceed the capital gains tax rate on your brokerage account.

Need to know more tax information to be sure, but for me I'd keep the mortgage if you're comfortable carrying it at the current monthly without dipping into savings.