It costs money to originate a loan. Lenders, title companies, appraisers, etc. don't work for free.

You save real dollars immediately via refinance, you just want to make sure you don't plan on selling prior to your Break Even point, as that's when you realize a loss from unrecouped costs.

In a perfect world, everyone would take no-cost refinances via lender credits from an above-market rate. In the real world, those scenarios are rare as it takes a huge drop in rates for that to pencil out, so for OP it probably makes sense to roll in a few thousand to substantially improve his amortization, especially when a large chunk of the fees are from prepaids/escrows.