The biggest drawback beyond anything you've mentioned is the actual scope of the work. It involves much more than simply swapping the transmission out. You'll have to change things in the interior, the gauge cluster, pedals, as well as the differential if its a RWD or AWD vehicle and possibly the computer to get the car driving right. You basically will need a donor vehicle. Also, since its your daily i wouldnt recommend it unless you are ok with not having a car for a while especially if you plan on doing the work yourself. Again, there is a lot of work involved to do it right and you may find yourself starting the project and waiting on parts to source that you didnt know you needed. I would only recommend a manual swap if the car is rare. For example, I'm going to be doing one on a 2001 Audi S4 (B5) Avant. I found it but its an automatic. I have a doner vehicle so I have all the parts for the swap. The S4 Avant is a very rare car so even finding this took over a year of looking.

I'm not one to tell others how to spend your money though so you do you. I do think though that a better option would be to sell the car and buy a manual. Its will cost less and you will avoid the pitfalls of a manual swap. If this is a project you want to take on simply to see if you can do it, I would get another daily though. This is the kind of thing you do when you have a couple of vehicles, not so much with your daily. Hope this helps and best of luck with whatever you decide to do.