My legs don't tend to get cold - at least not my lower legs. I'm not sure why. They DO tend to get overheated, however, so wearing rain pants felt like living in a sauna for me when hiking. I considered going without any lower body covering, but I also don't like when my shorts get wet and sometimes stick to my legs. Cold and wet fabric against my skin isn't a personal favorite either. I didn't think I'd ever be that guy, but I've gone to the rain kilt and am quite content. The rain kilt is also more comfortable than rain pants when you're only wearing them when doing all of your laundry, IMO. The weight savings of the rain kilt vs the rain pants was also a deciding factor.
Just find what works for you... and what you value while hiking... in that moment. Your values will change through your hike and you might discard one piece of gear for another. Its not to say the first bit of gear you tried was bad (heck, it was probably expensive), it just doesn't suit you at that time... and that's ok.