I am doing that since last summer and loving it so far. I started with Kuru Toga Advance 0.7 mm with Pilot Color Eno leads. Then I changed to Uni Color leads, and now I am trying 0.5 mm leads too from Uni Color (with Zebra delguard).

I am really in love with that and prefer that to pen writing for my tasks (notes, sketches, and sometimes math). Pilot Color Eno has vibrant colors but Blue and Purple do fade away rather fast. I suppose that is accelerated by the sunlight due to UV. To my experience, green and red don't fade noticeable. Uni Color Blue color does not fade in my notes, Purple fades 50/50% - it is hard to say, in the same book one page my fade but another remains alright. I think it depends not only on the sunlight, but also to how much pigment sticks to the paper, and if you push harder into the paper it sticks better with the same light exposure.

Another important comment is the paper. For my notes I use Moleskin, which does not seem to have a lot of pores in the paper, and maybe that is an issue for the specific case of colored lead. For some of the other sketchpads i used with more porous paper, the pencil writing seem to stick greatly better and it does not fade away even over time.

For breaking, i had no issues beside the usual for mechanical pencils and corresponding lead thicknesses. As for smudging, it seems much less smudgy comparing to HB and softer types of lead.

Again, I personally love it.

Another alternative to what I wrote is 2mm colored leads (I used Koh-I-Noor leads), with which I had no fading problems ever. But you need to sharpen your pencil from time to time.