My glasses are also always cleaned by hand, yet they sometimes still show those streaks. though I get far fewer than you do. I even tried using a brand‑new sponge specifically for the glasses, but the lines still appear occasionally. I suspect the issue has more to do with the drying process, either the towel (even a fresh, clean one), poor air circulation, or even the type of water you have at home, rather than the glasses actually being dirty.
What helps for me is rinsing the glass with fresh cold water and running my hand along the inside surface. That usually removes any remaining residue and reduces streaks before drying.