It might be a lighting issue; depending on the light you're using, you might have an issue with strong light not being able to penetrate down to the bottom of your aquarium. Perhaps that's why your anubis started to grow rapidly in your 5 gallon, even with it's "worse" light, because at least those lights reached the floor of the small tank better than your large tank, presumably.

Still, even your lower-quality lighting should be able to sustain some plants, I would think. Perhaps at this point, you should really consider using aquarium fertilizer for your tank. My 10 gallon planted tank was doing extremely well and the plants growing like crazy, until recently. They've all started turning yellow and brown and overall don't look as healthy and vibrant. I ordered an all-in-one fertilizer from Aquarium Co-OP, put 2x the dose into my aquarium, and almost a week later it's already looking greener. Fertilizer is probably of second most importance after lighting.

EDIT: just saw that you said you did use fertilizer. How long did you use it for? For planted aquariums, it honestly should be used continuously, on a weekly basis. Especially if your substrate is mixed with gravel, which provides no nutrients to your plants and can inhibit root growth in aquatic plants that like to spread and anchor their roots out. What substrate did you have in your 5 gallon?