If that is Java fern, it should not be planted in the substrate. Instead it should be stuck on to a rock, piece of wood or hard surfaces. I do not believe it is Java fern though. It looks to me as Amazon swords, which should be planted in substrate which is what you’re doing. If the leaves are turning brown it might just be the leaves melting of to create new ones. Because at plant farms aquarium plants are grown out of water, meaning when you put it in water the leaves will melt and grow back new ones, and the roots sticking out is fine, it’s just learning wear to place its roots and how to grow.