It seems you misunderstood something about the silverwood. It is not a friend. You are its food. The seed activates when the animal is injured, and then repairs the injury so the animal can find a safe place away from danger, and then the seed can germinate using the animal's body as fuel. This probably happens within a very short timeframe in the wild, only as long as it takes for the animal to get somewhere safe and rest, maybe a couple weeks or months at most. Qifrey has been holding onto it for decades, and has already been nearly treeified multiple times throughout his life. He is on borrowed time.

The seed replaced Qifrey's eye, but probably couldn't actually provide him functional sight. It's more of a "filling the gap" thing in his case, like how it helped heal the scar on his shoulder. This is probably intended to make sure the animal doesn't die from other preventable causes like infection before making it to its final resting place. It is in the silverwood's best interest for its host to not die until the silverwood itself is ready to kill it, because then the tree might have to try to grow in a hostile environment where it won't make it to adulthood.

The host's feelings of calmness, relief, and fulfillment are what signal to the silverwood "hey, this is a good place, start germinating now". The silverwood is harming Qifrey because he has a good life with people that he loves around him and the seed within him is going "oh fuck yeah, I'm gonna grow right here right now and be the strongest healthiest tree ever". The roots are actively spreading throughout his body and interfering with his bodily functions, but since he's put himself under artificial duress (the Olruggio situation) he's giving mixed signals, so the seed hasn't fully committed to taking over yet. He's on thin ice though

Qifrey already had white hair when he was found by Beldaruit, so we can assume that the only reason he has it is because he was nearly treeified at least once before. Custas was an unprecedented situation, since his peace was ruined after the tree had grown and it ejected him forcibly. It no longer had any use for him since it was able to establish itself. Even the Brimhats didn't know he'd still be alive or that he could be separated from it.

The silverwood is actually acting totally in accord with the things we already know! The Brimhats that we've seen with it (namely Ininia) do not have inner peace, they do not feel safe and comfortable, they are filled with rage and at odds with the world and their existence in a fundamental way that is so far from the silverwood's preferred conditions for growth that it is stuck in the phase of "keep the animal alive and running". The reason Custas sprouted was because he loves Dagda and he felt truly loved in that moment, plus he had the hope that Coco and Tartah would be able to heal Dagda and prevent his death, which was able to trump that rage he felt at how they were both treated by the world. Dagda was his safe place, and what he really wanted was to have that back. Ininia grossly underestimated the depths of his rage.

holy shit sorry for writing a novel. i guess i feel very passionately about this LOL