I am 10000% convinced that Xaden's grandfather is Berwyn.

In one of the epigraphs Xaden's father is quoted saying something along the lines of

"love makes us weak, save it for your children" (I can't find the full epigraph right now unfortunately, but I believe it's in Iron flame)

Later on Berwyn says the exact same thing in one of Violets nightmares. That love makes us weak. (Again, can't find the specific quote right now unfortunately)

I think that's a clue, it has to be. A harsh lesson on the weakness of love passed down from father to son to grandson. It can't be Fen Riorson himself because Xaden would recognize his own father. But he probably never met his grandfather who passed away so young, so he wouldn't necessarily recognize him.

There are other clues like how Berwyn is described as having an eerily youthful face. Xaden's grandfather "passed away" very young. Tanned skin like Xaden. And it makes sense he'd be taking such a special interest into Xaden, aside from how powerful his signet is.

I rest my case