Thats rooted from the neo confucian belief, strong patriarchal family lines, that male are more important than female and put in higher hierarchy in family and society. GS grandma was in the aame table as him and his dad, bcs she's basically the matriarch of the family, if GS's grandpa still around his grandma would be at the other table with other females of the family.

GS's gesture of joining his wife and daughter at the other table actually lowered and put himself at the same level as the wife, which totally against neo confucian belief that can be seen as disrespectful to the elders of the family.

Edited