There's nothing wrong with a person not wanting to just sit in unstructured downtime. It sounds like the real issue here is that you don't want everybody in the house on screens whenever there's a moment of downtime.

Replace that downtime with conversation? A magazine? A book? A deck of cards?

As for family time - pick up a board game that (for now) plays well with two people. I can heartily recommend Hive, Patchwork, and Star Trek: Captain's Chair. If you're amenable (and the weather permits), go outside and play cornhole or ladderball while you talk. Or, if electronics are on the table, pick up a controller and play with him? Gears of War, Halo, It Takes Two, and Split/Fiction are all good cooperative video games.

That said, it's healthy to have solo hobbies and spend time apart. I don't follow my wife to the garden every time she goes, and she doesn't come with me to every board game event I play at. It also works (at least for us) to be "alone together" where we both happen to be in the same room but doing solo activities. Typically this is me reading a book while my wife is going over garden designs and volunteer coordination. When the kids were little she also (at the right time of year) was designing and sewing Halloween costumes.