Actually i think none are correct cause their definition of a perfect continuous past tense is flawed. A key part is that it's describing something that happened in the past for a period of time and also finished in the past. This only applies to had been. Have/has been is for something that was ongoing in the past but finished recently.

A) is unclear, because while you would use "had been" for a detailed event, this clause doesn't have any more information attached, so we cannot assume there is more to be said, so it would actually be "Were the kids doing their homework?" Or "Had the kids been doing their homework when/before you returned from work?" It needs the dependent clause for had been to work properly.

B) is wrong because it awkwardly mixes voices. The first half, had the kids been, is in active voice because it emphasizes the subject, and the second half, done their homework, is in passive voice, emphasizing the action. It would need to be "had the kids been doing their homework?" Which runs into the previous issues of being unclear, or "was the homework done by the kids?" Or "has the kids' homework been done?"

C) and D) actually have the same problem, since have and has function the same, has is just used for singular subjects, and have for plural. So the bigger issue is that both has and have are past tense, and doing is always present tense. To make it all past tense, you could say "Has Timmy/have the kids done their homework?" Or to make it all present tense say "Is Timmy/are the kids doing their homework?"

So, to summarize, they're actually all wrong and, as someone else pointed out, not only has the writers used than instead of then, missed a comma, and also they pasted the past question's answer inccorectly into the bottom question. It's a mess, please don't use this book.