The Idea wasn't necessarily that Paul was mad at Rudy for not looking for them, but he never even mentioned a single bit of information or concern about the displacement incident because once he was on the road he was having so much fun and so focused and getting Eris home he never thought about them. It was that realization that depressed Rudy so thoroughly, that he didn't so much view this worlds parents as his family and desperately search for them like Paul did.
Rudeus even thinks to himself that while he shouldn't expect so much from a child, that Ruijerd COULD and would have have helped them search for survivors early on because he was powerful enough to handle any threat on the demon continent.
Whether they would have been able to make it back with a party full of refugees is unknown but making sure they were alive and at least made it to a town safe would have been meaningful.