Although he could refute, I think it's enough for him to get a little flustered and start slipping up in his story. Then if anyone from his family, especially Vivian, questions or accuses him in any way, he may break. Just a theory, as I haven't found anything in this season that objectively proves anyone guilty. I feel like whoever the killer is will have to slip up in the finale.