It actually has to do as much with their mother as their father. But yes, his appearance is a big thing later in the seasons. But it all goes back to that night shown in episode 1.