Think of it this way...
If your father and mother were just in a one-night stand relationship, YOU grow up with no father. YOU PAY THE PRICE
If your mother sins and lies to her boss and he fires her, YOU HAVE NO MONEY. YOU PAY THE PRICE.
Families are affected by their ancestors sins. I think that is part of the "sins of our fathers" argument.