2 Samuel 12
22 And he said, While the child was yet alive, I fasted and wept: for I said, Who can tell whether GOD will be gracious to me, that the child may live?
23 But now he is dead, wherefore should I fast? can I bring him back again? I shall go to him, but he shall not return to me.
24 And David comforted Bathsheba his wife, and went in unto her, and lay with her: and she bare a son, and he called his name Solomon: and the LORD loved him.
Davids son dies as a child, and he says clearly in verse 23. I SHALL GO TO HIM, BUT HE SHALL NOT RETURN TO ME. That because David will see him again....where? In heaven.
Paul said, "I would not have come to know sin except through the Law" (Rom 7:7).
The baby does not know the law, so yes babies go to heaven.