Was Ishmael the son of Abraham? Yes
Was Ishmael the son of Abraham and Sarah? NO
Was Ishmael the son of the Promise/convenant? No
Ishmael had been "born according to the flesh"
and
Isaac had been "born according to the promise"
Galatians: 4
22. For it is written, that Abraham had two sons, the one by a bondmaid, the other by a freewoman.
23. But he who was of the bondwoman was born after the flesh; but he of the freewoman was by promise.
24. Which things are an allegory: for these are the two covenants; the one from the mount Sinai, which gendereth to bondage, which is Agar.
25. For this Agar is mount Sinai in Arabia, and answereth to Jerusalem which now is, and is in bondage with her children.
26. But Jerusalem which is above is free, which is the mother of us all.
27. For it is written, Rejoice, thou barren that bearest not; break forth and cry, thou that travailest not: for the desolate hath many more children than she which hath an husband.
28. Now we, brethren, as Isaac was, are the children of promise.
29. But as then he that was born after the flesh persecuted him that was born after the Spirit, even so it is now.
30. Nevertheless what saith the scripture? Cast out the bondwoman and her son: for the son of the bondwoman shall not be heir with the son of the freewoman.
31. So then, brethren, we are not children of the bondwoman, but of the free.
Was God correct in calling Isaac "Your Only Son"? YES
Which is why only Christians can truly call God, Father.