Hi Joshua,
Your question can not be answeredbecause your question is not correct. Muslims do not believe that Islam started with Ishamael (pbuh), it started with Adam (pbuh). You need to first proove through Quran that it started with Ishamel. If that is not prooven how can we answer? You started with Muslims believe that Islam started with Ishamel, but Muslims have told you that they do not believe so. If I were to say Jews believe in Trinity, would my sayng of this make it the truth? No. If I were to say Jews believe that Jesus (pbh) was a Prophet, , would my sayng of this make it the truth? Hence, you need to first prove that Muslims believe that Ishmael (pbuh) started the religion.
Second thing, what do you thing Prophet Moses (pbuh) was doing when he was being raised as son of Pharoah, wasnt he worshipping demi gods? Before God revealed Himself to him.
Bye for now.... Ali