Since Allah Almighty ordered Abraham the father of Ishmael and Isaac peace be upon them to build the Kaaba, people called it the House of GOD Almighty, and it became the direction for us to bow down to Allah Almighty in worship:
"Behold! We gave the site, To Abraham, of the (Sacred) House (i.e., the Kaaba), (Saying): 'Associate not anything (In worship) with Me; And sanctify My House For those who compass it round, Or stand up, Or bow, or prostrate themselves (Therein in prayer). And proclaim the Pilgrimage among men: they will come to thee on foot and (mounted) on every kind of camel, lean on account of journeys through deep and distant mountain highways:" (The Noble Quran, 22:26-27)
So when Muslims prostrate to Allah Almighty, they prostrate in the direction that is as close as possible to Him. It was also prophesied in the Noble Quran that people from all over the world would come to the city of Mecca (where the Kaaba is located) for pilgrimage.
During the early days of Islam, the Muslims were commanded to bow down to Allah Almighty in the direction of the Farthest Mosque (Al-Masjid Al-Aqsa) in Jerusalem, because it was the Holiest Mosque/Temple to worship Allah Almighty in back since it was not defiled with idols. Allah Almighty then commanded Prophet Muhammad to order the Muslims to prostrate to Allah Almighty in the direction of the Kaaba in Mecca, because it was the House of GOD Almighty which was built by Prophet Abraham peace be upon him (mentioned also in the Bible), and because after the Muslims liberated Mecca in a 10,000 men army (this event was prophesied in the Bible), they cleaned the Kaaba from all idols.
"Say: 'O People of the Book (i.e., Jews and Christians)! Come to common terms as between us and you: That we worship none but Allah; that we associate no partners with Him; that we erect not, from among ourselves, Lords and patrons other than Allah.' If then they turn back, say ye: 'Bear witness that we (at least) are Muslims (bowing to Allah's Will)." (The Noble Quran, 3:64)
"God forgiveth not that partners should be set up with Him; but He forgiveth anything else, to whom He pleaseth; to set up partners with God is to devise a sin Most heinous indeed." (The Noble Quran, 4:48)
"Who can be better in religion than one who submits his whole self to God, does good, and follows the way of Abraham the true in Faith? For God did take Abraham for a friend. (The Noble Quran, 4:125)
The Muslims on the hand are not idol worshipers. And neither were the Bible's Prophets (including Jesus), peace be upon them, who prostrated to GOD Almighty.
The Kaaba is only the House of GOD Almighty and is the meeting place for us to meet the Almighty LORD by bowing down our faces in humility down to the ground before Him in that direction.
On the other hand, the hindus' and the buddhists' idols are statues. The same also for those who worship the fire, sun, moon, stars, etc... Allah Almighty Said:
"Among His Sings are the Night and the Day, and the Sun and the Moon. Adore not the sun and the moon, but adore God, Who created them, if it is Him ye wish to serve." (The Noble Quran, 41:37)
"They ask thee Concerning the New Moons. Say: They are but signs To mark fixed periods of time In (the affairs of) men, And for Pilgrimage. It is no virtue if ye enter the houses from the back: It is virtue if ye fear Allah. Enter houses Through the proper doors: And fear Allah: That ye may prosper. (The Noble Quran, 2:189)
*The Kaaba is the Qibla i.e the direction Muslims face during their prayers. It is important to note that though Muslims face Kaaba during prayers, they do not worship the Kaaba. Muslim worship and bow to none but Allah. It is mention in the Quran
"We see the turning of thy face (for guidance) to the heavens: now shall We turn thee to a Qibla that shall please thee. Turn then thy face in the direction of the Sacred Mosque:wherever ye are, turn your face in that direction." (Al-Baqarah 2:144)
* Kaaba is the Centre of the World Map. The Muslims were the first people to draw the map of the world. They drew the map with the south facing upwards and the north downward. The Kaaba was at the centre. Later, Western Cartographers drew the map upside down with the north facing upwards. Yet, the Kaaba is at the centre of the world map.
* Tawaf around the Kaaba for indicating one God - when Muslims go to al-masjid al-Haram in Mecca, they perform tawaf or circumambulation around the Kaaba. This act synbolizes the belief and worship of One God, since just as every circle has one centre, so also there is only one Allah worthy of worship.
* People stood on Kaaba and gave the adzan - At the time of the Prophet, people even stood on the Kaaba and gave the adzan or the call to prayer. One may ask those who allege that Muslims worship the Kaaba; which idol worshipper stands on the idol he worships?