There were no aliens walking around (apart from the Spaniards themselves), there were no giants, there was no fancy technology, the technology you also have no idea off but keep pushing anyway.
I know there were no visitors around when the Spanish arrived. They had probably departed thousands of years before. You are making the very point that I'm making; that had the Inca have made the precision relics they would have told the Spanish about it surely? It would be part of their established culture and architectural method.
Yet the Spanish reported no such thing. All they got from the Inca was denial. The deduction from this is that the Inca were the inheritors of the megalithic relics. It's clear.