It was mainly Geography and the swiss preparations for such an attack, it was war gamed a few times by the reichs military but it was unfeasible.
Also Switzerland was an enormous help to germany, IBM, Ford, GM, Standard oil and various banking groups from the US were able to keep lines of communication, finance, assistance and in some cases, required parts for the Nazi war machine were able to move through the all to eager to help Swiss.
Germany was able to take advantage of the Swiss banking system and used Switzerland like an enormous Pawn shop for stolen art, gold and other valuables.
Switzerlands help probably prolonged the war by months