How is an ATM secure?
I think the assumption here is wrong. They don't have physical access to the machine. They have supervised access to a very limited control panel for a machine which is built into a bomb-proof safe, bolted to the ground and hooked up to an alarm system with an armed response force.
Get the machine out of the vault and away from supervision and then yes... all bets are off.
ATM are supposed to be tamper resistant, and to actively react upon any detected breach of physical security, notably by marking bills with some highly conspicuous and hard to remove ink, and also by committing honourable seppuku. For that matter, an ATM should be compared with HSM, payment terminals and smart card. You can imagine the ATM as a kind of Davy Crockett entrenched in Alamo fort and shouting "you'll never take me alive !". By comparison, a basic PC lacks all forms of tamper-resistance and would be more adequately compared with an open buffet at a charity event guarded by non-violent buddhist monks who will discourage discourteous behaviour by making stern faces and striking perpetrators with severe glares only.
In practice, most attacks on ATM are attacks on the ATM environment, e.g. skimming: the ATM itself is untouched, but the debit card is spied upon during its physical transit from the owner's wallet to the ATM entrails.
The adage is still accurate. Physical access to the machine is not the same as physical ability to interact with the machine. The vast majority of attacks against a physical box involve actually altering the hardware and there is a limited amount you can do to alter the hardware of an ATM as it is locked in a safe, away from the user.
It is, however, worth noting that one of the most successful attacks against ATMs is to cover the keypad and card reader with a card reader and keypad of the attacker's own design. They can then use this to scan the card and get the pin. This allows them to clone the card and access the ATM themselves.
The machine itself is still safe since physical access is restricted, but the interface is not physically protected and is thus easily open to security threats. This is why cameras often watch the ATM to look for the installation of such hardware.