Where can I find the transaction protocol used by Automated Teller Machines?

Well, there are lots of interbank networks. I would guess that each of them communicate differently. The stickers on the ATM (Cirrus, STAR, Pulse, etc...) identify which network the machine participates in. I do believe, though, that the "structure" of the message is dictated by an ISO standard. Cirrus is a Mastercard owned network and PLUS is a Visa owned network... I'd scour their sites to see if they publish any API details.

Edit, by request:

Have a look at the following ISOs 15022, 20022, 9362 and 4217 -- http://en.wikipedia.org/wiki/Category:Financial_routing_standards

ISO 8583 is dominant.

Also, take a look at EMV.

The ATM to bank link can be proprietary or standard. It is only upstream where inter-organisation wire level interoperability is needed, that standards become always necessary.

ISO 15022 definitely doesn't cover ATM to bank. So far, it covers further upstream. And is now superseded by ISO 20022 - "originally named ISO 15022 2nd edition".

ISO 20022 covers the total scope of financial services, and acts as a super forum for ISO financial services protocols.