Building a complete online payment gateway like Paypal

What you're talking about is becoming a payment service provider. I have been there and done that. It was a lot easier about 10 years ago than it is now, but if you have a phenomenal amount of time, money and patience available, it is still possible.

You will need to contact an acquiring bank. You didnt say what region of the world you are in, but by this I dont mean a local bank branch. Each major bank will generally have a separate card acquiring arm. So here in the UK we have (eg) Natwest bank, which uses Streamline (or Worldpay) as its acquiring arm. In total even though we have scores of major banks, they all end up using one of five or so card acquirers.

Happily, all UK card acquirers use a standard protocol for communication of authorisation requests, and end of day settlement. You will find minor quirks where some acquiring banks support some features and have slightly different syntax, but the differences are fairly minor. The UK standards are published by the Association for Payment Clearing Services (APACS) (which is now known as the UKPA). The standards are still commonly referred to as APACS 30 (authorization) and APACS 29 (settlement), but are now formally known as APACS 70 (books 1 through 7).

Although the APACS standard is widely supported across the UK (Amex and Discover accept messages in this format too) it is not used in other countries - each country has it's own - for example: Carte Bancaire in France, CartaSi in Italy, Sistema 4B in Spain, Dankort in Denmark etc. An effort is under way to unify the protocols across Europe - see EPAS.org

Communicating with the acquiring bank can be done a number of ways. Again though, it will depend on your region. In the UK (and most of Europe) we have one communications gateway that provides connectivity to all the major acquirers, they are called TNS and there are dozens of ways of communicating through them to the acquiring bank, from dialup 9600 baud modems, ISDN, HTTPS, VPN or dedicated line. Ultimately the authorisation request will be converted to X25 protocol, which is the protocol used by these acquiring banks when communicating with each other.

In summary then: it all depends on your region.

• Contact a major bank and try to get through to their card acquiring arm.
• Explain that you're setting up as a payment service provider, and request details on comms format for authorization requests and end of day settlement files
• Set up a test merchant account and develop auth/settlement software and go through the accreditation process. Most acquirers help you through this process for free, but when you want to register as an accredited PSP some will request a fee.
• you will need to comply with some regulations too, for example you may need to register as a payment institution

Once you are registered and accredited you'll then be able to accept customers and set up merchant accounts on behalf of the bank/s you're accredited against (bearing in mind that each acquirer will generally support multiple banks). Rinse and repeat with other acquirers as you see necessary.

Beyond that you have lots of other issues, mainly dealing with PCI-DSS. Thats a whole other topic and there are already some q&a's on this site regarding that. Like I say, its a phenomenal undertaking - most likely a multi-year project even for a reasonably sized team, but its certainly possible.