What is the difference in security between a VPN- and a SSL-connection?

VPN means "Virtual Private Network". It is a generic concept which designates a part of a bigger network (e.g. the Internet at large) which is logically isolated from the bigger network through non-hardware means (that's what "virtual" means): it is not that we are using distinct cables and switches; rather, isolation is performed through use of cryptography.

SSL (now known as TLS) is a technology which takes a bidirectional transport medium and provides a secured bidirectional medium. It requires the underlying transport medium to be "mostly reliable" (when not attacked, data bytes are transferred in due order, with no loss and no repetition). SSL provides confidentiality, integrity (active alterations are reliably detected), and some authentication (usually server authentication, possibly mutual client-server authentication if using certificates on both sides).

So VPN and SSL are not from the same level. A VPN implementation requires some cryptography at some point. Some VPN implementations actually use SSL, resulting in a layered system: the VPN transfers IP packets (of the virtual network) by serializing them on a SSL connection, which itself uses TCP as a transport medium, which is built over IP packets (on the physical unprotected network). IPsec is another technology which is more deeply integrated in the packets, which suppresses some of those layers, and is thus a bit more efficient (less bandwidth overhead). On the other hand, IPsec must be managed quite deep within the operating system network code, while a SSL-based VPN only needs some way to hijack incoming and outgoing traffic; the rest can be down in user-level software.

As I understand your question, you have an application where some machines must communicate over the Internet. You have some security requirements, and are thinking about either using SSL (over TCP over IP) or possibly HTTPS (which is HTTP-over-SSL-over-TCP-over-IP), or setting up a VPN between client and server and using "plain" TCP in that private network (the point of the VPN is that is gives you a secure network where you need not worry anymore about confidentiality). With SSL, your connection code must be aware of the security; from a programming point of view, you do not open a SSL connection as if it was "just a socket". Some libraries make it relatively simple, but still, you must manage security at application level. A VPN, on the other hand, is configured at operating system level, so the security is not between your application on the client and your application on the server, but between the client operating system and the server operating system: that's not the same security model, although in many situations the difference turns out not to be relevant.

In practice, a VPN means that some configuration step is needed on the client operating system. It is quite invasive. Using two VPN-based applications on the same client may be problematic (security-wise, because the client then acts as a bridge which links together two VPN which should nominally be isolated from each other, and also in practice, because of collisions in address space). If the client is a customer, having him configure a VPN properly looks like an impossible task. However, a VPN means that applications need not be aware of security, so this makes it much easier to integrate third-party software within your application.

Both have security issues if not configured correctly. But first lets start with some definitions:

Cisco have a good definition of a VPN:

VPN can take several forms. A VPN can be between two end systems, or it can be between two or more networks. A VPN can be built using tunnels or encryption (at essentially any layer of the protocol stack), or both, or alternatively constructed using MPLS or one of the “virtual router” methods. A VPN can consist of networks connected to a service provider’s network by leased lines, Frame Relay, or ATM, or a VPN can consist of dialup subscribers connecting to centralized services or other dialup subscribers. https://www.cisco.com/c/en_in/products/security/vpn-endpoint-security-clients/what-is-vpn.html

As for SSL:

SSL (Secure Sockets Layer), also known as TLS (Transport Layer Security), is a protocol that allows two programs to communicate with each other in a secure way. Like TCP/IP, SSL allows programs to create "sockets," endpoints for communication, and make connections between those sockets. But SSL, which is built on top of TCP, adds the additional capability of encryption. http://www.boutell.com/newfaq/definitions/ssl.html

In relation to your question, the main difference is that SSL often makes use of the browser to encrypt data between end user and the server, and is commonly used for areas of websites that require the protection of confidentiality and integrity of the data.

VPN/IPSEC requires specific VPN Client software and is generally for providing remote access to systems or networks. Also there is the option to go for L2TP or L2F instead of IPSEC.

However, SSL VPNs are becoming more prevalent as a means to provide access to networks / systems via the web browser. This approach has many benefits as it uses the common web browser to enable the secure connection. The granularity of this approach is also a good way to control accesses to specific applications.

As for security issues -

SSL -

• Weak security cyphers could lead to the ability to conduct man-in-the-middle style attacks against the end user, resulting in a loss of confidentiality / integrity of the data.

  • Poorly configured mix of HTTP / HTTPS content could also lead to a loss of confidentiality / integrity of the data.

IPSEC -

• Introduction of a potential DoS condition. An example of this would be http://www.cisco.com/en/US/products/products_security_advisory09186a0080b20ee5.shtml

• Loss of confidentiality issues such as a 2008 Microsoft issue whichcould cause systems to ignore IPsec policies and transmit network traffic in clear text. https://docs.microsoft.com/en-us/security-updates/securitybulletins/2008/ms08-047

Some very good answers here, I won't repeat what was already said.
However, one point I found to be lacking - SSL is a lot easier to setup on an ad-hoc basis, especially if you don't have a requirement for client certificates.
IPsec, on the other hand, always requires client certificates (assuming a normal, typical setup), and there are also other difficulties in initial setup and distribution.

As such, IPsec is usually more fit for a controlled network, and less so over the wild wild unknown Internet. See some more info at this other question: "IPsec (Internet Protocol Security) facts".

Thus, getting back to your actual question, in almost all cases where you're putting the server on the Internet, you wouldnt expect your users to connect using a VPN. (Exceptions exist, of course.)
Instead, just simply set up SSL certs on your server, point your clients at it, and you're good to go (just make sure you explicitly validate the certificate, depending on what language/technology/library you're using...)