What are the differences between HSM and SE?

A hardware security module (HSM), a secure element (SE), a smart card, a trusted execution environment (TEE) and a secure enclave (SE again) are all computing environments designed for secure execution. They generally have some properties in common:

• They are isolated environment with a degree of tamper resistance. They're designed to make it hard to extract data from or crack.
• They hold some unique cryptographic keys which allow off-device entities to communicate securely with one specific device. In particular, they can't be cloned.
• They either only run code from the manufacturer, or if they allow third-party code, it has to be signed, often not by the end-user of the device.
• They offer a limited set of interfaces in order to reduce the attack surface.

The difference between these various types of environments is their form factor. This can influence how much they fulfill the various criteria; for example an environment that is isolated by software can't reach the same level of resistance to side-channel attacks as one that has physical protection. The exact definitions can vary. Typically:

• A hardware security module comes in a big box and lives in a server room. It either has its own shelf on the rack or plugs into a PC extension card slot. It has some physical tamper resistance, designed to self-destruct if you try to open the box. It's plugged in 24/7, has a battery inside, and will die if the battery runs out. The physical connectivity can vary; Ethernet, PCI and USB are common. HSM usually have a proprietary communication protocol but offer at least a PKCS#11 interface.
• A smart card is a chip which is usually embedded in a credit card-sized piece of plastic. Some are smaller, for example SIM. It takes its power from the device it's plugged into (or the device it's in radio range of, if it's contactless) and sits in someone's wallet while not in use. Calling something a smart card generally implies a ISO/IEC 7816 interface.
• A secure element is smaller than your nail and soldered to a board or may even be part of a system-on-chip package. It's fully isolated from other computing environments on the same board or package, but it may or may not be resistant to physical tampering. A secure element plugs in to whatever bus the chip has and has a proprietary logical interface (but often the manufacturer will provide a PKCS#11 interface on top).
  Sometimes, if the secure element is inside the same chip package as the main processor, it's called a secure enclave. This is the case of Apple's secure enclave, for example. But sometimes a “secure enclave” is only logically isolated and not physically isolated.
• A trusted execution environment or secure enclave is a software environment which runs on the same processor as a less-secure environment. It's isolated by a small piece of software that relies on hardware functionality (e.g. Arm TrustZone, Intel SGX). Terminology and interfaces vary depending on whether you're in the Intel or Arm world.

Once again, keep in mind that what I'm presenting here are typical characteristics, not formal definitions. In a nutshell, if it hurts when you drop it on your foot, it's an HSM. If you carry it in your wallet, it's a smartcard. If it's a non-removable smartcard, it's a secure element.

The insides of a smart card, small HSM (USB dongle size), TPM, SE, etc. are all the same. There is a question of how much storage it has, which primitives/algorithms are supported, and how many operations per second it can do with those primitives, and the specifics of the native API it supports (it has to implement the TCG spec to be a TPM, and regardless of native API, you can translate from whatever native API it has to something like pkcs11 and make it more generally useful).

You can specify "is part of a bigger SoC, is meant to be used over I2C from the general purpose CPU in the SoC", "same as before but has DMA", "is meant to be used over NFC", "is meant to be used over USB port", "is meant to be soldered onto PCB and used over this or that serial-like bus", "is meant to be rack mounted and used over ethernet" to say what which kind of device you mean exactly.

That chip can be used to implement a TPM or HSM or other things, by exposing the appropriate API (might require additional hardware).

In practice, it seems "SE" is a marketing term that has no specific technical meaning. If you want to be understood, you have to say whether you mean a device usable by an end-user or a component meant to be integrated into a device, and its form factor (and capabilities).