Attacks via physical access to USB (DMA...?)

Before we start, a quick word about how USB works. A USB device connects to a piece of hardware inside the computer called the USB host controller. This decodes the complex USB protocol, which, much like a network protocol like TCP, contains packets, checksums, the equivalents of SYNs, ACKs, etc. The USB host controller decodes each individual connected USB device and presents it to the operating system, where it can interpret it however it wants (e.g. as a block device, or a mouse, or even a dialup modem, you name it). The USB controller itself connects to the PCH (which is like a relay station) over PCI. It's not a bulky PCI slot, but it's PCI nevertheless. The USB controller is what you see when you look at the output of lspci on your system and see mentions of USB. You probably have a few USB controllers, each connected over PCI to the PCH. The PCH in turn connects to the CPU. See this great Stack Exchange post that has a very good graphical description of USB controller anatomy.

Now let's break the possible types of attacks into three parts.

Intentional behavior

Most Linux distros auto-mount USB devices with filesystems, and some even have the ability to autorun, though this is rarely enabled by default. This was more a problem in the olden days of Windows, and never was really an issue in Linux. However, USB 3.1 does support DMA, so you might want to avoid that. I do believe it is disabled by default though, so it's not a massive risk. Just be aware that the USB controller allows it.

Exploiting software

There have been multiple instances of vulnerabilities in the partition table parsing code, where simply plugging in a USB stick can exploit the system by way of a specially designed but corrupt partition table. Though that's pretty rare (I am only aware of two that allowed for arbitrary code execution), vulnerabilities for auto-mounting filesystems are much more common, considering how incredibly complex filesystems are. They parse a huge amount of structures, and they prioritize performance above almost all else, so sanity checks are often not done unless they are necessary. This gives way to an increased number of vulnerabilities. But even if you aren't affected by any of those, there are plenty of vulnerabilities for your favorite desktop environment and how it manages displaying folder contents, such as thumbnail generation... Just recently there was a vulnerability in ffmpeg that allowed it to phone home to arbitrary servers, and before that, a vulnerability in libpng that allowed arbitrary code execution.

Exploiting hardware

This is by far the scariest and hardest to mitigate. While (most) USB protocols do not allow for DMA, and none allow for it by default, the USB controller is connected to the PCH over PCI, which has DMA abilities. The USB controller promises never to do evil, of course, and promises it will tell any mean USB devices to fsck off if they refuse to follow the protocol, but it is certainly possible for the controller to be hijacked by a malicious USB device which presents it with corrupt packets. Once it gains control of the protocol, it has full DMA and can do anything it wants. The only way to protect from this is to use your IOMMU to isolate the PCI device. There are several ways to do this, such as using Qubes OS which, while it has its share of shortcomings, is specifically designed to protect against hardware attacks such as this one. Or you could be creative. I went and made my own small bootable Linux distro which I use in combination with QEMU/KVM, a technology called VFIO, a driver called usbip, and a few custom scripts and programs which mitigate the issue. But if you rely on your IOMMU to protect you, just make sure it supports interrupt remapping. Otherwise it can be bypassed fairly easily and provides little security benefit.

USB 3.0 can be especially problematic. I'm not talking about 3.1 allowing DMA when the host says OK, but running outside the kernel. USB 3.0 runs as a binary blob in the BIOS, much like the Intel Management Engine. See this and this. It has a very large attack surface, adding to the already large surface area of the USB host controller hardware. You can disable it in many BIOSes, usually under a name like "xHCI controller". Doing this will effectively turn all 3.0 ports into 2.0 ports which decreases their speed, but it improves security a good bit. In terms of paranoid security, 1.0 is not good, 2.0 is bad, and 3.0 is a nightmare.

There is a kernel patch called grsecurity which has a huge number of security features that vastly increases the security of the Linux kernel. One of the security features is "deny access to new USB devices", which does just as it sounds. Either after you toggle a one-way setting, or after boot, all new USB devices will no longer be allowed access. All the complex, potentially vulnerable kernel code which deals with USB devices is disabled. It will completely mitigate the first two issues, but unfortunately does nothing at all to mitigate the third. Even compiling a custom kernel and removing the USB drivers does not mitigate the third.