Is momentary physical access dangerous?
That all depends on the system, the attacker, and the level of preparation they had. If they have unlimited preparation, they could do effectively anything that they could do with an unlimited access window. Even if they do not have in-depth knowledge of the specific system, it would not be difficult to very quickly inject malicious code that allows for subsequent remote access. They could:
Connect a PCMCIA or PCIe card and dump memory or inject code.
Splice a hardware keylogger in between the keyboard's PS/2 or USB cable.
Quickly download and execute malicious code, or modify existing code.
Access sensitive files and save them (e.g. with a camera or USB flash drive).
Physically destroy the computer (e.g. with a hammer or a power surge over USB).
Simply grab the system and pawn it off for a quick buck.
Time for a story. I once had a target who I would be in close proximity to for a brief period. My goal was to gain persistence on their laptop to exfiltrate sensitive documents. I knew I had only a few seconds every time they went out of sight, so I couldn't just grab their laptop and take my time. I obviously also could not steal it. Luckily, I came prepared. I had a programmable USB device that I plugged in. As soon as it was plugged in, it simulated keyboard input to open PowerShell and execute a few commands to download a payload I had set up earlier. The scenario went like this:
I waited until this person had left to get something for me in another room.
I leaned over on the table where the laptop was and surreptitiously plugged in the device.
I waited a few seconds to be safe, unplugged it, and tried to keep a straight face.
After they gave me what I asked for, I thanked them and left.
When I got home, I got on my computer and connected to their machine.
It was not difficult, did not take an extensive period of preparation, and was moderately stealthy. I could have made it even more stealthy if I used something that looked like a cell phone so I could claim I was just charging the device and didn't see any other USB ports around. The moral is that you can do a lot with just a few seconds of access, so you must never underestimate the risk.
So how do you protect against these threats? You need to develop a threat model. Figure out who your adversary is, what assets of yours they are after, and what their resources are. If you don't want your mother to see your porn when you're at her house, you probably don't need to worry about exploits abusing corrupt EDID in a VGA or HDMI cable. If you are holding extremely valuable company secrets in a highly competitive industry (robotics, epoxy, etc) and are going to a high-risk country like France or China, you absolutely need to worry about sophisticated attacks, because industrial espionage (aka the more illicit side of "corporate intelligence") is rampant. Stay with your computer at all times in adversarial situations. Lock it if you are going out of its line of sight, and bring it with you or physically secure it in a safe if you are going to be away for a longer period.
I think you're also missing that there's other malicious things to be done beyond hacking. There are USB devices that can literally fry a computer. So even if they don't gain access or install dangerous software, a "few seconds" could cause thousands of dollars in damage, not to mention downtime to repair the affected systems.
I can think of one way to prolong the physical access.
USB drives can be so tiny that the entire electronic portion fits under the contacts, and the tab is just so you can pull it out. You may modify one by cutting the tab off to fit stealthily inside a slot, so that the owner might not notice something's there right away. And by the time they do, who knows when or where someone may have inserted it? How often do you check your USB ports? :p
Also, many laptops have that plastic placeholder SD card to keep the lint and dust out. Do you check that often too?
And thirdly, with this stealthy approach, you may be able to manufacture a USB cable which includes a controller which acts as a normal charge/data cable but can become a man-in-the-middle for any connected devices. Or use an already existing "USB flash drive in a cable" accessory like this one:
