Is physical security less important with disks on a server being encrypted?

Physical access, in many, likely most, situations means a total loss of security - for a variety of reasons (this all assumes encrypted disks):

• Theft - An attacker could steal the server or disks, to attack at their pace. This allows an attacker to take their time, and you have no idea if they've actually gained access to data.
• Physical Modification - If I can access a server, I could add hardware, this could be anything from USB or keyboard logging to adding a wireless interface to allow remote access.
• Cold Boot Attack - There are attacks that can be used to extract encryption keys, allowing decryption of the disks.
• etc.

There are others of course, but this is just a sample of what can happen if an attacker has physical access. There are possible attacks that are still somewhat theoretical, such as applying backdoored UEFI images and the like.

Possibly the worst part of a physical attack, is that you may not even know what exactly was done, so there's a real problem with being able to trust the hardware afterwards.

Physical access is total access. Kinda. Give me physical access to a server with an encrypted disk and the first thing I'd do is plug a key logger into the keyboard to take care of that pesky encryption.

Show up at my door with an encrypted hard drive and I'll format it and dump movies on it.

Encryption is most commonly defeated not by breaking it but by going around it. It only protects you as well as you use it. You have access because you have something that gives you access. Be it password, RFID, finger print, whatever. Give me physical access while you're still using it and I'll figure out how you get access.

Disk encryption can be defeated by replacing the machine with a malicious one that looks and behaves exactly the same but its only purpose being to fool the legitimate user into typing in the FDE password. In case of a local user it can be as simple as an USB keylogger, in case of a remote user (entering the key via SSH) you need to extract the SSH private keys (which are located on the unencrypted part of the storage since the FDE key isn't yet available) and then start your own SSHd with that key and wait for the user to return.