Is it safe to store critical passwords in server environment variables?
If you're on a Linux system, look at /proc/*/environ and decide if environment variables are a good place to store sensitive information or not. /proc/self is the current process:
$ tr '\0' '\n' < /proc/self/environ
USER=me
LOGNAME=me
HOME=/home/me
PATH=/usr/bin:/bin:/usr/sbin:/sbin
MAIL=/var/mail/me
SHELL=/usr/bin/sh
SSH_CLIENT=1.2.3.4 58195 22
SSH_CONNECTION=1.2.3.4 58195 7.8.9.0 22
SSH_TTY=/dev/pts/1
TERM=xterm
Never mind that the thing setting the environment variable is probably reading a file somewhere.
The thing to remember is that using a password means the password is available to the program. If this password is not provided by a user typing it in every time a program needs it, that password must be accessible based on only the program's access. You can encrypt the password locally and have the program decrypt using a key, but all that does is obscure the password against accidental disclosure; someone who has the same access as the program can do the same things the program can do, which includes reading the encryption key.
The right way to do this is to have the application run as a restricted account, and store the password in a file protected with filesystem-level permissions. Hopefully you can "include" a file or similar in order to keep the password out of a version control system (assuming the VCS has no security controls). To protect against inadvertent disclosure, obscure the password however you want - base64 encode it, use pgp to encrypt, whatever makes sense in your server program's set of options. If you're writing a program to do this, about the best you can do is to prompt a user for the password only when needed, and then purge that password from memory as soon as it's used.
Ultimately, if you have any data that needs to be read as well as written, you're going to end up protecting something with a password (or if you're really paranoid, with a physical hardware smart card and a PIN), no matter how many layers of encryption you have.
This boils down to the basic question of system security vs. convenience. You can add "defense in depth" by having lots and lots of layers of security controls that a malicious actor would have to breach in order to get to the "goods", but then when a legitimate actor wants to read or change some data, they have to go through a bunch of hoops. The alternative is plaintext passwords in text files.
What I'd do if I really wanted to protect some information in a mission-critical system:
Use Full Disk Encryption, so that the contents of the entire persistent storage is encrypted.
Restrict physical access to the machines. Lock the machine's chassis with a secure locking mechanism and control physical access to the keys. Hire muscle (armed guards) to be gatekeepers for access.
Enforce fine-grained Mandatory Access Control (MAC) in the operating system of the device. You can start with something like SELinux on GNU/Linux and set it to Enforcing, and then tailor the policy to the exact needs of the production software, allowing those accounts exactly (and only) the permissions they need to the files they need.
If you're going to have system-specific passwords, and version control for configuration files, you really want to avoid the possible mistake of having a plaintext password mistakenly committed to version control, since it can be hard to dislodge a leaked password from a VCS's cache. Environment variables are one of several viable options for that. The other is a password prompt when the program starts up, but then rebooting the machine and restoring operational status is a manual effort and can't be done autonomously, so there's that convenience vs. security again.
Make sure you have networking specialists on hand to take care of firewall permissions, to minimize your exposure to an attack over the network. Audit (penetration test as well as whitebox test the code) any software that interfaces with external systems, especially the public Internet. "Interfaces" includes not only direct network connections, but also reading or writing "untrusted" data (data whose bytes originated from outside the RAM/disk/CPU of the secure server).
This isn't a complete list, but especially point 4 is probably relevant to you, although if you don't perform at least steps 1 through 3, consideration of point 4 and point 5 is not going to help you very much, because your system is not secure at a fairly fundamental level.
Passing a password in an environment variable is as safe as having the program read it from a file. Only processes running as the same user may read a process's environment, and these processes are allowed to read the same files anyway.
Note that this is different from passing a password on the command line. Command line arguments are readable by all processes running on the same machine (barring hardening measures), not just processes running as the same user.
If you pass a variable through the environment, beware if the program launches other programs. Those other programs will inherit their parent's environment. So don't do this if you fear that the other programs might accidentally leak the contents of their environment.
The flaw in your scenario is “create an appropriate environment variable when the server system is set up”. An environment variable is a dynamic property of a process. You can't create it when setting up a system, not if by setting up you mean something that survives a reboot. What you mean is presumably that the administrator arranged for this variable to be present in the environment when a certain user logs in. This is done through a configuration file (typically ~/.pam_environment or ~/.profile or a file read from ~/.profile). So this solution does not, in fact, move the password out of configuration files.
Setting things up so that passwords are in a user's login-time environment is not a good idea. It means that every process running as that user will have the secret, so it's vulnerable to a leak anywhere.
A password should be put in a file that's aside from the configuration files that are under version control and from the normal deployment mechanisms. It's ok to put the password in the environment at some point if it's convenient, but it should be done for as small a set of programs as possible.