How do I start a screen session using an upstart job with a non privileged user?

Invoking screen via upstart is indeed somewhat tricky. The first problem regarding non-existing /var/run/screen can be easily solved though.

On Ubuntu 10.10 to 13.10 there is an upstart task which is responsible for cleaning up and (re-)creating /var/run/screen on bootup so you need to make sure your upstart script will run after it:

start on stopped screen-cleanup

On Ubuntu 10.04 and earlier as well as Ubuntu 14.04 and later that code is in the init script /etc/init.d/screen-cleanup which means upstart jobs can refer to it as the result of rc:

start on stopped rc

However, screen will probably still complain about /var/run/screen permissions. This can be workarounded by invoking screen via setsid:

exec setsid screen -Dm /some/command

Your screen session will fork once, so you need to add the "expect fork" stanza to make sure upstart follows the correct pid.

Here's a complete example script (needs at least Ubuntu 12.04):

# screen startup script
# requires upstart v1.4 or newer

description "running top in screen session"

start on ( local-filesystems
           and stopped rc )
stop on runlevel [!2345]

respawn

setuid test
setgid test

# "setsid screen -Dm" only forks once
expect fork

# use setsid to avoid screen complaining about /var/run/screen permissions.
exec setsid screen -Dm -S mytopsession /usr/bin/top