Apple - Install El Capitan with VirtualBox on OS X

The dmg file (Install OS X El Capitan.app/Contents/SharedSupport/InstallESD.dmg) is no bootable image.

To create a bootable file use the following script. You may have to modify some paths (i.e. the user name in the last line and the path to Install OS X El Capitan.app (and the inner InstallESD.dmg file) in the 6th line):

#!/bin/bash  

# Create bootable El Capitan ISO

# Mount the installer image  
hdiutil attach "/Applications/Install OS X El Capitan.app/Contents/SharedSupport/InstallESD.dmg" -noverify -nobrowse -mountpoint /Volumes/esd

# Create empty cdr image  
hdiutil create -o ElCapitan.cdr -size 8000m -layout SPUD -fs HFS+J

# Mount the cdr image  
hdiutil attach ElCapitan.cdr.dmg -noverify -nobrowse -mountpoint /Volumes/iso

# Restore Base System to the cdr image 
asr restore -source /Volumes/esd/BaseSystem.dmg -target /Volumes/iso -noprompt -noverify -erase

# Remove Package link and replace with actual files  
rm /Volumes/OS\ X\ Base\ System/System/Installation/Packages

# Copy Base System  
cp -rp /Volumes/esd/Packages /Volumes/OS\ X\ Base\ System/System/Installation
cp -rp /Volumes/esd/BaseSystem.chunklist /Volumes/OS\ X\ Base\ System/
cp -rp /Volumes/esd/BaseSystem.dmg /Volumes/OS\ X\ Base\ System/

# Unmount the installer image  
hdiutil detach /Volumes/esd

# Unmount the cdr image  
hdiutil detach /Volumes/OS\ X\ Base\ System

# Convert the cdr to ISO/CD master 
hdiutil convert ElCapitan.cdr.dmg -format UDTO -o ElCapitan.iso

# Rename the ISO and move it to the desktop  
mv ElCapitan.iso.cdr $HOME/Desktop/ElCapitan.iso

# Clean up temporary file
rm ElCapitan.cdr.dmg

Paste the content in a text editor to a new document as plain text and save it as prepare_elcapitan_cdr.sh on your desktop.

Open Terminal, chmod the script and execute it:

chmod +x ~/Desktop/prepare_elcapitan_cdr.sh
cd ~/Desktop
./prepare_elcapitan_cdr.sh

Detach elcap.dmg from the VM. Use the final ElCapitan.iso as installer iso and attach it to the virtual SATA Port 1. You can delete the ElCapitan.cdr.dmg file.