Use .htaccess to set 404 error document located in the same folder

The URL-path in the argument of the ErrorDocument directive is always relative to the DocumentRoot.

However, you can work around this using the mod_rewrite module.

RewriteEngine on
#if requested resource isn't a file
# and isn't a directory
# then serve local error script 
RewriteCond %{REQUEST_FILENAME} !-f  
RewriteCond %{REQUEST_FILENAME} !-d  
RewriteRule .* 404.php [L] 

Make sure the 404.php does actually emit a 404 response header!

I haven't tested it, but this should get you started. Also note that using the ErrorDocument directive is always preferable to this, and asking where the site will reside relative to DocumentRoot would be sensible, so you'd be able to write

ErrorDocument 404 /path/to/my/site/404.php

While a number of directives are sensitive to relative directory while within an .htaccess or <Directory> context, ErrorDocument is not. Per the documentation:

URLs can begin with a slash (/) for local web-paths (relative to the DocumentRoot), or be a full URL which the client can resolve.

Source