Compiling x264 with mp4 support

There is no --enable-mp4-output option. MP4 support is automatically detected, but l-smash or gpac are required.

For a script that might compile x264 see HOWTO: x264.
For another example see Compiling X264 on Ubuntu.
Also see the FFmpeg and x264 Encoding Guide.