How to install libx265 for ffmpeg on Mac OSX

Try using --with-x265 instead of --enable-libx265 as described in the official FFmpeg wiki. https://trac.ffmpeg.org/wiki/CompilationGuide/MacOSX

Those guides you linked to are not specifically for homebrew, which is why they used --enable instead of --with.

Seems like you can't just do brew install ffmpeg --with-x265 anymore (this is Oct 2019). What worked for me is this:

brew tap homebrew-ffmpeg/ffmpeg
brew install homebrew-ffmpeg/ffmpeg/ffmpeg

This is from https://github.com/homebrew-ffmpeg/homebrew-ffmpeg. It says the default already includes x265, but you can then optionally pass some arguments like

brew install homebrew-ffmpeg/ffmpeg/ffmpeg --with-xvid --with-fdk-aac

You can take a look at what options are available by doing

brew options homebrew-ffmpeg/ffmpeg/ffmpeg