FFMPEG: chroma key / greenscreen filter for images / video

The answer (now) is yes, there is a filter for generating chroma-keys and overlaying them. The filter name is "color key". There are examples on the site, here's the command:

ffmpeg -i <base-video> -i <overlay-video> -filter_complex '[1:v]colorkey=0x<color>:<similarity>:<blend>[ckout];[0:v][ckout]overlay[out]' -map '[out]' <output-file>

where <color> is the rgb color to match in hex (ex: 0x000000 for black), <similarity> is a tolerance on the color-match (ex: 0.3), and <blend> (ex: 0.2) controls whether the opacity is on-off or how gradual it is. (See the documentation for more).

Minimal runnable example with test data

The answer at https://stackoverflow.com/a/32291842/895245 was correct, here's just a minimal concrete example of that.

Download input media:

wget https://github.com/cirosantilli/media/raw/master/Ciro_Santilli_with_a_stone_carved_Budai_in_the_Feilai_Feng_caves_near_the_Lingyin_Temple_in_Hangzhou_in_2012.jpg
wget https://github.com/cirosantilli/media/raw/master/opengl-rotating-triangle.mp4

Make the image size match the video size of 1024x1024. The video size can be determined with ffprobe:

convert Ciro_Santilli_with_a_stone_carved_Budai_in_the_Feilai_Feng_caves_near_the_Lingyin_Temple_in_Hangzhou_in_2012.jpg -resize 1024x1024! background.jpg

Do the actual conversion:

ffmpeg -i background.jpg -i opengl-rotating-triangle.mp4 \
  -filter_complex '[1:v]colorkey=0x000000:0.1:[ckout];[0:v][ckout]overlay[out]' \
  -map '[out]' out.mp4

Convert to gif just for previews on this answer:

ffmpeg -i out.mp4 -r 5 -vf "scale=300:-1,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" out.gif

So in my example, I had a black background, which is getting converted into a fixed image.

Outcome preview (horrendous FPS to fit GIF in 2MB for upload here):

Actual video output: https://www.youtube.com/watch?v=3aY6x7u86QQ

Original input files for reference:

opengl-rotating-triangle.mp4

Ciro_Santilli_with_a_stone_carved_Budai_in_the_Feilai_Feng_caves_near_the_Lingyin_Temple_in_Hangzhou_in_2012.jpg

It also just works with a video background.

wget https://upload.wikimedia.org/wikipedia/commons/f/f9/STS-132_Liftoff_Space_Shuttle_Atlantis.ogv
ffmpeg -i STS-132_Liftoff_Space_Shuttle_Atlantis.ogv -i opengl-rotating-triangle.mp4 -filter_complex '[1:v]colorkey=0x000000:0.1:[ckout];[0:v]trim=start=0:end=8[cut0];[cut0][ckout]overlay[out]' -map '[out]' out.mp4

Here I've added another the parameter shortest=1 to the overlay filter:

overlay=shortest=1

to make the overlay stop as soon as the triangle video stops, otherwise it defaults to stopping when the much longer Atlantis video ends.

You might also want to use the trim filter instead to select an arbitrary segment.

The Atlantis video is 1920x1080, and the final output had that size as well. Preview:

Actual video: https://www.youtube.com/watch?v=HI8XniA2Bk8

Tested on Ubuntu 20.10, FFmpeg 4.3.1.

Not specifically that I know of

Opencv contains all the functions you need to read video, convert to RGB, split the color planes, replace pixels base don color, merge frames and write video.

It's good to research with but it's not going to create a plugin directshow filter to do this automatically