How is the photoshop cutout filter implemented?

Edge detection is usually a Sobel or Canny filter then the edges are joined together with a chain code.
Look at something like the OpenCV library for details

Did you see this post. It explains how to get the same result using ImageMagic, and IM is opensource.

I'm not sure it could be some kind of cell shading, but it also looks like a median filter with a very big kernel size or which was applied several times.

The edge simplicity/fidelity might be options which help decide whether or not to take in account an adjacent pixel (or one which falls inside the kernel) based on difference of color with the current pixel.